The price of an AI token has collapsed. Almost nobody's AI bill has.
Stanford's Institute for Human-Centered AI put a number on the first half of that. In its 2025 AI Index, it reports that the cost of querying a model performing at GPT-3.5 level on the MMLU benchmark "dropped from $20.00 per million tokens in November 2022 to just $0.07 per million tokens by October 2024". That is roughly 280 times cheaper in under two years. The same report notes hardware price performance improving about 30% a year and energy efficiency about 40% a year.
Cheaper inputs did not produce smaller bills, because the cheaper it became to ask, the more we asked, and the sloppier we became about how we asked. Our view is that this is the beginning of a genuine professional divide. The people who get to the same result using a fraction of the tokens will have a real and durable cost advantage over the people who do not, in the same way that early spreadsheet literacy separated people who could answer a question in an afternoon from people who needed a week.
This article covers what you are actually paying for, the billing mechanism almost everyone misunderstands, why a larger context window is not a larger budget, and a concrete set of habits that cut token spend without cutting output quality.
What you are actually paying for
A token is a chunk of text, usually a common word or a fragment of a longer one, and it is the unit models read and write in. As a working approximation, one token is about four characters of English, so a thousand tokens is roughly 750 words.
Every provider bills two separate meters. Input tokens are everything the model reads: your question, the system instructions, any documents you attached, and the entire conversation so far. Output tokens are what it writes back, and they are typically priced several times higher than input.
That split is where the first savings hide, and it is why "make the model briefer" is usually the highest-yield change available to a non-technical user. Asking for a five-line answer rather than an essay reduces the expensive meter directly.
Every turn re-sends the whole conversation
This is the mechanism most people never learn, and it explains almost every surprising invoice.
Language models are stateless. The model does not remember your last message. To continue a conversation, the software around it re-sends the entire history every single turn: your first question, its first answer, your second question, its second answer, and so on, right up to the new thing you just typed. What feels like an ongoing chat is actually a series of independent requests, each one carrying the whole transcript with it.
The billing consequence compounds. Take an illustrative conversation where each exchange adds about 1,500 tokens of question and answer:
| Turn | New text you typed | Input tokens actually billed |
|---|---|---|
| 1 | 500 tokens | ~500 |
| 5 | 500 tokens | ~6,500 |
| 10 | 500 tokens | ~14,000 |
| 20 | 500 tokens | ~29,000 |
By turn twenty you are paying to re-read your opening question for the twentieth time. The numbers above are arithmetic on a simplified example rather than a vendor benchmark, but the shape is exactly right, and it is the reason a long meandering chat can cost more than fifty short focused ones.
Two practical consequences follow immediately. Starting a fresh session for an unrelated task is not tidiness, it is cost control. And a chat that has drifted off topic is carrying dead weight that you are paying to transport on every subsequent message.
Most providers now offer prompt caching, which stores a repeated prefix so it does not have to be reprocessed from scratch, and both Anthropic and OpenAI document it. Caching softens the curve considerably for developers building on the APIs. It does not flatten it, and it does nothing for the second problem below.
A bigger context window is not a bigger budget
Providers advertise context windows in the hundreds of thousands or millions of tokens, and the natural reading is that a bigger window means you can simply put more in. The research says otherwise.
Chroma's July 2025 technical report, Context Rot, evaluated 18 leading models, closed and open, from Anthropic, OpenAI, Google and Alibaba. Its finding was blunt: "model performance degrades as input length increases, often in surprising and non-uniform ways". Not gracefully, and not predictably.
Anthropic's engineering team names the underlying resource in its guidance on effective context engineering for AI agents, describing context rot as the effect where "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases", and noting that models have an "attention budget" they draw on when parsing large volumes of context. The principle it lands on is worth memorising: find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome".
Matt Pocock has the most useful practitioner framing of the same phenomenon, in his AI Coding Dictionary. Early in a session the model is in a "smart zone", sharp and with good recall. As the session grows it drifts into a "dumb zone": "sloppier, forgetful, more mistakes". Same model, same setup, just more context. He puts the onset on frontier models at commonly around 125,000 to 150,000 tokens, and makes the point that matters most: "the zones don't track the context window limit. A session can be deep in the dumb zone with most of the window still free."
So a million-token window does not buy you a million tokens of good thinking. It buys you a larger space in which to degrade quietly, with no error message and no visible boundary. You pay more per turn and you get worse answers, which is the worst available combination and the reason most people's "the AI got dumber" complaint has nothing to do with the model being changed.
The misconception in one line
Stuffing everything into the prompt feels thorough and is actively counterproductive. Relevance beats volume. A model given three pages of the right material outperforms the same model given three hundred pages containing those same three pages.
The spreadsheet comparison
When spreadsheets arrived, the tool was available to everyone and the results were not. Two people with identical software produced wildly different work, because one understood how to structure a model and the other typed numbers into cells. That gap decided careers for twenty years.
The same split is opening now, and it will not sit where people expect. It will not be about who can produce a finished output with AI, because within a year or two almost everyone will be able to do that. It will be about the path taken to get there: how many attempts, how much wasted context, how many tokens burned on approaches that were never going to work.
Two people are given the same brief. One dumps the entire company handbook into a chat, asks a broad question, gets a mediocre answer, argues with it for fifteen turns, and finishes an hour later having spent forty times what the task warranted. The other spends four minutes deciding what the model actually needs to see, supplies exactly that, gets a usable answer in two turns, and moves on. Identical tool. Identical output quality. Wildly different cost, and the difference compounds across every task and every person in the business.
This skill now has a name. Context engineering is the deliberate practice of curating what goes into the model's context, and it is the successor to prompt engineering, which concerned itself mainly with the wording of the question. The wording matters far less than what surrounds it.
How to spend fewer tokens on the same result
These are the habits we actually use, in rough order of how much they save.
- 1.One task per session. The single highest-yield change, and free. Every unrelated thing you did earlier in a session is re-sent with every later message and eats into the smart zone. A new chat costs nothing to open.
- 2.Cap the output explicitly. Ask for "five bullet points, no preamble" rather than letting the model produce nine hundred words you will delete. Output tokens are the expensive meter.
- 3.Hand off rather than compact, when accuracy matters. Anthropic sets out three strategies for tasks that outgrow a context window: compaction, which summarises the history and restarts with the summary; structured note-taking, where the model writes notes to a file outside the context window and reads them back later; and sub-agents, where a focused task is delegated and only a condensed result returns. Compaction is convenient and silently loses detail. A written handover note is the one to reach for when the details matter, because you can read it and correct it before the next session starts.
- 4.Retrieve, do not dump. Pointing a model at the three relevant pages instead of the whole document set cuts cost and improves accuracy at the same time. This is why retrieval-augmented generation is the standard pattern for business assistants, and how the chatbots we build work under the hood.
- 5.Match the model to the job. Classifying an email does not need a frontier reasoning model. Providers publish small and large models at very different prices, and routing routine work to the cheap one is often the largest line-item saving available to a developer.
- 6.Watch for the dumb zone and restart. If the model forgets an instruction you gave twenty turns ago, or repeats a mistake it already corrected, do not re-explain. Re-explaining adds context and makes it worse. Start fresh with a clean brief.
Habits one, two and six require no technical skill at all and are available to every person in your business this afternoon.
The Australian angle
Cost discipline matters more here than in larger markets, because the businesses adopting AI are smaller. The National AI Centre's adoption data for December 2025 to February 2026 found 43% of Australian SMEs reported some level of AI adoption. For a business of that size an AI bill that quietly triples is not a rounding error on a departmental budget, it is a line item that gets the whole experiment cancelled by someone who never saw the mechanism that caused it.
We have watched this happen. A business trials AI, usage grows, the invoice grows faster than the value, and the conclusion drawn is "AI is too expensive for us" when the accurate conclusion was "nobody told us the whole conversation gets re-sent every message".
The Enki Approach
We treat token efficiency as a design constraint, not an afterthought. When we build an automation or an assistant, the question of what the model sees at each step is part of the architecture: retrieval scoped to the relevant documents, prompts that ask for structured output rather than prose, small models for classification and large models only where the reasoning genuinely requires them.
The same thinking runs through our AI education work, because the habits above are worth more to a business than any single tool. A team that instinctively starts a fresh session per task and asks for tight output will spend a fraction of what an untrained team spends, on the same software, for the same work.
Per-token prices will keep falling. That is a safe prediction, and it is also why the skill is durable rather than temporary: every price cut gets absorbed by more usage, so the advantage stays with whoever gets to the answer in fewer tokens. Cheap does not mean free, and it never has.