The Economics of a Token

At one request a day, a token costs nothing worth counting. At tens of billions a month, it is the line on the invoice that quietly decides whether there is a business behind the product at all.

And a request is never one token, or even one call. A single thing a customer asks for fans out inside the loop into a handful of model calls, sometimes a dozen, each one reasoning, reaching for a tool, reading the result, checking its own work. Every one of those calls re-reads the conversation from the top, because the model has no memory and the whole story gets handed back each time. You are not paying per message. You are paying per message, times every step it takes to answer, times every word you chose to carry along for the ride.

Put rough numbers on it. Say the question and its context are 2,000 tokens, and answering takes a dozen passes through the loop, each one re-reading everything before it and adding a couple hundred tokens of reasoning and tool output. Pass one reads 2,000. Pass twelve reads past 4,000, because it carries all eleven passes before it. Add them up and one reply the customer thinks of as a single message costs you north of 37,000 tokens read before the answer exists. They sent 2,000. You were billed for eighteen times that. That multiple, not the sticker price per token, is the number that decides whether there's a business.

One message, twelve laps, one invoice

Pass 12,000

Pass 22,200

Pass 32,400

Pass 42,600

Pass 52,800

Pass 63,000

Pass 73,200

Pass 83,400

Pass 93,600

Pass 103,800

Pass 114,000

Pass 124,200

Customer sent2,000

You were billed37,200 ≈ 19x

A token is rent, not a purchase

This is the part the pricing page hides. A token looks like something you buy: a price, a quantity, a line item. But the model forgets everything between turns, so the words you keep aren't bought once and owned. They are re-read on the next pass, and the one after that, billed again every time. You are not buying intelligence by the token. You are renting attention, and the rent comes due every loop.

Once you see it as rent, the question stops being "how cheap is a token" and becomes "what is this word earning, every single turn it stays in the window." Most words, asked that way, don't make rent.

Cheaper tokens made us spend more

The price of a token has been falling like a stone. At a fixed level of quality, inference has dropped about tenfold a year, a slide a16z named LLMflation. GPT-4-class capability cost around thirty-six dollars per million tokens at its March 2023 launch; two years later the same quality ran for cents, and Epoch AI found the slide accelerating rather than flattening, the price of matching GPT-4 on hard reasoning falling some fortyfold a year. The natural assumption is that the bill falls with it. It doesn't. Ours went up.

A cheaper token doesn't shrink what you spend; it lowers the bar for what you'll attempt. When a call is nearly free you add one more verification step, one more retry, one more pass of reasoning you'd never have paid for a year ago. The floor drops and the building gets taller. The savings the price cut promised get spent, instantly and invisibly, on ambition you didn't used to be able to afford.

There's a name for this, and it's older than computing. In 1865 William Stanley Jevons noticed that more efficient steam engines burned more coal, not less: efficiency made coal cheap enough to use for things nobody would have wasted it on before. Tokens are coal now. Every tenfold cut in price has been answered by more than a tenfold rise in use, which is why the people whose unit costs fell hardest are the same people watching their inference bills climb.

So the discipline matters more as tokens get cheaper, not less. The cheaper they are, the faster they accumulate when nobody is watching, and the easier it is to mistake a falling unit price for a falling bill.

A cheaper token never shrank the bill. It only raised the ceiling on what we'd try.

The cheapest token is the one you didn't send

If you can't win on price, you win on volume, by sending fewer. This is where memory stops being a feature and becomes a budget.

The memory work I've described elsewhere as a reliability move (keeping the recent turns whole, summarising the old ones, stamping a gap instead of replaying hours of dead history) reads, from the invoice, as pure budget. A full transcript carried turn after turn is a real cost buying an outcome you can get for a fraction of the tokens, so we stopped paying it. None of this is clever. It is just refusing to send words that aren't earning their rent.

And for the words you can't drop (the system prompt, the rules, the early turns that have to stay) there's a second lever the pricing page buries: prompt caching. The first time the model reads that unchanging prefix you pay full freight; every call after, a cache hit can cost as little as a tenth of a fresh read. Most of what a long conversation re-reads never actually changes, so the rent on it should fall to almost nothing. Skip caching and you pay new rent every loop on words you wrote once and never touched again: the closest thing to a free lunch in this business, and the easiest to leave on the table.

Price per token is the wrong number

The number on the pricing page is per token. The number that lands on your invoice is per finished task, and the two come apart faster than anyone expects. A model that is cheap per token but second-guesses itself, narrates, retries, and wanders can cost more to reach an answer than a pricier one that gets there in three clean turns. You don't buy tokens. You buy outcomes, and tokens are merely how you're billed for them.

The gap is narrower than the sticker swears, and sometimes it runs the other way. GPT-4o-mini lists about seventeen times cheaper per token than GPT-4o. But let the cheaper model second-guess itself across a dozen-plus calls, narrating and retrying, and by the time each has actually finished the same task, that seventeen-fold edge can shrink to nothing, and then invert. You were never paying per token. You were paying per answer.

	Cheap per tokenbut second-guesses itself	Dearer per tokenbut answers cleanly
Price per 1M output tokens	$0.60	$10.00
Model calls to finish the task	14	3
Output tokens generated	170,000	9,000
Cost of the finished task	$0.102	$0.090

This is also the real argument for a small model, and it isn't the one people usually make. The point of a small, cheap model, open weights a fraction the size of the famous ones, isn't only that each token costs less. It's that inside a tight loop with a clean context, it finishes in fewer tokens. Cheaper per token and fewer tokens to the answer: the saving multiplies. At our volume that per-token rent is also why the model runs on our own hardware instead of someone else's meter, because past a certain number of tokens a month, renting each one simply stops making sense.

What this is not is an argument for a drawer full of models. A drawer of models is a drawer of quirks, each with its own way of failing and its own place to drop context on the handoff. Routing the occasional hard turn to something larger is a footnote you add after the small model is already finishing the work, not the plan itself. The lever was never how many models you get to choose between. It was building one that finishes.

As a runbook, the levers go in this order, cheapest first, and you reach for the next only when the one before it is exhausted:

• Don't send it. Prune dead turns, summarise old ones, stamp a gap instead of replaying hours of history. The token you never send is the only one that's truly free.
• Cache what can't change. Any prefix reused across calls (the system prompt, the rules, the early turns) should be cached, where a hit can run a tenth of a fresh read. If most of your prompt is unchanging, this is the biggest number you're leaving on the table.
• Right-size the model. The smallest model that finishes the task in a clean loop, not the one that tops a leaderboard. Cheaper per token and fewer tokens to the answer, so the saving multiplies.
• Route the exceptions. Send the rare hard turn to something larger after the small model already carries the load, never before.
• Own the meter. Once volume is real and steady, past a certain number of tokens a month, self-hosting the small model beats renting every token off someone else's meter.

I should say where this discipline turns into a vice, because I've been on the wrong side of it. Token-counting is a late-stage virtue. Early on, while you're still finding out what the product even is, the rent on a few thousand tokens is a rounding error against the cost of building the wrong thing, and shaving it is just polishing a draft you're about to throw away. The invoice only earns its authority once the thing is right and the volume is real. Optimise the bill before then and you've won an argument that wasn't worth having.

Reliability and cost are the same line

This is the part that surprised me. I expected cost work and reliability work to pull against each other, the cheap thing being the flimsy thing. At scale they turned out to be the same job. The messy context that blew the budget was the same messy context that made the model lose the thread. Every token we pruned for the invoice was a token that was no longer there to distract.

So we stopped treating the bill as something to apologise for and started reading it as a diagnostic. A conversation that costs too much is usually a conversation the harness is handling badly. The invoice was telling us where the system was sloppy, in the one language nobody on the team could argue with.

The model sets what a token can do. The harness sets how many you spend to get there. Only one of those shows up on the invoice every morning.