Lesson 5.1
Counting tokens and cost
Providers bill by the token. To track cost, the gateway counts the tokens going in and the tokens coming out.
Text is sold in chunks, not words.
A taxi charges by the mile, not by the trip. Models charge by the token: a small chunk of text, roughly four characters, about three quarters of a word. "Gatekeeper" might be one token or two; a long rare word splits into several. You do not have to count them by hand, but you do have to know they are the unit on the meter.
Every request has two piles of tokens. The prompt tokens are everything you send in: the system message, the history, the user's question. The completion tokens are what the model writes back. Both are counted, and here is the catch worth remembering: they are priced separately, and the output side usually costs more than the input side.
A token is a chunk of text, about four characters. Each request has prompt tokens (what you send) and completion tokens (what comes back), and the two are billed at different rates.
The cost of one request.
Once you know how many tokens went each way, the bill is just two multiplications added together. Prices are quoted per thousand tokens, so each side is its token count times its rate.
prices are per 1000 tokens, and the two sides use different rates
That is the whole idea. The hard part is not the arithmetic; it is making sure the counts are honest. Which is why the gateway does not guess them.
Work one out by hand.
Say a request used 1000 prompt tokens and 500 completion tokens. Take an input rate of $0.50 per 1000 tokens and an output rate of $1.50 per 1000 tokens. These numbers are illustrative, not a real quote; treat them as round figures to follow the math.
Notice the 500 output tokens cost more than the 1000 input tokens. Half the count, but a higher rate, so the output side carries more of the bill. That asymmetry is why output-heavy work (long answers, generated code) gets expensive faster than it looks.
The gateway reads the count, it does not invent it.
When the provider replies, it includes a small usage block alongside the answer: how many prompt tokens it charged you for, and how many completion tokens it produced. The gateway reads those two numbers straight from the response, applies the rates, and writes the result to its ledger: a running record of who called, what it cost, and when.
Using the provider's own count matters. If the gateway guessed token counts on its own, its numbers would drift from the actual bill. Reading the usage block keeps the ledger matched to what you are really charged. That ledger is the foundation for everything in this unit: limit how fast requests flow (next lesson) and cap how much each key can spend (the one after).
Cost is tokens times price, counted per request. The provider returns a usage block; the gateway reads it, applies the rates, and records the cost in its ledger. It reads the count, it never guesses it.
Try it yourself
A request used 2000 prompt tokens and 1000 completion tokens, at the same rates ($0.50 in, $1.50 out per 1000). What does it cost? Then ask: if you cut the prompt in half but the answer stayed the same length, how much would you save, and why is the saving smaller than you might hope?
Key takeaways
Models bill by the token, a chunk of about four characters. Each request has prompt tokens (input) and completion tokens (output), priced separately, with output usually the dearer of the two.
Cost = prompt_tokens times input_price plus completion_tokens times output_price. For 1000 in and 500 out at $0.50 and $1.50 per 1000, that is $0.50 + $0.75 = $1.25.
The provider returns a usage block; the gateway reads those counts, applies the rates, and writes the cost to its ledger. That ledger feeds rate limiting and budgets, the next two lessons.