Exact-match caching

The monk has seen this traveler before.

Picture the gatekeeper monk with a small notebook. When a traveler asks a question and he works out the answer, he writes both down. The next time someone walks up and asks the exact same thing, he does not think it through again. He flips to that line in the notebook and reads the answer straight back. Instant, and it cost him nothing.

That notebook is a cache (a store of past answers you can reuse). For an LLM gateway it matters more than for most systems, because every call to a provider costs real money and real time. A model can take seconds to answer and bills you per token. If your app sends the same request again, paying for it twice is pure waste.

The key is the whole request, squeezed into one fingerprint.

To look something up in the notebook, the monk needs a way to find the right line fast. A cache does this with a key: a short label that stands for the request. The simplest scheme is exact-match. Take the full request (the model name, the messages, and the settings like temperature and max tokens), and run it through a hash (a function that turns any input into one fixed-size fingerprint). Identical input gives an identical fingerprint every time.

That fingerprint is the key. The stored answer is the value. When a new request arrives, you hash it the same way and check: is this key already in the store?

• Hit: the key is there. Return the stored answer, skip the provider entirely.
• Miss: the key is new. Call the provider, get the answer, then store it under this key for next time.

One detail decides whether two requests are "the same": you must hash everything that changes the answer. Same messages but a different model is a different request. Same messages but temperature 0.0 versus 0.9 is a different request. Leave a field out of the key and you risk returning an answer that was generated under different settings.

Answers should not live forever.

A notebook entry from last year might be stale. Maybe the question was "what is the latest version of the library?" and the answer has moved on. So each cached answer gets a TTL (time to live: how long the entry stays valid). Store the answer with a stamp that says "good until 3pm." After that, the entry is treated as gone. The next matching request is a miss, goes to the provider, and refreshes the entry.

The TTL is a dial you tune. A long TTL saves more money and time, because answers get reused for longer. A short TTL keeps answers fresher, at the cost of more provider calls. Pick it based on how fast the right answer goes stale for your use case.

Two calls, worked by hand.

Your app sends the same request twice in a row. Same model, same single message, same settings. Walk the two calls through the cache:

The second call is the whole point: the answer was already in the notebook, so it came back instantly and cost nothing. Multiply that across a busy app where the same handful of prompts repeat all day, and the cache pays for itself many times over.

It only catches the exact same question.

Here is the catch built into the word "exact." The key is a fingerprint of the literal bytes. Change a single character and the fingerprint changes completely, so it is a different key, so it is a miss. To the cache, these are three unrelated questions:

You and I read all three as the same question. The hash does not. Extra spaces, a dropped question mark, a lowercase letter: each one produces a different key and a fresh provider call. Exact-match caching catches only true repeats, byte for byte.

That is the opening for the next lesson. To catch questions that mean the same thing in different words, you need a cache that compares meaning, not bytes (semantic caching).

Key takeaways