The Embedding Cache

A sticky note with the answer already on it.

Imagine someone keeps asking you the same arithmetic, "what is 47 times 12?" The first time you work it out. After that you scribble the answer on a sticky note and just read it off. You did the hard part once; every repeat is a glance.

Turning text into a vector (embedding it) is the arithmetic here. It is real work: the request goes to a model, the model runs, you wait and you pay. A cache is the sticky note. It is a small store that remembers "this exact text became this exact vector," so the next time you see that text you read the vector off instead of recomputing it.

Keyed by the text, checked before the work.

The cache is keyed by the text itself. Before you call the embedding model, you ask the cache: "have I embedded these exact words before?" If yes, that is a hit, and you take the stored vector and skip the model call entirely. If no, that is a miss, you do the work, and you write the result into the cache so the next identical request is a hit.

The key has to be exact. "reset my password" and "Reset My Password" are different keys unless you normalize the text first (lowercase it, trim the spaces). How aggressively you normalize is a trade: looser keys catch more hits but risk treating two genuinely different questions as one.

Two kinds of text, two timings.

There are two places embedding happens, and they cache differently.

The passages in your knowledge base are embedded once, at ingest, well before any user shows up. You chunk the documents, embed every chunk, and store the vectors in the index. That work never repeats per query. In a real sense the index is already a permanent passage cache: you do not re-embed a document just because someone asked a question.

The queries are different. They arrive live, and you have to embed each one before you can search. That is the embedding call sitting on the user's clock, inside the latency budget from the last lesson. The good news is that real queries are not random. A support bot hears "how do I reset my password?" and "where is my invoice?" over and over. Those hot queries repeat, so a query cache earns its keep fast.

A worked example.

Suppose, for illustration, embedding a query takes 90ms and 40 percent of your incoming queries are repeats of something asked recently (a 40 percent hit rate). On a hit you pay almost nothing, call it under a millisecond to read from memory. So the embedding stage costs 90ms on a miss and roughly 0ms on a hit.

Average that across 100 queries: 60 misses at 90ms and 40 hits at 0ms is 5400ms total, or about 54ms per query instead of 90ms. You shaved a third off the embedding stage on average, and you made fewer model calls, which is fewer dollars too. Push the hit rate up (more traffic, hotter queries) and the saving climbs with it.

You cannot keep everything.

A cache lives in limited memory, so it cannot hold every query forever. When it fills, it has to drop something to make room, which is called eviction. The common rule is to drop whatever was used least recently (the idea being that a query nobody has asked in a while is unlikely to come back soon). That keeps the hot, repeating queries in the cache and lets the one-off oddities fall out. You do not need to build a clever policy to start; "keep the recent ones, drop the stale ones" gets most of the benefit.

Key takeaways