Lesson 4.2
Caching by meaning
Two questions can mean the same thing in different words. A semantic cache matches meaning, not bytes.
The notebook misses questions it should recognize.
Last lesson the gatekeeper monk matched questions letter for letter. That works for true repeats, but it is brittle. A traveler asks "What's the capital of France?" One minute later another asks "capital city of France?" Same question, same answer, yet the exact-match notebook treats them as strangers and pays the provider all over again.
Real users phrase things every way imaginable: different word order, synonyms, a typo, extra politeness. Exact-match catches none of these near-duplicates. To reuse an answer for a question that means the same thing, the monk needs to compare meaning instead of spelling.
Exact-match caching only fires on identical bytes, so it misses every near-duplicate. Most repeated questions in the wild are reworded, not retyped.
Turn each question into a point that captures its meaning.
The trick is to stop comparing text and start comparing meaning. A special model, an embedding model, reads a piece of text and returns an embedding: a list of numbers (a vector) that stands for what the text means. The useful property is that text with similar meaning gets similar numbers, even when the words are completely different.
Think of it as placing every question as a point on a map. "What's the capital of France?" and "capital city of France?" land almost on top of each other, because they mean nearly the same thing. "How do I reset my password?" lands far away in another part of the map. The map's geography is meaning: close together means similar, far apart means different.
An embedding turns text into a vector where similar meaning lands in nearby points. That lets you compare two questions by how close their points are, not by their spelling.
Match the new question against the meanings you have seen.
A semantic cache stores, for each past question, its embedding alongside the answer. When a new request arrives, the gateway embeds it too and asks: is any stored question's point close enough to this new one? It needs three pieces to answer that:
- An embedding model to turn each question into a point.
- A similarity measure: a number that says how close two points are. A common one is cosine similarity, which runs from 0 (unrelated) to 1 (identical meaning).
- A threshold: how close counts as "the same question." Score above it, hit. Below it, miss.
So a hit is no longer "the bytes match." It is "the meaning is close enough," where the threshold draws the line on "enough."
A hit, worked by hand.
Suppose the cache already holds the question "What's the capital of France?" with its answer. A new request comes in: "capital city of France?". Embed the new question, compare it to the stored one, read off the similarity:
The two phrasings scored 0.93, which clears the 0.90 threshold, so the gateway returns the cached answer without calling the provider. Exact-match would have missed this completely. That is the whole value of semantic caching: it recognizes the same question dressed in different words.
The threshold is a safety dial, and it cuts both ways.
The threshold decides how generous the cache is, and getting it wrong is dangerous in a way exact-match never was. Exact-match can only ever return a right answer or no answer. A semantic cache can return a wrong answer if the threshold is too loose.
Set the threshold too low (say 0.60) and questions that are merely related get treated as the same. "What's the capital of France?" and "What's the population of France?" might score 0.7: clearly different questions, but a loose cache would hand back the capital when someone asked for the population. Set the threshold too high (say 0.99) and you go back to almost exact-match, catching very few rewordings.
So you tune the threshold to balance two failures: too loose risks confidently wrong answers, too tight wastes the whole point by missing real duplicates. Most systems start strict and loosen carefully while watching for bad hits.
Embed the meaning, compare to past questions, reuse when close enough. The threshold is the safety dial: too loose returns wrong answers, too tight misses real duplicates.
Work it out
Your threshold is 0.90. A new question scores 0.88 against the closest stored question. Hit or miss? Now your boss complains the cache misses too many obvious rewordings. You drop the threshold to 0.80. What new risk did you just take on? (Answers: 0.88 is a miss, just under the line. Dropping to 0.80 risks returning a stored answer for a question that only looks similar, a confidently wrong hit.)
Key takeaways
Semantic caching compares meaning, not bytes, so it can reuse an answer for a question worded differently.
It needs an embedding model to place questions as points, a similarity measure, and a threshold for "close enough."
The threshold is a safety dial: too loose returns wrong answers, too tight misses duplicates. Next we see how the cache actually finds the closest point.