Glossary

Every term the course uses, in plain English first and the precise name second. Grouped by the unit it belongs to.

RAG (retrieval-augmented generation)

Answering with a model that reads sources you hand it, instead of relying on memory. Retrieve the relevant passages first, then generate the answer from them.

Grounding

Tying the answer to specific retrieved passages, so it reflects a source rather than what the model recalls from memory.

Hallucination

A fluent, confident answer that is not supported by any source. What a pure-memory model produces when it has no real answer available.

Knowledge cutoff

The date the training data ends. Anything after it is absent from the weights, so the model can only guess about it.

Embedding

A list of numbers (a vector) that places a piece of text in a space where nearby means similar in meaning.

Vector

The list of numbers itself. A point in a high-dimensional space.

Cosine similarity

A closeness score read as the angle between two vectors: 1 is the same direction (very similar), 0 is unrelated, negative is opposite.

Chunking

Splitting a document into smaller passages so each can be embedded and retrieved on its own. Chunk size and overlap trade precision against context.

Vector store

A store of (vector, passage) pairs. You insert passages at ingest and ask it for the ones nearest a query.

Nearest-neighbor search

Finding the stored vectors closest to the query vector. Exact (brute force) compares against all of them; approximate trades a little accuracy for speed.

ANN (approximate nearest neighbor)

A faster search that returns almost-nearest results, used when comparing against every vector is too slow.

Top-k

The k nearest passages the search returns, for a chosen number k.

Recall@k

Of all the truly relevant passages, the fraction that landed in the top k. The main score for how good retrieval is.

Hybrid retrieval

Combining keyword search (good at exact terms) with vector search (good at meaning), because each catches what the other misses.

Token

A small chunk of text the model reads, roughly a word or word-piece. Both the prompt and the answer are measured in tokens.

Context window

The maximum number of tokens a model can take in at once. The hard limit on how much you can put in the prompt.

Context budget

The room left for passages after the question and the expected answer are accounted for, within the context window.

Lost in the middle

The tendency of a model to attend to the start and end of a long prompt more than the middle, so placement of a key passage matters.

Citation

A pointer in the answer back to the passage it came from, so a reader can check it.

Faithfulness

Whether the answer actually follows from the provided context. Distinct from correctness: an answer can be true yet unfaithful, or faithful yet wrong about the world.

Streaming

Sending the answer one token at a time as it is generated, so the reader sees words appear instead of waiting for the whole reply.

Time-to-first-token

How long until the first word appears. Streaming lowers this even when the total time is unchanged.

Bi-encoder

The fast scout. It embeds the query and each passage separately, so passage vectors can be precomputed and scored cheaply, at the cost of accuracy.

Cross-encoder

The careful judge. It runs the query and a passage through the model together, which is far more accurate and far more expensive, so it only runs on a few candidates.

Reranking

A second pass that re-scores the top candidates from search with a more accurate model, to push the best passages to the top.

Retrieve-then-rerank

The two-stage pattern: cast a wide net cheaply with search, then rerank the top of it carefully. More quality, more latency.

Latency

How long a request takes. Measured across the whole pipeline and per hop.

p50 and p99

Percentiles of latency: p50 is the typical request, p99 is the slow one-in-a-hundred (the tail). Users feel the tail.

Latency budget

A ceiling you hold the p99 under. Because the hops run in sequence, the total is their sum and the slowest hop dominates.

Embedding cache

A store of already-computed embeddings keyed by the text, so repeated queries and passages skip the embedding step. Measured by hit rate.

Hit rate

The fraction of lookups the cache can answer without recomputing. Higher hit rate means more latency and cost saved.