Glossary

Adapter

The piece that translates the gateway’s one request shape into whatever fields and endpoint a given provider expects.

A per-provider translator: it maps the gateway’s unified request and response onto a specific provider’s API contract.

Backoff (exponential)

Wait longer between retries, doubling the wait each time, so you stop hammering a provider that is already struggling.

Exponential backoff: the delay before attempt n grows geometrically (for example base times 2 to the power of the attempt number).

Budget

A spending ceiling tied to an API key. Cross it and further calls get rejected or slowed down.

A per-key cost cap, usually over a window; once exceeded, requests are rejected (HTTP 402/429) or throttled.

Cache

A store of past answers, so a repeat request is served instantly instead of going to the provider again.

A keyed store of prior responses; a lookup returns a stored result rather than re-invoking the model.

Cache hit / miss

A hit means the answer was already stored. A miss means it was not, so the request goes to the provider.

Hit ratio (hits over total lookups) is the headline metric of cache effectiveness.

Completion tokens

The output tokens the model generated. These usually cost more than the ones you sent in.

Output tokens billed at the completion (output) rate, typically higher than the prompt rate.

Cost

Tokens times price, added up per request and written into the ledger.

Per request: prompt tokens times input price plus completion tokens times output price, summed and recorded.

Deadline

The total time budget for a whole operation, counting every retry inside it.

An absolute end-time for an operation across all attempts, distinct from a single attempt’s timeout.

Embedding

A list of numbers that captures the meaning of a piece of text, so similar meanings end up close together.

A dense vector produced by an embedding model; semantic similarity maps to small distance in that space.

Fallback (failover)

When the chosen provider fails, automatically try the next one in your ranked list.

Failover: on error, route to the next candidate provider in priority order until one succeeds or the list runs out.

Gateway (LLM gateway)

One API endpoint sitting in front of many model providers. Your app only ever talks to it.

An LLM gateway: a single unified API that fronts multiple providers and handles routing, retries, caching, and accounting.

Idempotency

Doing an operation twice has the same effect as doing it once, which is what makes a retry safe.

A property where repeated identical requests produce the same end state as a single request.

Idempotency key

An id the client sends so the server can spot a duplicate and return the first result instead of redoing the work.

A client-supplied unique token; the server stores the first result against it and replays it for any repeat.

Jitter

Randomness added to retry waits so a crowd of clients does not retry at the exact same instant.

Random perturbation of backoff delays that de-synchronizes retries and avoids the thundering herd.

Nearest-neighbor search

Finding the stored point closest to a query point in vector space. It is the core of semantic caching and vector search.

Given a query vector, return the stored vectors with smallest distance; the engine behind semantic cache and vector search.

Prompt tokens

The input tokens you send: the messages and the system prompt.

Input tokens billed at the prompt (input) rate, typically lower than the completion rate.

Provider

A company or service that runs the actual model, like OpenAI, Anthropic, Google, or a local model.

An upstream model service reachable through its own API; the gateway speaks to it via an adapter.

Proxy (reverse proxy)

A server that takes a request and forwards it to another server on your behalf, then relays the response back.

A reverse proxy: it receives client requests, forwards them to upstream servers, and returns the upstream response.

Rate limiting

Capping how many requests are allowed in a given slice of time.

Enforcing a maximum request rate per window per key or client, rejecting or queueing the overflow.

Retry

Trying the same failed call again, for failures that might succeed on a second attempt.

Re-issuing a request after a transient failure (429, 5xx, timeouts); only safe for idempotent operations.

Routing

Choosing which provider serves a request when more than one of them could.

The decision step that selects a provider per request from the eligible set.

Routing policy

The rule that decides the choice: cheapest, fastest, highest quality, round-robin, or weighted.

A configurable strategy mapping a request to a provider by cost, latency, quality, round-robin, or weights.

Semantic cache

A cache that matches questions by meaning, not by exact bytes, using embeddings and a closeness cutoff.

A cache keyed on embedding similarity above a threshold rather than exact string equality.

Similarity threshold

How close two meanings must be to count as the same cached question. It is the safety dial of a semantic cache.

The minimum similarity (or maximum distance) for a semantic cache hit; too low risks wrong answers, too high lowers hit rate.

Streaming (SSE)

Sending the answer token by token as it is generated, so text starts appearing right away.

Server-Sent Events: a long-lived response that emits incremental chunks as the model produces them.

Time-to-first-token (TTFT)

How long until the first token of the answer shows up. Streaming makes this shorter.

TTFT: the latency from request to the first streamed token, distinct from total generation time.

Timeout

The longest you are willing to wait on a single attempt before giving up on it.

A per-attempt time limit; distinct from a deadline, which spans every retry.

Token

A chunk of text, roughly four characters. Models read and bill in tokens.

The unit of text a model processes and prices on; not a whole word, but a sub-word piece.

Token bucket

A way to rate-limit: a bucket of permits refilled at a steady rate, one permit per request, allowing short bursts up to its size. Here "tokens" means permits, not text tokens.

A rate-limit algorithm: permits accrue at a fixed rate up to a capacity; each request spends one, so bursts are bounded by the bucket size.

TTL (time to live)

How long a cached answer stays valid before it expires.

Time to live: the lifespan of a cache entry, after which it is treated as stale and refetched.

Usage

The provider’s report of how many prompt and completion tokens a request actually used.

The usage object a provider returns (prompt, completion, and total token counts) that the ledger records.

Vector

An ordered list of numbers. An embedding is a vector; the distance between two vectors measures how different two meanings are.

A point in n-dimensional space; cosine or Euclidean distance between vectors quantifies semantic difference.