How an LLM gateway works

What a gateway is, and why one door in front of many models.

1. 0.1 What a gateway is
2. 0.2 A request's journey
3. 0.3 Why a gateway at all

Forwarding a request, and what "the same API" really means.

1. 1.1 What a proxy does
2. 1.2 One API, many providers
3. 1.3 Streaming the answer

Retries, backoff and jitter, timeouts, and idempotency.

1. 2.1 Ways a call fails
2. 2.2 Idempotency: safe to retry?
3. 2.3 Backoff and jitter
4. 2.4 Timeouts and deadlines

Routing policy: cost vs latency vs quality, and fallback.

1. 3.1 Many providers, one choice
2. 3.2 Cost, latency, quality
3. 3.3 Fallback when a door is closed

Exact vs semantic caching, embeddings, nearest-neighbor lookup.

1. 4.1 Exact-match caching
2. 4.2 Caching by meaning
3. 4.3 Nearest-neighbor lookup

Token accounting, cost, rate limits, and per-key budgets.

1. 5.1 Counting tokens and cost
2. 5.2 Rate limiting
3. 5.3 Budgets and fairness