Many providers, one choice

Three roads, all going to the same town.

A traveler reaches the gate and needs to get to the next town. There are three roads. One is short but full of potholes. One is smooth but long. One is paid, fast, and well kept. All three arrive. The gatekeeper has to point down exactly one of them. He cannot say "take all three."

That choice is routing: when more than one provider can serve a request, the gateway decides which one actually gets it. Several providers can answer a chat request equally well. Only one of them will. Routing is the act of choosing among the ones that can.

The important word is can. Routing only kicks in when you have real options. If only one provider supports the model the client asked for, there is nothing to route; the request goes there. Routing matters the moment two or more doors could each handle the same traveler.

The rule behind the pick is a policy.

The gateway does not choose on a hunch. It follows a stated rule, the same one every time, so the choice is predictable and you can reason about it. That rule is the routing policy. A few common ones:

To follow any of these, the gateway needs facts about each provider: its price per request, its observed latency (how fast it has actually been answering lately), and its capability (does it even support the model and features this request needs). Without those, a policy is just a wish.

Fixed order, or read the road as you go.

Policies come in two flavors. A static policy is a fixed preference order decided ahead of time: always try provider A, then B, then C. It never changes mid-flight. Simple, predictable, and it does not need live measurements. The downside is that it ignores what is happening right now. If A is having a bad minute, a static policy keeps sending to A anyway.

A dynamic policy decides per request, using live signals: which provider is currently healthy, which has been fastest in the last few minutes, which is not rate-limited at the moment. It reacts to the road as it is, not as it was when you wrote the config. The cost is that it needs that live data to be trustworthy, and it is harder to predict and debug.

Neither is "better." A static order is a fine default. You reach for a dynamic policy when providers actually vary minute to minute and you want the gateway to notice.

Same three providers, two policies, two winners.

Say three providers can all answer this request. Here is what the gateway knows about each, for this call. Cost is the price of the request; latency is how long it has been taking lately.

Run the cheapest policy: scan the cost column, pick the smallest. That is A at $0.002, even though A is the slowest of the three.

Now run the fastest policy on the very same table: scan the latency column, pick the smallest. That is B at 300 ms, even though B costs three times what A does.

Same providers, same numbers, same request. The only thing that changed was the policy, and the winner flipped from A to B. The pick was never about the providers alone. It was about what you told the gateway to optimize.

Key takeaways