Lesson 5.2
Rate limiting
Cap how fast requests flow, so one caller cannot overwhelm the gateway or a provider. The token bucket.
A bucket of permits that fills back up.
Counting cost (last lesson) tells you what happened. Rate limiting decides what is allowed to happen next: a cap on how fast requests may flow. One caller stuck in a loop should not be able to flood the gateway or hammer a provider into rejecting everyone. So the gatekeeper holds the pace.
The classic way to do this is the token bucket. Picture a bucket that holds a fixed number of permits. It refills at a steady rate, a few permits per second, up to its size. Each request takes one permit out. If the bucket has a permit, the request goes through. If the bucket is empty, the request is turned away (rejected) or made to wait (queued).
Name clash, read carefully. Here a "token" means a permit to make a request. It is NOT the text token from lesson 5.1 (the chunk of words you pay for). Same word, different thing. This lesson says "permit" throughout to keep them apart.
Drag the refill rate, then spend permits.
Drag the knob to set how many permits arrive per second. Hit "Request" to spend one. Drain the bucket faster than it fills and you will hit empty, and the next request flashes red: rejected. Slow down to the refill rate and requests keep flowing.
Drag the knob to set the refill rate, then send requests
These permits are not the text tokens from lesson 5.1. One permit = one allowed request.
permits: 5.0 / 10 · refill: 2/sec
The bucket size sets how big a sudden burst you tolerate; the refill rate sets the pace you settle into once the burst is spent.
Burst now, steady after.
Take a bucket of size 10 that refills at 2 permits per second, and say it starts full. Watch what a caller can do.
So a short burst of 10 sails through (that is the bucket's whole stored-up supply), but the long-run rate is pinned to the refill: roughly 2 per second, forever. The bucket buys you slack for spikes without ever letting the average run away.
Why not just a hard wall?
You could instead say "no more than N per second, ever," a flat ceiling. The trouble is real traffic is bursty. A page load might fire six requests in the same instant, all legitimate. A hard wall rejects the honest spike along with the abuse.
The token bucket is gentler. It lets a burst through up to the bucket size, absorbing the spike, then quietly settles everyone to the refill rate. Bursts are fine; sustained flooding is not. That is the shape you usually want.
A token bucket allows bursts up to its size, then settles to the refill rate. It bounds long-run throughput without a hard wall, so honest spikes pass while sustained floods are held back.
Try it yourself
A bucket of size 5 refills at 1 permit per second and starts full. A caller fires 5 requests instantly, then keeps firing 1 request every second. How many of the next 5 (one per second) get through? Now what changes if the bucket size were 1 instead of 5, with the same refill?
Key takeaways
Rate limiting caps how fast requests flow so one caller cannot overwhelm the gateway or a provider. The token bucket is the standard way to do it.
A bucket holds permits, refills at a steady rate, and each request spends one. Empty bucket means the request is rejected or queued. (These permits are not the text tokens of lesson 5.1.)
Size 10 refilling at 2 per second lets a burst of 10 through, then limits you to about 2 per second. Bursts allowed up to the size; the long-run rate is the refill. Next: per-key budgets.