Exercise 03 of 4/Foundations/1 hour
A rate limiter for a payment gateway
Your bank now talks to an upstream payment gateway that allows 100 requests per second and starts charging you for breaches. You need a limiter that is correct when 500 goroutines hit it simultaneously, and that does not spend a goroutine per caller.
The bank works. Now it settles through an upstream gateway with a hard contract: 100 requests per second, and every breach is both a rejected payment and a line item on an invoice.
You cannot retry your way out of this, because retries are exactly what pushes you over.
This is the first exercise where the shared state is not a number. It is time — and a counter can be guarded where a clock cannot be locked.
Write the obvious limiter
Do this
Write a Limiter with one method, allow(), returning true at most rate times per second no matter how many threads call it.
The shape people reach for first:
public final class Limiter {
private final int rate;
private int count;
private long windowStart = System.nanoTime();
public synchronized boolean allow() {
long now = System.nanoTime();
if (now - windowStart > 1_000_000_000L) { // one second
count = 0;
windowStart = now;
}
if (count < rate) { count++; return true; }
return false;
}
}type Limiter struct {
mu sync.Mutex
count int
rate int
windowAt time.Time
}
func (l *Limiter) Allow() bool {
l.mu.Lock()
defer l.mu.Unlock()
if time.Since(l.windowAt) > time.Second {
l.count = 0
l.windowAt = time.Now()
}
if l.count < l.rate {
l.count++
return true
}
return false
}class Limiter:
def __init__(self, rate: int) -> None:
self._rate = rate
self._count = 0
self._window_at = time.monotonic()
self._lock = threading.Lock()
def allow(self) -> bool:
with self._lock:
now = time.monotonic()
if now - self._window_at > 1.0:
self._count = 0
self._window_at = now
if self._count < self._rate:
self._count += 1
return True
return FalseIt is properly synchronised and has no data race. Is it correct?
AnswerCommit to one first
No — and notice that the bug is not a concurrency bug at all. The locking is fine. The definition of "per second" is wrong.
Find the two timestamps that break it
rate is 100. Find two moments, milliseconds apart, where 200 requests get through.
AnswerCommit to one first
12:00:00.999 and 12:00:01.001.
A hundred calls land just before the boundary and fill that window. A hundred more land just after and fall into a fresh one. Both bursts are allowed, and in the two milliseconds spanning them 200 requests went out.
Set rate to 100. Let 100 calls land at 12:00:00.999, and 100 more at 12:00:01.001. Both bursts are allowed: the first fills the window ending at 12:00:01, the second falls into a fresh one. In the 2-millisecond interval spanning them, 200 requests went out. The gateway does not care about your window boundaries — it measures any rolling second, and it just invoiced you.
This is the "boundary burst", and it is the reason the fixed-window counter is the wrong primitive. The number you are enforcing is not "calls per window" but "calls per second, measured from any instant".
Build the real thing
Do this
Under 500 concurrent threads for three seconds, no rolling one-second interval may contain more than rate allowed calls. No thread per caller, no background thread per limiter, and a limiter must be collectable without being closed.
Hint 1Stop counting calls, start accruing permission
Rather than counting what has happened since a boundary, track how much allowance has built up since the last call. Allowance accrues at rate per second and is capped at some burst size. A call costs one unit.
Hint 2You do not need a ticker
The tempting implementation starts a goroutine with a time.Ticker that refills a buffered channel. It works, and it costs a goroutine and a timer per limiter, and it leaks both unless every caller remembers to Close(). You can get the same behaviour by computing the accrual lazily at call time from a single stored timestamp — no background work at all.
Hint 3Do not sleep holding the lock
For Wait, the natural code locks, discovers it has to wait, and sleeps. Every other caller is now blocked on the mutex behind a sleeping goroutine, and they all wake to find the token gone. Work out when your turn is while holding the lock; sleep after releasing it.
Solution — Java: nanoTime, and the clock that goes backwardsTry it first
public final class Limiter {
private final ReentrantLock lock = new ReentrantLock();
private final Condition ready = lock.newCondition();
private final double ratePerNano, burst;
private double tokens;
private long last; // System.nanoTime()
public Limiter(double perSecond, double burst) {
this.ratePerNano = perSecond / 1_000_000_000d;
this.burst = burst;
this.tokens = burst;
this.last = System.nanoTime();
}
private void advance(long now) { // caller holds lock
long elapsed = now - last; // subtraction handles wraparound
if (elapsed <= 0) return;
tokens = Math.min(burst, tokens + elapsed * ratePerNano);
last = now;
}
public boolean tryAcquire() {
lock.lock();
try {
advance(System.nanoTime());
if (tokens < 1) return false;
tokens -= 1;
return true;
} finally { lock.unlock(); }
}
public void acquire() throws InterruptedException {
long waitNanos;
lock.lock();
try {
advance(System.nanoTime());
tokens -= 1; // reserve, may go negative
if (tokens >= 0) return;
waitNanos = (long) (-tokens / ratePerNano);
} finally { lock.unlock(); }
try {
TimeUnit.NANOSECONDS.sleep(waitNanos);
} catch (InterruptedException e) {
lock.lock();
try { tokens += 1; } finally { lock.unlock(); } // hand it back
throw e;
}
}
}Use System.nanoTime(), never System.currentTimeMillis(). This is the single most important line in the Java version. currentTimeMillis is wall-clock: NTP can step it, an operator can set it, and a daylight-saving change can move it. A limiter built on it will, the moment the clock jumps backwards, compute a negative elapsed time and — without the guard — remove tokens and stall every caller; on a forward jump it will grant a burst it never earned. nanoTime is monotonic and has no meaning as an absolute time, which is exactly what you want. Go hides this decision from you by putting a monotonic reading inside every time.Time; Java makes you choose, and the wrong choice is the more familiar method name.
Compare nanoTime values by subtracting, never by <. The value can overflow long; the documented contract is that the difference is valid. now - last > 0 is correct across a wraparound and now > last is not. This is why the guard above reads elapsed <= 0.
Condition.awaitNanos versus sleeping. The version above releases the lock and sleeps, which is correct and simple. If you want callers served in strict arrival order, replace the sleep with ready.awaitNanos(...) in a loop and signal on release: ReentrantLock(true) plus a condition gives you FIFO fairness that a bare sleep does not. You pay for it in throughput — fair locks are materially slower under contention.
In production, use Guava's RateLimiter. It is this algorithm, with the warm-up and fairness cases handled. Bucket4j is the richer option if you need distributed limits. Write it once so you know what they are doing.
Solution — Go: a lazy token bucketTry it first
No goroutine, no ticker, one timestamp.
type Limiter struct {
mu sync.Mutex
tokens float64
burst float64
rate float64 // tokens per second
last time.Time // when tokens was last brought up to date
}
func NewLimiter(rate, burst int) *Limiter {
return &Limiter{
tokens: float64(burst),
burst: float64(burst),
rate: float64(rate),
last: time.Now(),
}
}
// advance brings the bucket up to `now`. Caller must hold l.mu.
func (l *Limiter) advance(now time.Time) {
elapsed := now.Sub(l.last)
if elapsed <= 0 {
return // a non-monotonic clock must never remove tokens
}
l.tokens += elapsed.Seconds() * l.rate
if l.tokens > l.burst {
l.tokens = l.burst
}
l.last = now
}
func (l *Limiter) Allow() bool {
l.mu.Lock()
defer l.mu.Unlock()
l.advance(time.Now())
if l.tokens < 1 {
return false
}
l.tokens--
return true
}The whole idea is in advance: permission is not counted, it is accrued. Because accrual is computed from a real elapsed duration rather than a window boundary, there is no boundary to burst across. Cap the bucket at burst and no rolling second can ever contain more than rate + burst calls — set burst to 1 and it is exactly rate.
Two details that are easy to get wrong:
time.Since versus now.Sub(l.last). Call time.Now() once and pass it in. Calling the clock twice inside one operation lets the two readings disagree, and the arithmetic between them stops being consistent.
The elapsed <= 0 guard. Go's time.Time carries a monotonic reading, so in practice this cannot go backwards — but the guard costs nothing and it means the invariant "tokens never decrease except by spending" holds unconditionally, including if someone later reconstructs a Time from a wall clock.
Wait: compute under the lock, sleep outside it
func (l *Limiter) Wait(ctx context.Context) error {
l.mu.Lock()
l.advance(time.Now())
if l.tokens >= 1 {
l.tokens--
l.mu.Unlock()
return nil
}
// Reserve the token now, and work out when it will have accrued.
deficit := 1 - l.tokens
l.tokens -= 1 // may go negative: this caller owns a future token
delay := time.Duration(deficit / l.rate * float64(time.Second))
l.mu.Unlock()
t := time.NewTimer(delay)
defer t.Stop()
select {
case <-t.C:
return nil
case <-ctx.Done():
// Hand the reservation back, or a cancelled caller starves the next one.
l.mu.Lock()
l.tokens++
l.mu.Unlock()
return ctx.Err()
}
}Three things are load-bearing here.
The lock is released before the sleep. This is the mistake the hint warned about. Sleeping under the mutex serialises every caller behind the slowest one and destroys throughput.
tokens is allowed to go negative. That is the reservation. A caller who will be served in 40ms decrements immediately, so the next caller computes its own delay relative to a bucket that already accounts for the one ahead of it. This is what turns a stampede into a queue: waiters are naturally spaced 1/rate apart. Without it, every waiter computes the same wake-up time and they all collide on the same token — a thundering herd you built yourself.
Cancellation returns the token. Miss this and every timed-out request permanently reduces your throughput. It is a slow leak of capacity that looks, from the graphs, exactly like the gateway getting slower.
Worth knowing: golang.org/x/time/rate is this algorithm, carefully done, and in production you should use it. Write it once so you know what it is doing.
Solution — Python: two answers, and they do not mixTry it first
Python is the only one of the three where the concurrency model changes the answer, so it gets two implementations.
Threads:
import threading, time
class Limiter:
def __init__(self, rate: float, burst: float) -> None:
self._rate = rate
self._burst = burst
self._tokens = burst
self._last = time.monotonic() # never time.time()
self._lock = threading.Lock()
def _advance(self, now: float) -> None: # caller holds the lock
elapsed = now - self._last
if elapsed <= 0:
return
self._tokens = min(self._burst, self._tokens + elapsed * self._rate)
self._last = now
def allow(self) -> bool:
with self._lock:
self._advance(time.monotonic())
if self._tokens < 1:
return False
self._tokens -= 1
return True
def wait(self, timeout: float | None = None) -> bool:
with self._lock:
self._advance(time.monotonic())
self._tokens -= 1 # reserve
if self._tokens >= 0:
return True
delay = -self._tokens / self._rate
if timeout is not None and delay > timeout:
with self._lock:
self._tokens += 1 # hand it back
return False
time.sleep(delay) # lock released
return Trueasyncio:
import asyncio
class AsyncLimiter:
def __init__(self, rate: float, burst: float) -> None:
self._rate, self._burst = rate, burst
self._tokens = burst
self._last = asyncio.get_event_loop().time()
self._lock = asyncio.Lock()
async def acquire(self) -> None:
async with self._lock:
now = asyncio.get_running_loop().time()
elapsed = now - self._last
if elapsed > 0:
self._tokens = min(self._burst, self._tokens + elapsed * self._rate)
self._last = now
self._tokens -= 1
delay = 0.0 if self._tokens >= 0 else -self._tokens / self._rate
if delay:
await asyncio.sleep(delay) # lock released firsttime.monotonic(), not time.time(). Same reasoning as Java's nanoTime: time.time() is the wall clock and it can move backwards. time.perf_counter() is also monotonic and higher-resolution; either is fine, and time.time() is a bug.
Never time.sleep() in async code, and never asyncio.Lock across threads. These are the two ways this gets written wrong. A time.sleep() inside a coroutine blocks the entire event loop — your limiter stops the whole server rather than one caller. And asyncio.Lock is not thread-safe: it synchronises coroutines on one loop, nothing more. Two models, two implementations, and mixing them produces a bug that looks like the limiter "sometimes not working".
The GIL is not the limiter. It is tempting to think a serialised interpreter naturally rate-limits you. It serialises bytecode, not the outbound HTTP calls, which are exactly the thing the gateway is counting: every one of them releases the GIL while waiting on the socket, so a hundred threads issue a hundred concurrent requests. The GIL constrains your CPU and not your egress.
asyncio.Semaphore is a different tool. A semaphore of 100 caps concurrency — how many requests are in flight at once. The gateway's limit is a rate — how many start per second. With 50ms responses, a semaphore of 100 permits roughly 2,000 requests per second. Conflating the two is the most common version of this mistake in production code.
Add the blocking variant
Do this
Implement wait(), which blocks until a token is available instead of returning false, and returns early if the caller is cancelled.
The obvious version locks, sees no token, and sleeps. What does that do to every other caller?
AnswerCommit to one first
They all queue behind a sleeping thread.
The sleeper is holding the mutex, so nobody else can even check whether a token is available — and when it finally wakes, they all contend for the one token it just took. You have converted a rate limiter into a serialiser.
Work out when your turn is while holding the lock. Sleep after releasing it.
Your reservation lets tokens go negative. Why is that the point rather than a bug?
AnswerCommit to one first
Because it is what turns a stampede into a queue.
A caller who will be served in 40ms decrements immediately, so the next caller computes its delay against a bucket that already accounts for the one ahead of it. Waiters end up naturally spaced 1/rate apart.
Without it, every waiter computes the same wake-up time, they all wake together, and they collide on the same token — a thundering herd you built yourself.
Handle cancellation
A caller times out while waiting and gives up. What must happen to its reserved token?
AnswerCommit to one first
It has to be returned.
Miss this and every timed-out request permanently reduces your throughput. It is a slow leak of capacity that looks, on the graphs, exactly like the gateway getting slower — which is the wrong thing to go and investigate.
Rate limiting in three languages: the clock is the hard part
The algorithm is the same in all three. Every real difference comes from time and from what each runtime considers a thread.
Java
The clock is safe only if you pick the right method. currentTimeMillis and nanoTime sit side by side and only one is monotonic; the familiar one is the broken one. nanoTime values must be compared by subtraction because they can wrap. Against that, Java gives you the best fairness story of the three — ReentrantLock(true) with a Condition yields real FIFO queueing — and with virtual threads a blocking acquire() is now as cheap as Go's, which removes the last reason to prefer a callback-based limiter.
Go
The clock is safe by default. time.Time carries a monotonic reading, so now.Sub(last) is immune to wall-clock steps without your having to know that. time.Timer plus select gives cancellation for free, and a goroutine blocked in Wait costs a few kilobytes — so the blocking variant is cheap and obvious. Go's one trap is the ticker-and-channel design: idiomatic, readable, and it leaks a goroutine and a timer per limiter unless every caller closes it.
Python
Two concurrency models, two limiters, and they must not be mixed. time.monotonic() is the correct clock and time.time() is the one people reach for. The threaded version uses threading.Lock and time.sleep; the asyncio version uses asyncio.Lock and await asyncio.sleep, and each primitive is useless-to-harmful in the other's world — a time.sleep in a coroutine stops the whole event loop. And the GIL limits your CPU, not your outbound requests, so it provides no rate limiting whatsoever.
The transferable lesson. Every bug in this exercise was a bug about time, not about locking: a window boundary that the upstream does not share, a clock that can move backwards, a sleep held under a lock, a reservation not returned on cancellation. Concurrency writing tends to obsess over mutual exclusion because that is what the tutorials cover, but in production code that touches a clock, the clock is where the defects live. Choose a monotonic source, read it once per operation, and never hold a lock across a wait.
Before you move on
You should be able to explain each of these without looking.
- The two timestamps that break a fixed-window limiter, and why the gateway is right and you are wrong.
- Why accrual has no boundary to burst across.
- Why
Waitmust not hold the lock while sleeping, and what symptom that bug produces. - Why negative
tokensis the mechanism that spaces waiters out, and what happens without it. - Why a cancelled
Waitmust return its token. - Why
System.currentTimeMillis()andtime.time()are both wrong, and what breaks when the clock steps back. - Why
nanoTimevalues must be compared by subtraction. - Why a
time.sleep()inside a coroutine is worse than the bug it was meant to fix. - The difference between a semaphore of 100 and a rate of 100/second.
Go further
- Implement the ticker-and-buffered-channel version as well. It is a legitimate design and much easier to read. Then write the test that proves it leaks a goroutine when the limiter is dropped without
Close(), and decide which cost you would rather pay. - Add
AllowN(n int)for a batch settlement that consumes several tokens. Now think about fairness: a caller wanting 50 tokens can be starved indefinitely by a stream of callers wanting 1. - Make the limiter per-merchant, with a map of limiters. You now have a map that is read constantly and written rarely, plus an eviction problem for merchants who have gone quiet. Both are real production concerns and neither has a tidy answer.
- Under 500 goroutines, profile it. The single mutex is now the contended resource, and the limiter protecting your gateway has become the bottleneck in your own service.