API Rate Limiting in Go: Token Bucket, Middleware and Distributed Limits

The problem

An API with no rate limiting has exactly one throughput ceiling: whatever breaks first. A misbehaving client retrying in a tight loop, a scraper hammering a search endpoint, or a legitimate customer who just wrote a bad for loop calling your API — all of them can consume enough capacity to degrade the service for everyone else. Rate limiting isn't about being hostile to clients; it's about making the failure mode "this one client gets a 429" instead of "the database connection pool is exhausted for everyone."

Why it matters

Without limits, three things go wrong in production, usually at the worst time:

• Downstream dependencies get overloaded. An endpoint that queries PostgreSQL or calls a third-party API has a real cost per request. Enough concurrent callers and the bottleneck isn't your Go service — it's whatever it talks to.
• One client can starve every other client. Without per-client accounting, a single abusive or buggy caller consumes the same shared capacity as everyone else, and there's no way to isolate the damage.
• Cost scales with abuse, not with legitimate usage. If the API calls a metered external service or runs on autoscaled infrastructure, unbounded request volume is a direct cost problem, not just a performance one.

Rate limiting algorithms

Fixed window counts requests in discrete windows (e.g. "100 requests per minute, reset on the minute"). Simple to implement and reason about, but it allows a burst of 2×limit requests right at the window boundary — 100 requests in the last second of one window, another 100 in the first second of the next.

Sliding window fixes the boundary-burst problem by weighting the previous window's count based on how far into the current window you are. More accurate, more bookkeeping, rarely worth the complexity unless the boundary burst is a real problem for your traffic pattern.

Token bucket is the one worth defaulting to for APIs. Picture a bucket that holds up to N tokens. Tokens refill at a steady rate (e.g. 10/second) up to the bucket's capacity. Every request costs one token; if the bucket is empty, the request is rejected. This naturally allows short bursts (a client that's been idle has a full bucket) while still enforcing a long-run average rate — which matches how real clients actually behave better than a hard per-window cap.

Leaky bucket is the token bucket's mirror image: requests queue up and are processed ("leak out") at a fixed rate, smoothing bursts into a steady output stream. It's the right model when you need to protect a downstream system that genuinely can't handle bursts at all (e.g. a fixed-capacity worker pool) rather than just wanting fair usage accounting.

For most HTTP APIs, token bucket keyed per-client is the default choice: it tolerates bursty-but-legitimate traffic (a client loading a dashboard that fires 8 requests at once) without needing a queue.

Keying strategy

The algorithm doesn't change based on what you key by — only what counts as "one client" does:

• Per-IP — the only option before authentication happens (e.g. the login endpoint itself), but unreliable behind shared NATs or corporate proxies where many real users share one IP.
• Per-user — accurate once a request is authenticated; ties the limit to an actual account.
• Per-API-key — the standard for machine-to-machine APIs, and lets you offer different limits per pricing tier by attaching the limit to the key's metadata instead of a single global constant.

Implementation: in-memory token bucket middleware

A single-process token bucket limiter is straightforward with the standard library — a bucket per client key, protected by a mutex, refilled lazily based on elapsed time rather than a background ticker:

package ratelimit
 
import (
	"net/http"
	"strconv"
	"sync"
	"time"
)
 
type bucket struct {
	tokens     float64
	lastRefill time.Time
}
 
// Limiter is a per-key token bucket rate limiter. It refills lazily on
// each Allow call rather than running a ticker per bucket, which keeps it
// cheap regardless of how many distinct keys are seen.
type Limiter struct {
	mu       sync.Mutex
	buckets  map[string]*bucket
	rate     float64 // tokens added per second
	capacity float64 // maximum tokens (burst size)
}
 
func NewLimiter(ratePerSecond float64, burst int) *Limiter {
	l := &Limiter{
		buckets:  make(map[string]*bucket),
		rate:     ratePerSecond,
		capacity: float64(burst),
	}
	go l.cleanupLoop()
	return l
}
 
func (l *Limiter) Allow(key string) (allowed bool, retryAfter time.Duration) {
	l.mu.Lock()
	defer l.mu.Unlock()
 
	now := time.Now()
	b, ok := l.buckets[key]
	if !ok {
		b = &bucket{tokens: l.capacity, lastRefill: now}
		l.buckets[key] = b
	}
 
	elapsed := now.Sub(b.lastRefill).Seconds()
	b.tokens = min(l.capacity, b.tokens+elapsed*l.rate)
	b.lastRefill = now
 
	if b.tokens >= 1 {
		b.tokens--
		return true, 0
	}
 
	deficit := 1 - b.tokens
	return false, time.Duration(deficit/l.rate*float64(time.Second))
}
 
// cleanupLoop evicts buckets that have been full and idle for a while, so
// a limiter serving many distinct clients doesn't grow its map forever.
func (l *Limiter) cleanupLoop() {
	ticker := time.NewTicker(10 * time.Minute)
	defer ticker.Stop()
	for range ticker.C {
		l.mu.Lock()
		for key, b := range l.buckets {
			if time.Since(b.lastRefill) > 10*time.Minute && b.tokens >= l.capacity {
				delete(l.buckets, key)
			}
		}
		l.mu.Unlock()
	}
}
 
func min(a, b float64) float64 {
	if a < b {
		return a
	}
	return b
}

The middleware wires it to the request's client key and returns a proper 429 with Retry-After when the bucket is empty:

func Middleware(limiter *Limiter, keyFunc func(*http.Request) string) func(http.Handler) http.Handler {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			allowed, retryAfter := limiter.Allow(keyFunc(r))
			if !allowed {
				w.Header().Set("Retry-After", strconv.Itoa(int(retryAfter.Seconds())+1))
				http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
				return
			}
			next.ServeHTTP(w, r)
		})
	}
}

Why in-memory limiting breaks with multiple replicas

This implementation is correct — for a single process. The moment the service runs behind a load balancer with more than one replica, each replica holds its own independent map of buckets. A limit configured as "100 requests/minute per API key" doesn't become 100 requests/minute overall; it becomes 100 × replica_count requests/minute, because the load balancer spreads a client's requests across replicas that have no idea about each other's state.

Concretely: with 5 replicas and a configured limit of 100/minute, a client that fans out its requests evenly across replicas can sustain roughly 500/minute before any single replica's bucket empties — five times the intended limit. This isn't a rare edge case; round-robin and least-connection load balancing both spread traffic this way by default.

A distributed approach with Redis

Fixing this requires the bucket state to live somewhere every replica can see — a shared store, not a step per instance:

The core operation — "atomically check and decrement a token count" — needs to happen without a race between replicas. Two common approaches:

• INCR + EXPIRE for a fixed-window counter: INCR ratelimit:{key}, and on the first increment (INCR returns 1) set an expiry equal to the window. Simple, but inherits fixed window's boundary-burst behavior.
• A Lua script for a real token bucket: Redis executes Lua scripts atomically, so a script that reads the current token count, computes the refill based on elapsed time, and conditionally decrements can implement the exact same algorithm as the in-memory version — just centralized. This is what libraries like redis-cell or a hand-rolled EVAL script provide.

This isn't free: every request now costs a network round trip to Redis before the handler even runs, Redis latency becomes part of every request's latency, and Redis itself becomes a dependency whose outage now affects every request path that's gated by it (the usual mitigation is failing open — allowing requests through if Redis is unreachable — rather than failing closed and taking the whole API down over an unrelated Redis blip). Some teams deliberately accept the imprecision of per-replica in-memory limiting for high-volume, low-stakes endpoints, and reserve Redis-backed distributed limiting for the handful of endpoints where an exact global limit actually matters (login attempts, payment endpoints, a metered third-party API call).

Production considerations

• Match the key to the endpoint's actual cost. A single global limit across wildly different endpoints (a cheap health check vs. an expensive report-generation endpoint) either starves the cheap ones or lets the expensive one through too often. Tier limits per endpoint group.
• Log rate-limit rejections with the key, not just a count — "which API key is hitting its limit" is what turns into a conversation with a customer about raising their tier, not just a noisy metric.
• Decide your failure mode for the limiter store up front. If using Redis, explicitly choose fail-open or fail-closed, and make sure whoever operates the service knows which one is configured.

Common mistakes

• Rate limiting only by IP for authenticated endpoints, which misattributes usage behind shared corporate NATs and is trivially evaded by anyone who can rotate IPs.
• Not returning Retry-After. Well-behaved clients use it to back off correctly; without it, a rejected client's next guess at when to retry is just as likely to hit the limit again.
• One limit for every endpoint, ignoring that a GET /health and a POST /reports/generate cost the backend wildly different amounts.
• Deploying a distributed limiter without deciding what happens when Redis is unavailable — an unplanned fail-closed turns a Redis blip into a full API outage.

Testing

The token bucket logic is pure enough to test without any HTTP layer, by controlling the clock indirectly through the bucket's lastRefill field:

func TestLimiter_AllowsBurstThenThrottles(t *testing.T) {
	limiter := NewLimiter(1, 3) // 1 token/sec, burst of 3
 
	for i := 0; i < 3; i++ {
		allowed, _ := limiter.Allow("client_1")
		if !allowed {
			t.Fatalf("request %d: expected allowed, bucket should have burst capacity", i)
		}
	}
 
	allowed, retryAfter := limiter.Allow("client_1")
	if allowed {
		t.Fatal("expected the 4th immediate request to be rejected")
	}
	if retryAfter <= 0 {
		t.Fatal("expected a positive retry-after duration")
	}
}
 
func TestLimiter_KeysAreIndependent(t *testing.T) {
	limiter := NewLimiter(1, 1)
 
	if allowed, _ := limiter.Allow("client_a"); !allowed {
		t.Fatal("client_a's first request should be allowed")
	}
	if allowed, _ := limiter.Allow("client_b"); !allowed {
		t.Fatal("client_b should have its own independent bucket")
	}
}

Summary

Token bucket is the right default algorithm for API rate limiting — it tolerates realistic bursty traffic while still enforcing a long-run rate. An in-memory implementation is simple and fast, but only enforces the configured limit per replica; behind a load balancer with multiple replicas, the effective limit is configured_limit × replica_count. Fixing that requires centralizing the bucket state in something every replica can reach — typically Redis, with an atomic Lua script to avoid a check-then-act race — at the cost of a network hop per request and a new dependency whose failure mode you need to decide explicitly.