Retries, Timeouts and Backoff in Go: Building Resilient Services

The problem

A function call in a single process either returns or panics — there's no in-between. A call to another service over the network has a third outcome that dominates production incidents: you don't know what happened. The request may never have arrived. It may have arrived, succeeded, and the response was lost on the way back. The downstream service may just be slow, not down, and about to succeed on its own.

Naive code treats all of these the same way it treats a local error: return it and let the caller deal with it. Resilient code has to make an explicit decision about each failure mode — wait, retry, or give up — and that decision has to account for what a slow or degraded dependency does to everyone else calling it at the same time.

Why it matters

Distributed calls fail in ways a local call doesn't:

• Network partitions — packets don't arrive, and nothing tells you why.
• Slow, not down — the worst case. A dependency at 95% capacity looks fine to a health check but adds seconds to every call.
• Connection resets — a load balancer recycling a backend mid-request.
• DNS hiccups — a resolution that normally takes microseconds taking seconds, or failing outright.

Every one of these can either resolve itself on a second attempt or get dramatically worse if every caller retries at once. The difference between "resilient" and "the retry logic caused the outage" is timeouts, backoff, jitter, and knowing when not to retry at all.

Timeouts

A retry loop is meaningless without a timeout bounding each attempt — otherwise "retry" just means "wait longer on top of an already-unbounded wait." Use context.WithTimeout per attempt, and a parent context with its own deadline bounding the entire retry sequence. The mechanics of context.Context — cancellation, deadlines, propagation — are covered in depth in Context in Go; this article assumes that and focuses on what wraps around it.

Retryable vs non-retryable errors

Not every failure deserves a second attempt. Retrying the wrong ones wastes time and can cause harm:

The rule: retry failures that represent the request possibly not being processed. Don't retry failures that represent the request being processed and rejected.

Exponential backoff and jitter

Retrying immediately after a failure is close to the worst option available. If a dependency is struggling under load, an instant retry from every failed caller arrives at the exact moment it's least able to handle more traffic — the retries themselves become additional load on an already-overloaded system.

Exponential backoff spaces out attempts: wait base * 2^attempt before retrying, so attempt 2 waits longer than attempt 1, attempt 3 longer still. This gives a struggling dependency room to recover instead of being hit continuously.

Pure exponential backoff has its own failure mode: if a thousand clients all started at roughly the same time and all failed on attempt 1, they all compute the same backoff duration and retry in the same synchronized burst on attempt 2 — backoff alone doesn't desynchronize them. Jitter fixes this by randomizing the wait within a range, spreading retries out over time instead of clustering them.

Practical implementation

A retryable HTTP client combining timeout, retry, exponential backoff with jitter, and context cancellation — standard library only:

package resilientclient
 
import (
	"context"
	"errors"
	"fmt"
	"io"
	"math/rand"
	"net/http"
	"time"
)
 
type Client struct {
	http       *http.Client
	maxRetries int
	baseDelay  time.Duration
	maxDelay   time.Duration
}
 
func New(timeout time.Duration) *Client {
	return &Client{
		http:       &http.Client{Timeout: timeout},
		maxRetries: 3,
		baseDelay:  200 * time.Millisecond,
		maxDelay:   5 * time.Second,
	}
}
 
// Do executes req, retrying transient failures with exponential backoff
// and jitter. It never retries past ctx's deadline and never retries
// non-idempotent methods unless the caller has already ensured safety
// (see the idempotency article for why that matters).
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
	var lastErr error
 
	for attempt := 0; attempt <= c.maxRetries; attempt++ {
		if attempt > 0 {
			delay := c.backoff(attempt)
			select {
			case <-time.After(delay):
			case <-ctx.Done():
				return nil, fmt.Errorf("retry aborted: %w", ctx.Err())
			}
		}
 
		resp, err := c.http.Do(req.Clone(ctx))
		if err == nil && !isRetryableStatus(resp.StatusCode) {
			return resp, nil
		}
 
		if err != nil {
			lastErr = err
		} else {
			lastErr = fmt.Errorf("retryable status: %d", resp.StatusCode)
			io.Copy(io.Discard, resp.Body)
			resp.Body.Close()
		}
 
		if err != nil && !isRetryableError(err) {
			return nil, err
		}
 
		if ctx.Err() != nil {
			return nil, fmt.Errorf("context done during retry: %w", ctx.Err())
		}
	}
 
	return nil, fmt.Errorf("all %d attempts failed: %w", c.maxRetries+1, lastErr)
}
 
func (c *Client) backoff(attempt int) time.Duration {
	exp := c.baseDelay * time.Duration(1<<uint(attempt-1))
	if exp > c.maxDelay {
		exp = c.maxDelay
	}
	jitter := time.Duration(rand.Int63n(int64(exp) / 2))
	return exp/2 + jitter
}
 
func isRetryableStatus(status int) bool {
	return status == http.StatusInternalServerError ||
		status == http.StatusBadGateway ||
		status == http.StatusServiceUnavailable ||
		status == http.StatusGatewayTimeout
}
 
func isRetryableError(err error) bool {
	return errors.Is(err, context.DeadlineExceeded) || !errors.Is(err, context.Canceled)
}

The whole sequence — every wait and every attempt — has to fit inside the caller's context deadline. A retry loop that ignores the parent context and just runs its own fixed schedule will keep working long after the caller has stopped waiting for an answer.

Retry storms

The systemic failure mode retries can cause: a downstream service starts degrading under normal load — say latency creeps up and a fraction of requests start timing out. Every caller experiencing a timeout retries. Those retries add more load to a service that's already struggling, pushing its latency up further, causing more timeouts, causing more retries. What started as a minor degradation compounds into a full outage, entirely driven by the retry behavior of the callers, not the original problem.

This is why a maximum retry count and backoff aren't optional extras — without them, retries turn a partial degradation into a cascading failure. At larger scale, teams add a circuit breaker on top (stop attempting calls to a dependency entirely once its failure rate crosses a threshold, and periodically test whether it's recovered) — worth knowing exists, out of scope for the standard-library approach here.

When NOT to retry

• Non-idempotent operations without a safety mechanism. Retrying a POST /payments blindly can create a duplicate charge. See Idempotency in APIs for how to make a retry safe.
• During a known incident. If a dependency is already flagged as degraded, adding retry traffic to it works against recovery, not for it.
• User-facing requests where a fast failure beats a slow one. A request that will retry for 6 seconds before failing anyway is often a worse experience than failing immediately with a clear error the client can act on.

Production considerations

• Cap total time spent retrying, not just attempt count — three retries with a 30-second max delay each can still leave a caller waiting 90 seconds, which is often worse than failing fast.
• Log every retry with enough detail to diagnose the downstream problem: which dependency, which attempt number, what error, what the eventual outcome was. Retries are usually the first visible symptom of a dependency degrading — that signal is wasted if it's not logged.
• Make retry behavior configurable per dependency, not global. A latency-sensitive internal call and a best-effort webhook delivery don't belong on the same retry policy.

Common mistakes

Dangerous retry logic looks like this — no backoff, no cap, no context awareness:

// Don't do this: retries instantly, forever, ignoring the caller's context.
for {
	resp, err := http.Get(url)
	if err == nil {
		return resp, nil
	}
}

This retries as fast as the CPU allows, never stops, and will keep running after the caller has given up and moved on — exactly the pattern that causes a retry storm.

• Retrying non-idempotent requests by default. Not every POST is safe to repeat.
• Retrying client errors. A 400 will be a 400 again; it's not worth the extra round trip.
• No jitter. Backoff alone still synchronizes many clients into bursts.
• Ignoring the parent context. A retry loop with its own fixed timeout budget, disconnected from the caller's deadline, keeps working after nobody is waiting for the result anymore.

Testing

type flakyTransport struct {
	failuresLeft int
}
 
func (t *flakyTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	if t.failuresLeft > 0 {
		t.failuresLeft--
		return nil, errors.New("simulated network failure")
	}
	return &http.Response{StatusCode: http.StatusOK, Body: http.NoBody}, nil
}
 
func TestClient_RetriesUntilSuccess(t *testing.T) {
	c := New(time.Second)
	c.http.Transport = &flakyTransport{failuresLeft: 2}
 
	req, _ := http.NewRequest(http.MethodGet, "http://example.local", nil)
	resp, err := c.Do(context.Background(), req)
 
	if err != nil {
		t.Fatalf("expected eventual success, got %v", err)
	}
	if resp.StatusCode != http.StatusOK {
		t.Fatalf("expected 200, got %d", resp.StatusCode)
	}
}
 
func TestClient_StopsOnContextCancel(t *testing.T) {
	c := New(time.Second)
	c.http.Transport = &flakyTransport{failuresLeft: 100}
 
	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
	defer cancel()
 
	req, _ := http.NewRequest(http.MethodGet, "http://example.local", nil)
	_, err := c.Do(ctx, req)
 
	if !errors.Is(err, context.DeadlineExceeded) {
		t.Fatalf("expected context.DeadlineExceeded, got %v", err)
	}
}

The flaky-transport fake exercises the retry-until-success path without a real network dependency; the second test proves the loop actually respects cancellation instead of running its own independent schedule.

Summary

Timeouts bound how long any single attempt waits. Retries handle failures that might resolve on a second try — but only the ones that actually qualify as retryable. Exponential backoff keeps a retry from adding load at the worst possible moment; jitter keeps many callers from retrying in lockstep. Context cancellation ties the whole loop to what the caller actually still cares about. And retries are only safe to apply at all once the operation being retried is idempotent — which is the next problem to solve.