Circuit Breakers in Go: Preventing Cascading Failures

The problem

Service B starts responding slowly. Not down — slow. A database query that used to take 20ms now takes 4 seconds under load. That's the more common and more dangerous failure mode than a clean outage, because nothing has actually crashed: every health check still passes, every process is still running, and every caller still thinks it's talking to a working dependency. It's just slow.

Service A calls B on every request. Those calls start timing out. A's retry logic — correctly, per Retries, Timeouts and Backoff in Go — retries them. Now A is sending more load at B, at the exact moment B is least able to absorb it. A's own goroutines pile up waiting on B's slow responses; A's connection pool and memory fill up with in-flight requests going nowhere; A itself starts responding slowly to its callers. The degradation propagates outward through the call graph, one hop at a time, and services with no direct relationship to B start failing too. This is a cascading failure, and timeouts plus retries alone don't stop it — they can make it worse, because a retry is additional load, not less.

Why it matters

Retries assume the problem is transient and worth paying for again. Cascading failure is the case where paying for it again is precisely what turns a single degraded dependency into an outage that spans the system. The missing piece is a mechanism that notices "calls to B are failing at an abnormal rate" and responds by stopping calls to B entirely for a while — protecting A's own resources, and giving B room to recover instead of absorbing an ever-growing flood of retried requests on top of whatever is already slowing it down.

Circuit breaker states

A circuit breaker tracks the health of calls to one specific dependency and moves between three states:

• Closed — normal operation. Calls pass through to the real dependency, and failures are counted.
• Open — the failure threshold has been crossed. Calls fail immediately, without even attempting the network call, until a configured open-timeout elapses. This is what protects the caller: no goroutine sits waiting on a dependency that's already known to be unhealthy.
• Half-Open — once the open timeout elapses, a small number of trial requests are allowed through to test whether B has recovered. If they succeed, the breaker closes and normal traffic resumes. If any fail, it reopens and the timeout starts again.

Practical implementation

A breaker wraps a call and needs three pieces of state: the current status, a failure counter, and the time it last opened.

package breaker
 
import (
	"errors"
	"log/slog"
	"sync"
	"time"
)
 
type state int
 
const (
	closed state = iota
	open
	halfOpen
)
 
var ErrOpen = errors.New("circuit breaker is open")
 
type Breaker struct {
	mu               sync.Mutex
	state            state
	failures         int
	failureThreshold int
	openedAt         time.Time
	openTimeout      time.Duration
	halfOpenTrials   int
	trialsInFlight   int
	logger           *slog.Logger
}
 
func New(failureThreshold int, openTimeout time.Duration, logger *slog.Logger) *Breaker {
	return &Breaker{
		failureThreshold: failureThreshold,
		openTimeout:      openTimeout,
		halfOpenTrials:   1,
		logger:           logger,
	}
}
 
// Call executes fn if the breaker allows it, and records the outcome.
func (b *Breaker) Call(fn func() error) error {
	if err := b.before(); err != nil {
		return err
	}
 
	err := fn()
	b.after(err)
	return err
}
 
func (b *Breaker) before() error {
	b.mu.Lock()
	defer b.mu.Unlock()
 
	switch b.state {
	case open:
		if time.Since(b.openedAt) < b.openTimeout {
			return ErrOpen
		}
		b.transition(halfOpen)
		b.trialsInFlight = 0
		fallthrough
	case halfOpen:
		if b.trialsInFlight >= b.halfOpenTrials {
			return ErrOpen
		}
		b.trialsInFlight++
	}
	return nil
}
 
func (b *Breaker) after(err error) {
	b.mu.Lock()
	defer b.mu.Unlock()
 
	if err != nil {
		b.failures++
		if b.state == halfOpen || b.failures >= b.failureThreshold {
			b.transition(open)
			b.openedAt = time.Now()
		}
		return
	}
 
	if b.state == halfOpen {
		b.transition(closed)
	}
	b.failures = 0
}
 
func (b *Breaker) transition(to state) {
	if b.logger != nil {
		b.logger.Info("circuit breaker state change",
			"from", b.state, "to", to, "failures", b.failures)
	}
	b.state = to
}

before is where the state machine actually lives: an Open breaker whose timeout hasn't elapsed rejects immediately; once the timeout has elapsed, it moves to HalfOpen and allows exactly halfOpenTrials requests through as a test. after records the outcome — a failure while half-open reopens immediately (one bad trial means it's not ready yet), a success while half-open closes the breaker and resets the failure count.

Wiring it around an HTTP call is a thin wrapper:

func (c *PaymentClient) Charge(ctx context.Context, req ChargeRequest) (*ChargeResponse, error) {
	var resp *ChargeResponse
	err := c.breaker.Call(func() error {
		var callErr error
		resp, callErr = c.doCharge(ctx, req)
		return callErr
	})
	if errors.Is(err, breaker.ErrOpen) {
		return nil, fmt.Errorf("payment service unavailable: %w", err)
	}
	return resp, err
}

Callers see a fast, explicit ErrOpen instead of hanging on a timeout they'd have hit anyway — the breaker turns a slow failure into an immediate one, which is the whole point.

How resilience mechanisms combine

Timeout, retry, circuit breaker, and rate limiting solve different problems, and stacking them in the wrong order or configuration actively makes things worse:

• Retrying inside a call that's already near its context deadline just extends one slow failure into a longer one — the retry loop must respect the same deadline the original call was bound by, not add its own on top.
• Retrying against a dependency whose circuit breaker is already open defeats the breaker's purpose entirely. The correct layering is the breaker wraps the retry loop — a single Call invocation that internally retries a few times sits inside one breaker check, so an open breaker stops the whole retry attempt before it starts, rather than the retry loop hitting the breaker fresh on every attempt and quietly working around it.
• A failure threshold that's too low trips the breaker on a single brief blip — a GC pause, one slow query — and then holds it open for the full timeout period even after the dependency has already recovered, rejecting healthy traffic for no reason. The threshold should reflect a real, sustained failure rate for the specific dependency's normal noise level, not the smallest number that's technically "more than one."

Production considerations

Use one breaker per dependency, not one global breaker. A single shared breaker means a failing payment provider trips protection for calls to an unrelated inventory service too — group breakers by the actual failure domain, which is almost always "one breaker per external dependency."

Alert on state transitions to Open, not just on raw error rates. A breaker tripping is a distilled, high-confidence signal that a specific dependency is unhealthy — it's a better paging signal than a generic error-rate threshold because it already accounts for the fact that a few scattered failures are normal.

Size the open-timeout to the dependency's actual recovery behavior. Too short, and the breaker flaps between Open and Half-Open repeatedly without giving the dependency real breathing room; too long, and it keeps rejecting traffic well after the dependency has actually recovered.

Common mistakes

• Retry logic with no breaker awareness, so retries keep hammering a dependency that's already open, defeating the breaker.
• One breaker shared across unrelated dependencies, so an unrelated failure trips protection for calls that were working fine.
• A threshold tuned to "any failure trips it," which reacts to normal jitter instead of sustained unhealthiness, and spends most of its time open for no real benefit.
• No visibility into state transitions — a breaker that trips silently just looks like elevated latency until someone digs through logs to find out why.

Testing

The state machine is deterministic given a sequence of failures and successes, which makes it straightforward to test without any real network calls:

func TestBreaker_OpensAfterThreshold(t *testing.T) {
	b := New(3, 50*time.Millisecond, nil)
	failing := func() error { return errors.New("boom") }
 
	for i := 0; i < 3; i++ {
		if err := b.Call(failing); err == nil {
			t.Fatalf("call %d: expected failure to propagate", i)
		}
	}
 
	if err := b.Call(failing); !errors.Is(err, ErrOpen) {
		t.Fatalf("expected breaker to be open, got %v", err)
	}
}
 
func TestBreaker_HalfOpenClosesOnSuccess(t *testing.T) {
	b := New(1, 10*time.Millisecond, nil)
	_ = b.Call(func() error { return errors.New("boom") }) // opens the breaker
 
	time.Sleep(15 * time.Millisecond) // let the open timeout elapse
 
	if err := b.Call(func() error { return nil }); err != nil {
		t.Fatalf("expected the half-open trial to succeed, got %v", err)
	}
	if err := b.Call(func() error { return nil }); err != nil {
		t.Fatalf("expected breaker to be closed after a successful trial, got %v", err)
	}
}

Summary

A circuit breaker exists to stop cascading failure: when a dependency's failure rate crosses a threshold, it fails fast instead of piling up retried requests against something that's already struggling. The three states — Closed, Open, Half-Open — give it a way to protect the caller immediately and test for recovery automatically. It only works correctly as part of a layered stack, though: wrap the retry loop with the breaker, not the other way around, and keep rate limiting and circuit breaking as the separate mechanisms they are — one protects you from your callers, the other protects you from your dependencies.