Building Background Workers in Go with Goroutines and Channels

The problem

Most backend services eventually need to do work off the request path: sending a confirmation email, resizing an uploaded image, processing an incoming webhook, re-indexing a search document. None of that belongs inside the HTTP handler that triggered it — the caller shouldn't wait 800ms for a thumbnail to render just to get a 201 back.

The obvious Go move is to spawn a goroutine and move on:

func (h *UploadHandler) Create(w http.ResponseWriter, r *http.Request) {
	upload := h.saveUpload(r)
 
	go processImage(upload) // fire and forget
 
	w.WriteHeader(http.StatusCreated)
}

This works. It works in local testing, it works in staging, it works in production for weeks — right up until traffic spikes, a downstream API slows down, or someone runs a bulk import. Then it fails all at once, because nothing in that line of code bounds how much concurrent work the process is willing to take on. This article builds the thing that should replace it: a worker pool with a fixed number of goroutines, a bounded job queue, graceful shutdown, and panic isolation — the pattern almost every production Go service ends up needing once "do it in a goroutine" stops being good enough.

Why "go process(job) for every job" is dangerous

Each of these failure modes comes from the same root cause: the number of concurrent jobs is decided by whoever calls go process(job), not by the process running it.

Unbounded goroutine growth. A goroutine's initial stack is small (2KB), but it grows as needed, and processImage almost certainly holds a decoded image, a buffer, or an HTTP response body in memory for its lifetime. Ten thousand of those in flight at once is easily gigabytes of memory that didn't exist a second earlier. Nothing caps how many jobs can be "in flight" — the number is just however many requests arrived before the first one finished.

Database overload. If processImage writes a row when it's done, every one of those goroutines eventually holds a connection from the pool at the same moment. This is the same pool-exhaustion failure covered in the connection-pooling article, just triggered from the other direction: there it's too many requests opening connections, here it's too many background goroutines opening them. A pool sized for 20 concurrent queries doesn't care whether the 500 concurrent callers were HTTP requests or goroutines — it queues or errors either way.

Downstream service overload. The same math applies to any external API a job calls — a payment provider, an email service, a third-party webhook target. Unbounded goroutines means unbounded concurrent outbound requests, and most third-party APIs rate-limit or start timing out well before "unbounded" is reached. Now jobs are failing and probably retrying, which multiplies the load instead of shedding it.

No backpressure. This is the part that makes the first three worse than they need to be. Nothing tells the producer to slow down. The HTTP handler that calls go processImage(upload) returns instantly no matter what's happening inside the process — it has no idea 50,000 of those goroutines are already queued up waiting for a slow downstream API. Jobs just keep piling up in memory until something gives: an OOM kill, a crashed database pool, or a cascading timeout. A system with no backpressure doesn't degrade gracefully, it falls over.

The worker pool pattern

The fix is to decouple accepting a job from running it. A fixed number of long-lived worker goroutines pull jobs off a shared channel, one at a time each. The channel's buffer becomes the job queue; the number of workers becomes the concurrency ceiling. A producer can enqueue jobs as fast as it wants — the pool only ever runs NumWorkers of them at once.

Four workers, one queue. Whatever processImage does — hit a database, call an API, burn CPU — the pool never runs more than four of them concurrently, regardless of how many jobs the producer submits.

Implementation

Start with the job itself. Keep it a plain struct — a function value works for simple cases, but a struct makes it easy to add fields later (retry count, priority, a trace ID) without changing every call site.

package worker
 
import (
	"context"
	"fmt"
	"log/slog"
	"sync"
	"time"
)
 
// Job is one unit of background work. Fn receives the pool's shutdown
// context so long-running work can observe cancellation.
type Job struct {
	ID string
	Fn func(ctx context.Context) error
}
 
type Pool struct {
	jobs    chan Job
	wg      sync.WaitGroup
	logger  *slog.Logger
	cancel  context.CancelFunc
	rootCtx context.Context
}
 
// NewPool starts NumWorkers goroutines reading from a buffered channel of
// size queueSize. queueSize is the number of jobs that can wait without
// blocking the producer — it is the pool's backpressure threshold.
func NewPool(numWorkers, queueSize int, logger *slog.Logger) *Pool {
	ctx, cancel := context.WithCancel(context.Background())
	p := &Pool{
		jobs:    make(chan Job, queueSize),
		logger:  logger,
		cancel:  cancel,
		rootCtx: ctx,
	}
 
	for i := 0; i < numWorkers; i++ {
		p.wg.Add(1)
		go p.runWorker(i)
	}
 
	return p
}

The queue is a bounded channel, not an unbounded slice guarded by a mutex. That bound is doing real work: once queueSize jobs are already waiting, a send on p.jobs blocks until a worker frees up a slot. That's backpressure, and it's usually exactly the behavior you want — the producer feels the system slowing down instead of piling more work into memory that will never be processed in time.

Each worker loops forever, recovering from panics per job so one bad job can't take the whole worker down:

func (p *Pool) runWorker(id int) {
	defer p.wg.Done()
 
	for {
		select {
		case job, ok := <-p.jobs:
			if !ok {
				return // channel closed, no more jobs coming
			}
			p.runJob(id, job)
		case <-p.rootCtx.Done():
			return
		}
	}
}
 
func (p *Pool) runJob(workerID int, job Job) {
	defer func() {
		if r := recover(); r != nil {
			p.logger.Error("job panicked",
				"worker", workerID, "job_id", job.ID, "panic", r)
		}
	}()
 
	if err := job.Fn(p.rootCtx); err != nil {
		p.logger.Error("job failed",
			"worker", workerID, "job_id", job.ID, "error", err)
	}
}

The recover() lives inside runJob, not runWorker — if it were only in the outer loop, a single panic would unwind past the for and end the worker goroutine for good, silently shrinking the pool by one every time a job misbehaves. Recovering at the job boundary means the worker logs the failure and immediately goes back to select, ready for the next job.

Submitting a job is a plain channel send, with the option to fail fast instead of blocking forever if the pool is already shutting down:

// Submit enqueues a job. It blocks if the queue is full — this is the
// pool's backpressure. Returns an error if the pool has stopped accepting
// work.
func (p *Pool) Submit(job Job) error {
	select {
	case p.jobs <- job:
		return nil
	case <-p.rootCtx.Done():
		return fmt.Errorf("pool is shutting down, job %s rejected", job.ID)
	}
}

Shutdown mirrors the graceful-shutdown pattern used for HTTP servers: stop accepting new work, give in-flight work a bounded amount of time to finish, then force it.

// Shutdown stops accepting new jobs and waits for in-flight jobs to
// finish, up to timeout. Workers already running a job when Shutdown is
// called get to complete it; the context passed to Fn is cancelled once
// timeout elapses so well-behaved jobs can exit early.
func (p *Pool) Shutdown(timeout time.Duration) error {
	close(p.jobs) // no new jobs will be received; queued jobs still drain
 
	done := make(chan struct{})
	go func() {
		p.wg.Wait()
		close(done)
	}()
 
	select {
	case <-done:
		p.cancel()
		return nil
	case <-time.After(timeout):
		p.cancel() // propagate cancellation into any job still running
		<-done      // still wait — workers must exit before the process does
		return fmt.Errorf("worker pool shutdown timed out after %s", timeout)
	}
}

Two details here matter more than they look. First, close(p.jobs) stops new sends from being received (any pending Submit unblocks via p.rootCtx.Done() instead) but lets workers keep draining whatever is already buffered — that's why shutdown can still process a backlog instead of dropping it. Second, p.wg.Wait() runs in its own goroutine specifically so Shutdown can race it against a timeout; waiting on the WaitGroup directly would block forever if a job ignores cancellation and never returns.

Backpressure

Submit above blocks when the queue is full, which is the right default for most background work — a signup email that's a few seconds late because the queue was momentarily full is fine. But "block the caller" isn't always correct, and the pool should let the caller choose:

• Block (shown above) when the producer can tolerate waiting and you'd rather slow it down than lose work — batch jobs, internal pipelines.
• Drop and record a metric when the job is disposable and staying responsive matters more than completeness — a non-critical analytics event.
• Reject the request that would have enqueued the job when the job came from an HTTP request — return 503 instead of blocking the request goroutine indefinitely, using Pool.Jobs() len/cap or a non-blocking select with a default case in Submit.

Production considerations

Size the pool to what the jobs actually do, not to a round number. CPU-bound jobs (image resizing, hashing, compression) rarely benefit from a pool larger than runtime.NumCPU() — more workers just adds context switching without more throughput. I/O-bound jobs (calling an API, querying a database, waiting on a slow webhook) spend most of their time blocked, so a much larger pool — driven by how much concurrency the downstream system can absorb, not the CPU — is often correct. A pool of 50 workers calling a payment API that can handle 20 concurrent requests just moves the bottleneck; size against the downstream limit, not an arbitrary guess.

Monitor queue depth, not just worker count. A consistently near-full queue is the signal that the pool is undersized for current load — that's the metric to alert on and to scale against, well before jobs start timing out or Submit starts blocking noticeably. A queue that's consistently empty means the pool is oversized for what it's actually doing, which is wasted headroom but rarely urgent.

Common mistakes

• Spawning a goroutine per job with no ceiling. This is the whole problem this article exists to fix — if there's no fixed number of workers and no bounded queue, there's no concurrency limit at all.
• No panic recovery per job. Recovering only around the worker's outer loop, or not at all, means one malformed job permanently kills a worker and silently shrinks pool capacity — the pool degrades one panic at a time until nothing is left processing jobs.
• Forgetting to close the job channel on shutdown. If nothing ever calls close(p.jobs), workers block on <-p.jobs forever and the process can't exit cleanly — it has to be killed, and anything still queued is lost.
• Not waiting on the WaitGroup before exiting. Calling Shutdown and returning immediately without waiting for p.wg means the process can exit mid-job, silently dropping whatever those in-flight jobs were doing — the crash-on-deploy version of the same bug.

Testing

Verify the pool processes every job exactly once, regardless of how the work is split across workers:

func TestPool_ProcessesAllJobs(t *testing.T) {
	const numJobs = 500
	pool := NewPool(8, numJobs, slog.Default())
 
	var mu sync.Mutex
	seen := make(map[string]bool)
 
	for i := 0; i < numJobs; i++ {
		id := fmt.Sprintf("job-%d", i)
		err := pool.Submit(Job{
			ID: id,
			Fn: func(ctx context.Context) error {
				mu.Lock()
				seen[id] = true
				mu.Unlock()
				return nil
			},
		})
		if err != nil {
			t.Fatalf("submit failed: %v", err)
		}
	}
 
	if err := pool.Shutdown(5 * time.Second); err != nil {
		t.Fatalf("shutdown: %v", err)
	}
	if len(seen) != numJobs {
		t.Fatalf("expected %d jobs processed, got %d", numJobs, len(seen))
	}
}

And verify a panicking job doesn't take its worker — or any other job — down with it:

func TestPool_SurvivesJobPanic(t *testing.T) {
	pool := NewPool(2, 10, slog.Default())
 
	var processed atomic.Int32
 
	_ = pool.Submit(Job{
		ID: "panics",
		Fn: func(ctx context.Context) error {
			panic("boom")
		},
	})
 
	for i := 0; i < 5; i++ {
		_ = pool.Submit(Job{
			ID: fmt.Sprintf("ok-%d", i),
			Fn: func(ctx context.Context) error {
				processed.Add(1)
				return nil
			},
		})
	}
 
	if err := pool.Shutdown(5 * time.Second); err != nil {
		t.Fatalf("shutdown: %v", err)
	}
	if processed.Load() != 5 {
		t.Fatalf("expected 5 jobs processed after a panic, got %d", processed.Load())
	}
}

Performance

Don't take specific numbers from a blog post as gospel here — the honest comparison is about mechanism and shape, not a benchmark table. Consider a burst of 100,000 jobs arriving in a few seconds.

With go process(job) per job, the runtime tries to schedule up to 100,000 goroutines close to simultaneously. Memory climbs roughly linearly with however much state each job holds open — connections, buffers, in-flight HTTP requests — because nothing prevents all of them from being "in progress" at once. If the jobs touch a database or an external API, those systems see a spike of concurrent load that looks nothing like their normal traffic pattern, and the failure tends to be sudden: fine at 80,000, falling over at 95,000.

With a bounded pool of 20 workers and a generous queue, the same 100,000 jobs arrive, sit in the channel buffer (a queue of Go values, not 100,000 live goroutines each holding job state), and get processed 20 at a time. Memory usage is roughly flat regardless of whether 1,000 or 100,000 jobs are queued, because only 20 are ever actively running. The database and any downstream API see a steady 20 concurrent callers instead of a spike. Throughput might be lower at any single instant than the unbounded version briefly achieves right before it falls over — but the pool keeps running at that rate indefinitely instead of crashing, which is the entire point.

Summary

go process(job) for every incoming job isn't wrong because goroutines are expensive — it's wrong because it puts no ceiling on concurrency, and every downstream system (memory, the database pool, external APIs) has one whether you set it deliberately or not. A worker pool — a fixed number of goroutines reading from a bounded channel, tracked with a WaitGroup, recovering from panics per job, and shut down via context cancellation with a timeout — makes that ceiling explicit and gives the system a way to push back instead of falling over. Size the pool against what the jobs actually do, watch queue depth as the signal to change it, and choose deliberately what happens when the queue is full — block, drop, or reject — rather than letting that decision default to whatever Submit happens to do.