Profiling Go Applications with pprof

The problem

"This endpoint is slow" or "this service uses too much memory" are not actionable on their own. Guessing at the cause — maybe it's the JSON encoding, maybe it's a lock, maybe it's GC pressure — wastes time and often leads to optimizing the wrong thing. Go ships a profiler in the standard library specifically so you don't have to guess: pprof samples a running program and tells you exactly where CPU time and memory are actually going.

Why it matters

Profiling data changes what you optimize, and by how much:

• A function that looks expensive in code review might be 0.3% of CPU time in practice — optimizing it does nothing for the p99 latency that triggered the investigation.
• Memory growth is often not a leak but excessive allocation in a hot path — the fix is reducing allocations, not chasing a nil pointer that was never actually held.
• Goroutine counts that climb over time point at a specific blocked or leaked goroutine pattern, which a goroutine profile identifies directly instead of by code inspection.

Without a profile, performance work is trial and error against production traffic. With one, it's targeted at the function and line actually responsible.

Wiring pprof into an HTTP service

Importing net/http/pprof for its side effect registers profiling endpoints on the default mux. In production, serve them on a separate, non-public port rather than the main API mux:

import (
	"log"
	"net/http"
	_ "net/http/pprof"
)
 
func main() {
	// Debug/profiling server — bind to localhost or a private network only.
	go func() {
		log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
	}()
 
	startAPIServer()
}

This exposes /debug/pprof/ with profiles for CPU, heap, goroutines, blocking, and mutex contention, all fetchable with the go tool pprof command while the process is running.

CPU profiling

A CPU profile samples the call stack at a fixed rate over a time window and reports where the program actually spent time:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

This blocks for 30 seconds while the profiler samples a live workload — generate real or representative traffic against the service during that window, or the profile will just show it idling. Once it drops into the interactive pprof shell:

(pprof) top10
Showing nodes accounting for 2.1s, 84% of 2.5s total
      flat  flat%   sum%        cum   cum%
     0.9s  36.0%  36.0%      1.2s  48.0%  encoding/json.Marshal
     0.4s  16.0%  52.0%      0.4s  16.0%  runtime.mallocgc
     0.3s  12.0%  64.0%      0.3s  12.0%  crypto/sha256.block
     ...

flat is time spent in that function itself; cum (cumulative) includes time spent in everything it calls. A high cum with low flat on json.Marshal here means the real cost is in what's being marshaled — large structs, reflection overhead, or marshaling more data than the caller needs. top10 -cum sorts by cumulative time when you want to find the most expensive call path rather than the most expensive individual function.

For a visual view of the call graph, go tool pprof -http=:8081 profile.pb.gz renders an interactive flame graph in the browser — usually faster to read than scrolling top output for anything beyond the first few entries.

Memory profiling

The heap profile shows what's currently allocated and, more usefully for finding allocation hot paths, -alloc_objects / -alloc_space show cumulative allocations since the process started:

go tool pprof http://localhost:6060/debug/pprof/heap

(pprof) top10 -cum
      flat  flat%   sum%        cum   cum%
     512kB  20.0%  20.0%     1.8MB  70.0%  main.(*OrderHandler).ListOrders
     380kB  15.0%  35.0%     1.1MB  43.0%  encoding/json.Marshal
     ...

A function showing up here with high cumulative allocation but no matching business reason to allocate that much is the usual signal: unnecessary copies (passing large structs by value through several layers), building intermediate slices instead of streaming, or repeated small allocations inside a loop that could be reused via a sync.Pool.

Goroutine profiling

The goroutine profile lists every currently running goroutine and its stack:

go tool pprof http://localhost:6060/debug/pprof/goroutine

A goroutine count that only grows — visible by comparing this profile's total over time, or via the debug=1 text output at /debug/pprof/goroutine?debug=1 — almost always means goroutines are being started without a corresponding path to finish: a context that's never canceled, a channel send with no receiver, or a for loop over a channel that's never closed.

Interpreting profiles correctly

A profile only reflects the workload during the sampling window. A CPU profile taken while the service is idle tells you nothing about the endpoint you actually care about — generate load matching the scenario you're investigating.

flat vs cum answer different questions. Use flat to find the specific function burning CPU; use cum to find the most expensive call path, which might point at a caller doing something wasteful (calling an otherwise-cheap function too many times) rather than the function itself being slow.

Allocations aren't automatically a problem. The Go garbage collector is built to handle allocation; the actual cost is GC pressure at scale. Compare allocation profiles before and after a change to confirm an optimization reduced allocations, rather than assuming any non-zero number in a heap profile needs fixing.

Common performance mistakes pprof reveals

• Unnecessary JSON re-marshaling — encoding the same data multiple times across middleware and handlers instead of once.
• String concatenation in a loop instead of strings.Builder, showing up as repeated allocations in the heap profile.
• Interface boxing in hot paths — passing small values through interface{} parameters where a concrete type would avoid an allocation.
• Unbounded goroutine growth from a fan-out pattern with no worker-pool limit, visible as a steadily increasing goroutine count.
• Holding a mutex longer than necessary, visible in the mutex contention profile at /debug/pprof/mutex (enabled via runtime.SetMutexProfileFraction).

Profiling in a test or benchmark

For code that isn't behind an HTTP server yet, Go's benchmarking tool produces profiles directly, which is often a faster loop than wiring up net/http/pprof for a small, isolated hot path:

func BenchmarkMarshalOrder(b *testing.B) {
	order := sampleOrder()
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		if _, err := json.Marshal(order); err != nil {
			b.Fatal(err)
		}
	}
}

go test -bench=MarshalOrder -cpuprofile=cpu.prof -memprofile=mem.prof
go tool pprof cpu.prof

Summary

pprof turns "this feels slow" into a specific function, call path, or allocation site backed by real sampled data from a running program. Wire net/http/pprof behind a private port in every service, profile under realistic load rather than idle traffic, and let flat/cum and allocation deltas — not intuition — decide what to optimize next.