Load Testing a Go API: Finding the Real Bottleneck

The problem

A Go API passes every unit test, handles a handful of manual requests instantly, and gets deployed. Then real traffic arrives — hundreds of concurrent users instead of one curl request at a time — and it falls over in a way none of that testing predicted: not necessarily crashing, but getting slow enough that users give up, retries pile on top of already-slow requests, and the on-call engineer is staring at a graph with no idea which of six possible systems is actually the problem.

Load testing exists to find that failure point before production traffic finds it for you, and to answer a much harder question than "does it work": how does it behave as concurrency increases, and when it does slow down, what is actually the limiting resource?

Why it matters

A service that has never been load tested has an unknown breaking point. That's not a hypothetical risk — it shows up in specific, predictable ways:

• Capacity planning is a guess. Without a measured requests-per-second ceiling, "can we handle Black Friday" or "can we handle this new customer's traffic" has no real answer.
• The wrong fix gets shipped. Teams under pressure during an incident often guess at the bottleneck — "let's add more app replicas" — when the actual constraint is a database connection pool or a single slow query that more app replicas will only make worse.
• Average latency hides the real user experience. A dashboard showing "120ms average" can coexist with a meaningful fraction of requests taking 2+ seconds, and nobody notices until support tickets start arriving.

Load testing vocabulary

Throughput is the request rate the system sustains — requests per second (RPS) — under a given load. Latency is the time per request. Concurrency is the load-testing knob: how many virtual users (VUs) are hitting the system at once, each firing requests independently.

Latency should never be summarized as a single average. Report it as percentiles:

• p50 (median) — half of requests are faster than this. Good for "typical" experience, bad for judging tail risk.
• p95 — 95% of requests are faster than this; 1 in 20 is slower. This is where real, non-rare pain starts showing up.
• p99 — 1 in 100 requests is slower than this. At low traffic this looks like a rounding error. At 1,000 requests/second, a p99 of 2 seconds means 10 requests every single second take 2+ seconds — not an edge case, a constant stream of bad experiences.

The remaining signals: error rate (the percentage of requests failing — timeouts, 5xx responses, connection refusals) and saturation (how close a finite resource — CPU, memory, DB connections — is to its limit). Rising latency with a flat error rate means the system is slowing down but still coping. Rising latency with a rising error rate means you've found a real limit, not just added harmless load.

A load test with k6

k6 is a scriptable load-testing tool: test scenarios are written in JavaScript, executed by a Go-based load generator. A scenario against the GoBackend Starter's GET /api/v1/posts endpoint, ramping up to 100 concurrent users:

import http from "k6/http";
import { check, sleep } from "k6";
 
export const options = {
  stages: [
    { duration: "30s", target: 20 },   // warm up
    { duration: "1m", target: 100 },   // ramp to 100 VUs
    { duration: "2m", target: 100 },   // hold at 100 VUs
    { duration: "30s", target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_failed: ["rate<0.01"],       // fail the test if error rate > 1%
    http_req_duration: ["p(95)<300"],     // fail the test if p95 > 300ms
  },
};
 
const BASE_URL = __ENV.TARGET_URL || "http://localhost:8080";
 
export default function () {
  const res = http.get(`${BASE_URL}/api/v1/posts?limit=20`);
 
  check(res, {
    "status is 200": (r) => r.status === 200,
    "has posts array": (r) => JSON.parse(r.body).posts !== undefined,
  });
 
  sleep(1);
}

k6 run --env TARGET_URL=https://staging.example.com load-test.js

k6's summary reports exactly the numbers that matter:

     http_req_duration..............: avg=48.2ms  min=6.1ms  med=31.4ms  max=2.41s  p(90)=89.3ms  p(95)=142.7ms
     http_req_failed................: 0.42%   ✓ 42        ✗ 9958
     http_reqs......................: 10000   166.4/s

The thresholds block matters as much as the script itself — it turns a load test into a pass/fail gate you can run in CI against a staging environment, not just a number someone eyeballs once.

What to measure beyond client-side latency

k6 tells you what the client experienced. It doesn't tell you why. To find the real bottleneck you need server-side signals collected during the same run:

• CPU and memory on the Go service — is the process actually CPU-bound, or sitting idle waiting on something else?
• Database connection pool stats — is the pool exhausted, with requests queuing on pool.Acquire? See PostgreSQL Connection Pooling in Go for what pool.Stat() exposes and what healthy versus exhausted looks like.
• Correlated logs and traces for the slow requests specifically — not just "it was slow," but which downstream call inside that request took the time. This is exactly the correlation workflow described in Observability for Go Microservices.

Without these, a load test just confirms something is slow. With them, it tells you what to fix.

Finding the real bottleneck

Treat it as elimination, not guesswork:

• App CPU pegged near 100% → the bottleneck is in Go code itself. Stop guessing and profile it — see Profiling Go Applications with pprof.
• DB pool showing high AcquireDuration or AcquireCount growth → requests are queuing for a database connection. That's either a pool sized too small for the concurrency, or queries holding connections longer than they should.
• A specific query dominating time → run EXPLAIN ANALYZE in PostgreSQL against it — this is a different problem from pool sizing and a bigger pool won't fix it.
• A downstream API call is the slow part → the Go service is healthy; the dependency isn't. That changes the fix entirely (timeout budget, caching, or a circuit breaker) from anything above.
• None of the above, but the host is capped → container CPU/memory limits or network bandwidth on the load-generator side can bottleneck the test, not the service — always sanity-check the load generator itself isn't the constraint.

Before/after: a connection pool bottleneck

A realistic sequence: the GET /api/v1/posts endpoint holds up fine at 20 VUs, but at 100 VUs p99 balloons past 2 seconds.

Before — pgxpool.Config{MaxConns: 5}, left at a value nobody revisited since local development:

http_req_duration: avg=340ms  p(95)=1.8s   p(99)=2.6s
http_req_failed:   3.1%

Correlated pool.Stat() during the same run shows AcquireDuration climbing steadily and TotalConns pinned at 5 — every request beyond the first five is queuing for a connection, not doing real work.

Fix — raise MaxConns to a value sized for the actual concurrency and the database's real max_connections headroom (per the tuning guidance in the connection pooling article), not an arbitrary bigger number.

After — same script, same 100 VUs:

http_req_duration: avg=52ms   p(95)=118ms  p(99)=189ms
http_req_failed:   0.02%

The fix came from the pool statistics, not from the k6 output alone — the client-side numbers said "it's slow," the server-side signal said why.

Why p95/p99 matter more than average

Optimizing against the average alone can make things worse in a way that looks like an improvement on a dashboard. A cache layer, for example, drops the average latency dramatically because most requests now hit cache — but if it introduces periodic cache-stampede spikes on expiry, the p99 can get worse even as the average improves. Anyone watching only the average ships that change as a win. Anyone watching p95/p99 and error rate together catches the regression before it reaches production. Always evaluate a performance change against both ends of the distribution, not just the middle.

Common mistakes

• Testing against a warm cache or warm connection pool that doesn't exist for a real cold start or a fresh deploy, producing numbers that are optimistic compared to actual production behavior.
• Chasing throughput while ignoring error rate. A test that "handles" 500 req/s by shedding 15% of requests as errors hasn't found a successful capacity — it's found a failure mode.
• Running the load generator somewhere network-constrained (a laptop on Wi-Fi, an under-provisioned test VM) so the generator itself becomes the bottleneck being measured, not the service under test.
• Load testing once and never again. Capacity and bottlenecks shift every time the code, the data volume, or the traffic pattern changes; a load test from six months ago tells you about a system that no longer exists.

Summary

Load testing answers a question unit tests can't: how does the system behave under real concurrency, and where does it actually break? Measure percentiles, not averages — p95 and p99 are what real users experience at scale, not edge cases. Pair client-side numbers from a tool like k6 with server-side signals (CPU, database pool stats, correlated traces) so a slow test run points at a specific fix instead of a guess, and re-run the test after every fix to confirm it actually moved the tail, not just the mean.