Observability for Go Microservices: Logs, Metrics and Traces

The problem

An order service calls a payment service. Latency on POST /orders climbs from 80ms to 900ms for a slice of traffic. Your dashboards show CPU, memory, and request count all look normal. Nothing crashed. No alert fired on anything you thought to watch for. Somewhere between "order service" and "payment service," a specific class of request is slow, and none of the metrics you set up in advance tell you which one, for which customers, or why.

This is the gap between monitoring and observability, and it's the reason "we have Grafana dashboards" and "we can debug production incidents" are not the same claim.

Why it matters

Monitoring is watching for failure modes you already anticipated. "Alert if CPU exceeds 80%." "Alert if the error rate exceeds 1%." It answers questions you thought to ask before the incident happened.

Observability is having enough data emitted from the system — logs, metrics, and traces, correlated with each other — that you can answer questions you did not anticipate, after the fact, by querying and cross-referencing what the system already recorded.

Concretely: a metric tells you p99 latency on /orders spiked at 14:32. That's monitoring — you knew to watch p99 latency. Observability is being able to then ask "which specific requests," find that they all share one merchant_id, pull the traces for those requests, see that each one spends 800ms in a span calling the payment service's /verify endpoint, and pull the correlated logs for that span to see the payment service was retrying a timed-out call to a card network. You didn't predict "one merchant's payment verification will be slow" — you didn't need to, because the system gave you enough correlated detail to find it.

Logs

Logs are for what happened, in detail, for one specific event.

Structured, not string-concatenated. log.Printf("order %s failed: %v", id, err) is unsearchable at scale. Building a Production-Ready REST API with Go already uses log/slog for exactly this reason — the fields are queryable, not buried in a formatted string:

logger.Error("payment verification failed",
	"order_id", orderID,
	"merchant_id", merchantID,
	"correlation_id", correlationID,
	"error", err,
)

Request ID vs. correlation ID. A request ID is scoped to one service's handling of one request — useful for tying together the log lines emitted while that service processes it. A correlation ID is generated once, at the edge, and propagated across every service call the request triggers, so a single user action produces log lines in the order service and the payment service that share one value you can filter on. Without it, correlating a slow order to the payment service call that caused it means guessing at timestamps.

Log levels, applied with discipline. debug for detail you only want when actively troubleshooting; info for events worth knowing happened (order created, payment captured) — not every function entry; warn for recovered/degraded situations (a retry succeeded on the second attempt); error for failures that need attention. A service that logs every request body at info produces so much volume that the signal — the actual errors — gets expensive to search and easy to miss.

Useful fields, not a wall of text. Timestamp, level, service name, correlation ID, and the specific structured fields relevant to that event. A log line's job is to be filtered and joined, not read top to bottom.

Metrics

Metrics are for aggregate behavior over time, cheap to store and alert on.

The signals worth instrumenting on nearly every service:

• Request count — by endpoint and status code.
• Latency — as a histogram, not an average. An average of 50ms with a p99 of 4 seconds looks fine on a dashboard that only shows the average; the 1% of users hitting 4 seconds don't experience an average. Load Testing a Go API covers why percentiles matter for this in more depth — the same reasoning applies to production metrics, not just load test results.
• Error rate — as a ratio of failed to total requests, per endpoint, so one noisy endpoint doesn't hide in an overall average.
• Saturation — queue depth, database connection pool utilization (see PostgreSQL Connection Pooling in Go), goroutine count. Saturation metrics tell you the system is about to have a problem, before latency and errors make it obvious.

Prometheus is the standard way to expose and scrape these in a Go service (a /metrics endpoint, scraped on an interval), and Grafana is the standard way to visualize them. The tooling matters far less than picking the right four signals above and instrumenting them consistently across every service.

Traces

A trace represents one end-to-end request as it moves across services. A span represents one unit of work within it — the order service's handler is one span; its call to the payment service is a child span; the payment service's own handler, once the trace context crosses the network, is another child span nested under that.

Tracing across services requires propagating the trace context, usually via HTTP headers, so the payment service's spans get attached to the same trace the order service started rather than starting an unrelated one:

func (c *PaymentClient) Verify(ctx context.Context, orderID string) error {
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/verify", body)
	if err != nil {
		return err
	}
	// The tracing SDK's transport wraps http.RoundTripper and injects the
	// trace context into headers here — the application code just needs
	// to pass the request's context through, the same discipline
	// described in [Context in Go](/go/context-in-go).
	resp, err := c.httpClient.Do(req)
	...
}

OpenTelemetry is the standard instrumentation layer for this in Go — it provides the SDK for creating spans and the propagation format for carrying trace context across HTTP calls. The valuable part isn't the specific SDK; it's that a request's full path across every service it touched becomes visible as one connected structure instead of N separate, uncorrelated sets of logs.

How the three pillars work together

This is what makes the monitoring-vs-observability distinction concrete rather than theoretical. A real incident tends to move through all three:

1. A metric alert fires — error rate on /orders crossed 2%. This is monitoring: a threshold you defined in advance.
2. You find the affected traces — filtering traces for /orders in the alert's time window with an error status, you see they share a slow child span calling the payment service's /verify endpoint.
3. You pull the correlated logs for one of those specific traces using its correlation ID, and see the payment service logged a timeout retrying a call to a card network.

None of those three steps alone finds the root cause. The metric tells you something is wrong; the trace tells you where; the log tells you what specifically. That composition — not any single pillar — is what "observability" actually buys you over a dashboard full of averages.

Production considerations

Log volume has a real cost. At high request volume, logging every request at info gets expensive to store and slow to search. Sample verbose logging, or lower the default level, and rely on the ability to temporarily raise it (a runtime-configurable log level) when actively debugging.

Sample traces at high throughput. Capturing 100% of traces on a high-QPS service adds overhead and cost with rapidly diminishing returns — a representative sample (or 100% of error traces, sampled lower for successful ones) usually gives you what you need during an incident without the overhead of tracing every request forever.

Watch metric cardinality before it watches you. This is worth repeating: an unbounded label value (user ID, order ID, raw URL path with an embedded ID) turns one metric into an unbounded number of time series. This is one of the most common ways teams accidentally take down their own observability stack.

Common mistakes

• Logging everything at info with no correlation ID — producing volume without the one field that would make it useful for tracing a single request across services.
• Alerting only on averages — an average latency metric hides exactly the tail behavior (p95/p99) that affects real users first.
• Adding tracing only after an incident exposed a gap — by definition, the next unanticipated failure mode is the one you still don't have visibility into. The pillars are worth instrumenting before you need them, not after.
• Treating the three pillars as independent projects — logs from one team's dashboard, metrics from another, no shared correlation ID connecting them — instead of one coherent, correlated system.

Summary

Monitoring watches for the failure modes you thought to anticipate; observability is having enough correlated logs, metrics, and traces that you can diagnose the ones you didn't. Structured logs with a correlation ID answer "what happened, specifically." Metrics — tracked as percentiles, not averages, with bounded label cardinality — answer "is something wrong, and how bad." Traces answer "where, across which services." None of them replaces the others; a real incident is usually resolved by moving between all three.