Event-Driven Architecture with Go: Events, Consumers and Failure Handling

The problem

An order service needs to charge a card, update inventory, and notify the customer. Wire all three as direct calls from the order handler and the order service now has to know about payment, inventory, and notifications — and if any one of them is slow or down, placing an order is slow or down too. Add a fourth thing that needs to happen when an order is created (a recommendation-engine update, an analytics event) and it's another direct call, another dependency, another way for POST /orders to fail for a reason that has nothing to do with the order itself.

Event-driven architecture inverts this: the order service announces OrderCreated happened and moves on. Whoever needs to react — now or later, one service or five — subscribes to that announcement. The order service never finds out who's listening, and doesn't need to.

Why it matters

The payoff isn't just decoupling for its own sake — it's that a new consumer (say, a fraud-detection service added six months later) can start reacting to OrderCreated without touching the order service's code at all. The cost is that "it happened" and "someone reacted to it" are no longer the same moment, which means the failure modes are different from a direct call, not absent. Understanding those failure modes — specifically, that a message can and will arrive more than once — is the actual skill this architecture requires.

Commands vs events

These are different things wearing similar clothing, and conflating them is where most event-driven designs go wrong:

A command says "do this." It's addressed to exactly one recipient, the sender expects a specific outcome, and if it fails the sender needs to know — ChargeCard is a command. Something is supposed to happen, and "nothing happened" is a failure the caller cares about.

An event says "this happened." It's broadcast, not addressed — the producer doesn't know or care who's listening, if anyone. OrderCreated is an event: the order already exists by the time it's published: the event is a fact, not a request.

Topics, partitions and consumer groups

A topic is a named stream of related events — orders.created, payments.completed. Producers publish to a topic; they don't address individual consumers.

A topic is split into partitions for parallelism — think of a partition as an ordered log that one consumer instance reads at a time. This is also the unit of ordering: events within one partition arrive in the order they were published; events across different partitions of the same topic have no ordering relationship to each other at all.

A consumer group is a set of consumer instances that split a topic's partitions between them — each partition is owned by exactly one consumer in the group at any moment, which is how you get parallel processing without two instances processing the same partition simultaneously and racing each other.

Event ordering

Because ordering is only guaranteed within a partition, anything that needs to happen in order relative to something else needs to land in the same partition. The standard technique is keying by an aggregate ID: every event for order ord_123 — created, updated, cancelled — is published with ord_123 as the partition key, so the broker's partitioning (typically a hash of the key) always routes them to the same partition, and a single consumer processes them in the order they happened.

What this doesn't give you: ordering between different orders, or between different aggregate types entirely. That's rarely something a consumer actually needs — "did order A's events arrive in order" is a meaningful question; "did order A's events arrive before order B's" almost never is.

Why events are delivered more than once

Most brokers default to at-least-once delivery, and this is worth internalizing precisely: a consumer reads a message, processes it (updates inventory, say), and then crashes — or the network blips — before it commits its offset back to the broker. From the broker's point of view, that message was never acknowledged. On restart, the consumer (or another instance in its group) gets the same message again.

This isn't a rare edge case or a sign something is misconfigured — it's the normal operating behavior of an at-least-once system. The alternative, exactly-once delivery, is either unavailable or comes with enough caveats and coordination overhead that most systems don't actually get it in practice; designing for at-least-once and making that safe is the more reliable engineering bet.

Idempotent consumers

The direct consequence: every consumer must be safe to run twice on the same event. This isn't optional hardening — it's the actual contract an at-least-once broker offers. Idempotent Consumers in Go covers the implementation directly (a processed-events table with a unique constraint on the event ID is the usual mechanism); the point here is architectural: don't design a consumer's business logic assuming single delivery, because that assumption will be wrong in production, not just in theory.

Retries and dead-letter queues

A consumer that fails to process a message — a transient database error, a downstream timeout — should retry with backoff rather than either dropping the message or blocking the partition in a tight retry loop; see Retries, Timeouts and Backoff in Go for the mechanics, which apply here directly.

But retries need a ceiling. A message that fails every single time — a malformed payload, a permanently missing referenced record — will retry forever if nothing stops it, and because it's stuck at the front of its partition, every event behind it is stuck too:

func (c *Consumer) handle(ctx context.Context, msg Message) error {
	err := c.process(ctx, msg)
	if err == nil {
		return nil
	}
 
	if msg.DeliveryCount >= maxRetries {
		return c.deadLetter.Publish(ctx, msg, fmt.Errorf("exceeded %d attempts: %w", maxRetries, err))
	}
 
	return err // let the broker's retry/redelivery mechanism take over
}

Once a message exceeds its retry budget, moving it to a dead-letter queue — a separate topic or queue holding only failed messages — lets the rest of the partition keep flowing while the poisoned message waits for a human to look at it, instead of either blocking everything behind it or silently vanishing.

How this connects to the Outbox Pattern

Everything above assumes the order service reliably published OrderCreated in the first place. That's a separate problem — The Outbox Pattern with Go and PostgreSQL covers the producer side: writing the order and the "I need to publish this" record in the same database transaction, so an order can't exist without its event eventually being published, and vice versa. This article is what happens after that event reaches the bus; the outbox pattern is how it reliably gets there.

Observability for event-driven systems

The moment a request's effects span several services reacting asynchronously, there's no single call stack to read anymore — a slow or failed reaction shows up in whichever service is doing the reacting, possibly minutes after the original request. A correlation ID carried in every event's metadata, propagated the way Observability for Go Microservices describes, is what lets you reconstruct "everything that happened because of this one order" across services after the fact — without it, debugging an event-driven system means guessing which service's logs to check first.

Common mistakes

• Treating a topic like an RPC call. Publishing an event and expecting a specific downstream outcome to happen synchronously (or at all, on any particular timeline) reintroduces the coupling events are supposed to remove.
• Assuming ordering across partitions. Ordering is a per-partition guarantee; a design that needs global ordering across a whole topic needs a single partition (giving up parallelism) or a different approach entirely.
• No dead-letter path. Without one, a single malformed message can stall an entire partition indefinitely, silently, until someone notices consumer lag climbing.
• Consumers that aren't idempotent, discovered the first time a rebalance or a crash causes a redelivery in production rather than in a design review.

Summary

Event-driven architecture trades direct, synchronous coupling for decoupled, asynchronous broadcast — a real gain for extensibility, at the cost of a different set of failure modes. Ordering is scoped to a partition, delivery is at-least-once by default (meaning duplicates are normal, not exceptional), and a dead-letter queue is what keeps one bad message from stalling everything behind it. Get the producer side right with the outbox pattern, get the consumer side right by making every handler idempotent, and the asynchrony becomes a feature instead of a source of silent data corruption.