---
name: handling-async-messaging
description: Designs event-driven messaging between services — topics/queues, transactional outbox, idempotent consumers, event schema versioning, ordering, and poison-message handling (Kafka, RabbitMQ, SNS/SQS, NATS). Use when the user asks to publish or consume events between services, decouple services with a message broker, design event schemas or topics, fix lost/duplicated/out-of-order events, or implement the outbox pattern. Do not use for in-process job queues inside one application (writing-background-jobs) or webhook delivery to external customers (designing-webhooks).
---

<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Handling Async Messaging

## When to use / when NOT to use
- **Use for:** events and messages BETWEEN services: broker/topic design, producer and consumer contracts, delivery semantics, schema evolution.
- **Do NOT use for:** background jobs within one app (→ writing-background-jobs), webhooks to external customers (→ designing-webhooks), or choosing broker infrastructure sizing.

## Core rules

1. **The contract is at-least-once + idempotent consumers.** End-to-end exactly-once is a myth once side effects leave the broker; every consumer deduplicates on the event ID or a natural key.
   - ✅ `if seen(event.id): ack(); return`
   - ❌ assuming the broker's "exactly-once" flag makes handlers safe
2. **Publish through a transactional outbox.** Writing the DB and publishing to the broker are two systems — without an outbox one of them will lie after a crash. Same transaction: business write + outbox row; a relay publishes and marks it sent.
3. **Events are past-tense facts, named for what happened:** `order_placed`, `payment_failed`. An event commanding another service (`create_shipment`) is an RPC in disguise — if the producer needs a response or cares who handles it, make a direct call instead.
4. **Schema changes are additive only.** Add optional fields freely; renaming, retyping, or removing fields means a NEW topic/version (`order_placed.v2`) with both published during migration. Include `event_id`, `occurred_at`, and `schema_version` in every envelope.
5. **Ordering exists only per key.** Brokers guarantee order per partition/key at best; key by the entity (`order_id`) when sequence matters, and make consumers tolerate reordering across keys.
6. **Poison messages go to a DLQ after N attempts** (default 5) — never block the partition retrying forever, and alert on DLQ arrivals.
7. **Consumers own their offset/ack discipline:** ack only after side effects are durable. Ack-then-process converts every crash into silent data loss.
8. **Design for replay.** New consumers or bug fixes will re-read history; rule 1 makes replays safe, and time-sensitive handlers must check `occurred_at` before acting (don't send a "your order shipped" email from a 2-year-old event).

## Workflow

1. List the facts to publish (past-tense names), their producers, consumers, and the ordering key per topic.
2. Define envelopes: `event_id`, `occurred_at`, `schema_version`, payload of IDs + stable facts.
3. Implement the producer with a transactional outbox; implement consumers with dedup + ack-after-durable-side-effect.
4. Configure retries → DLQ (5 attempts) with alerting.
5. **Validate:** publish the same event twice and confirm one side effect; kill the consumer between side effect and ack, restart, and confirm no loss and no duplicate effect.

## Edge cases & failure modes
- **Consumer needs data the event lacks** → refetch from the producer's API by ID; do not fatten events into full snapshots reflexively.
- **Broker down at publish time** → the outbox absorbs it; the relay catches up. This is the pattern's main payoff.
- **Two consumers in one service want the same topic** → separate consumer groups; sharing a group splits the stream between them.
- **Burst of replayed events floods a downstream dependency** → consumers apply their own concurrency/rate limits.

## References
Outbox schema, envelope template, and dedup snippets: see [references/patterns.md](references/patterns.md).