# name: handling-async-messaging description: Designs event-driven messaging between services — topics/queues, transactional outbox, idempotent consumers, event schema versioning, ordering, and poison-message handling (Kafka, RabbitMQ, SNS/SQS, NATS). Use when the user asks to publish or consume events between services, decouple services with a message broker, design event schemas or topics, fix lost/duplicated/out-of-order events, or implement the outbox pattern. Do not use for in-process job queues inside one application (writing-background-jobs) or webhook delivery to external customers (designing-webhooks).

# Handling Async Messaging

# When to use / when NOT to use

• Use for: events and messages BETWEEN services: broker/topic design, producer and consumer contracts, delivery semantics, schema evolution.
• Do NOT use for: background jobs within one app (→ writing-background-jobs), webhooks to external customers (→ designing-webhooks), or choosing broker infrastructure sizing.

# Core rules

1. The contract is at-least-once + idempotent consumers. End-to-end exactly-once is a myth once side effects leave the broker; every consumer deduplicates on the event ID or a natural key.
   • ✅ if seen(event.id): ack(); return
   • ❌ assuming the broker's "exactly-once" flag makes handlers safe
2. Publish through a transactional outbox. Writing the DB and publishing to the broker are two systems — without an outbox one of them will lie after a crash. Same transaction: business write + outbox row; a relay publishes and marks it sent.
3. Events are past-tense facts, named for what happened: order_placed, payment_failed. An event commanding another service (create_shipment) is an RPC in disguise — if the producer needs a response or cares who handles it, make a direct call instead.
4. Schema changes are additive only. Add optional fields freely; renaming, retyping, or removing fields means a NEW topic/version (order_placed.v2) with both published during migration. Include event_id, occurred_at, and schema_version in every envelope.
5. Ordering exists only per key. Brokers guarantee order per partition/key at best; key by the entity (order_id) when sequence matters, and make consumers tolerate reordering across keys.
6. Poison messages go to a DLQ after N attempts (default 5) — never block the partition retrying forever, and alert on DLQ arrivals.
7. Consumers own their offset/ack discipline: ack only after side effects are durable. Ack-then-process converts every crash into silent data loss.
8. Design for replay. New consumers or bug fixes will re-read history; rule 1 makes replays safe, and time-sensitive handlers must check occurred_at before acting (don't send a "your order shipped" email from a 2-year-old event).

# Workflow

1. List the facts to publish (past-tense names), their producers, consumers, and the ordering key per topic.
2. Define envelopes: event_id, occurred_at, schema_version, payload of IDs + stable facts.
3. Implement the producer with a transactional outbox; implement consumers with dedup + ack-after-durable-side-effect.
4. Configure retries → DLQ (5 attempts) with alerting.
5. Validate: publish the same event twice and confirm one side effect; kill the consumer between side effect and ack, restart, and confirm no loss and no duplicate effect.

# Edge cases & failure modes

• Consumer needs data the event lacks → refetch from the producer's API by ID; do not fatten events into full snapshots reflexively.
• Broker down at publish time → the outbox absorbs it; the relay catches up. This is the pattern's main payoff.
• Two consumers in one service want the same topic → separate consumer groups; sharing a group splits the stream between them.
• Burst of replayed events floods a downstream dependency → consumers apply their own concurrency/rate limits.

# References

Outbox schema, envelope template, and dedup snippets: see references/patterns.md.