# name: designing-webhooks description: Designs outbound webhook systems with signed deliveries, exponential-backoff retries, delivery IDs for consumer idempotency, endpoint verification, and dead-letter handling under explicit at-least-once semantics. Use when the user asks to design, build, or review webhooks or event notifications to external consumers, sign or verify webhook payloads, add webhook retries, or document webhook delivery guarantees. Do not use for internal service-to-service queues or event buses (handling-async-messaging) or for the business logic of consuming a third party's webhooks.

# Designing Webhooks

# When to use / when NOT to use

• Use for: designing the producer side of webhooks (payloads, signing, retries, delivery semantics, subscriber management) and the receiver checklist you publish to consumers.
• Do NOT use for: internal async messaging between your own services → handling-async-messaging; what a consumer's handler should do with a specific third-party event.

# Core rules

1. Events are facts, named resource.action in past tense. invoice.paid, user.deleted — not commands (send_email) and not ambiguous (update).
2. Default to thin payloads. Send id, type, created_at, and the resource ID; the consumer fetches the current state via API. Fat payloads only for non-sensitive, low-churn data — a fat payload of stale or sensitive data is a liability.
   • ✅ {"type": "invoice.paid", "data": {"invoice_id": "inv_42"}}
   • ❌ full invoice with customer PII in every delivery
3. Sign every delivery: HMAC-SHA256 over timestamp + "." + body, sent as a header (Webhook-Signature: t=...,v1=...). Receivers must reject signatures older than 5 minutes (replay window) and compare in constant time.
4. State the guarantee plainly: at-least-once, unordered. Consumers WILL receive duplicates and out-of-order events. Publish this in the docs; never promise exactly-once or ordering you can't enforce.
5. Every delivery carries a unique delivery_id (stable across retries of the same event). Consumers deduplicate on it; your docs must show the dedup pattern.
6. Retry on failure with exponential backoff + jitter, bounded. Non-2xx or >10 s timeout → retry at ~1 min, 5 min, 30 min, 2 h, 12 h (5 attempts). After the last failure, park the delivery in a dead-letter store and surface it in the dashboard/API — never drop silently.
7. Verify endpoints before sending real events. On subscription, send a challenge the consumer must echo (or a signed ping they must 2xx). Auto-disable endpoints failing >7 days and notify the owner.
8. 2xx means accepted, nothing else. Consumers should enqueue and return 200 immediately; producers treat 3xx/4xx/5xx and slow responses identically — as failures to retry.

# Workflow

1. Enumerate events (rule 1) and choose thin/fat per event (rule 2).
2. Define the envelope: delivery_id, type, created_at, data.
3. Implement signing (rule 3) and the verification handshake (rule 7).
4. Implement the retry schedule and dead-letter store (rule 6).
5. Write the consumer documentation: signature check, 5-minute skew rejection, dedup on delivery_id, at-least-once warning (rules 3–5).
6. Validate: deliver a test event to a sample receiver, then force-redeliver the same event — the receiver's side effect must occur exactly once (dedup works) and both deliveries must verify the signature. Break the secret and confirm rejection.

# Edge cases & failure modes

• Consumer endpoint is down for a day → retries cover ~14.5 h; dead-letter after that with manual/API redelivery available.
• Secret rotation → support two active secrets per endpoint (v1 old + v1 new signatures during overlap) so consumers rotate without dropped events.
• Event storm (bulk import) → per-endpoint delivery rate cap and warn subscribers; do not interleave retries ahead of fresh events indefinitely — cap queue age.
• Consumer needs ordering → they must reorder on created_at/sequence in their own store; offer a GET /events?after= reconciliation API as the source of truth.

# References

Deeper patterns (envelope JSON, signing/verification code, retry table, receiver checklist): see references/patterns.md.