--- name: writing-background-jobs description: Designs and implements background jobs and task queues that survive retries, crashes, and duplicate delivery — idempotency, backoff, dead-letter queues, timeouts, and job observability. Use when the user asks to write a background job, worker, or task queue, move work out of a request (send emails, process images, sync data), add retries to a job, or fix duplicate/stuck jobs (Celery, Sidekiq, BullMQ, RQ, or hand-rolled workers). Do not use for inter-service event/messaging architecture (handling-async-messaging) or cron-style system administration. --- # Writing Background Jobs ## When to use / when NOT to use - **Use for:** designing or reviewing background jobs, workers, and task queues inside one application: enqueueing, retries, failure handling, monitoring. - **Do NOT use for:** events between services (→ handling-async-messaging), customer-facing webhook delivery (→ designing-webhooks), or OS-level cron administration. ## Core rules 1. **Every job is idempotent.** Queues deliver at-least-once; your job WILL run twice. Guard side effects with a natural key or an idempotency record. - ✅ `INSERT ... ON CONFLICT (payment_id) DO NOTHING`, then act only on inserted rows - ❌ `charge_card(amount)` executed unconditionally at the top of the job 2. **Payloads carry IDs, not objects.** Refetch current state at run time; serialized objects go stale between enqueue and execution. - ✅ `enqueue(send_invoice, invoice_id=42)` - ❌ `enqueue(send_invoice, invoice=invoice.to_json())` 3. **Explicit retry policy on every job:** exponential backoff with jitter, capped attempts. Default: base 30 s, factor 2, full jitter, max 5 attempts. Distinguish retryable (network, 5xx) from permanent (validation) failures — permanent failures skip retries and go straight to the dead-letter queue. 4. **Dead-letter queue with an alert.** Exhausted jobs land in a DLQ that pages someone; a silent DLQ is a data-loss buffer. 5. **Timeout on every job.** No default-infinite jobs: set an explicit per-job timeout slightly above p99 runtime, and make the handler kill-safe (rule 1 covers the rerun). 6. **No shared mutable state between jobs.** Workers run concurrently across processes and hosts; coordinate through the database (row locks, `SELECT ... FOR UPDATE SKIP LOCKED`) — never through process memory or files. 7. **Long work = chain of short jobs.** Split anything over ~1 minute into resumable steps (paginate by cursor, one page per job). Short jobs retry cheaply; hour-long jobs lose an hour per crash. 8. **Instrument jobs like endpoints:** duration, success/failure counters per job type, queue depth and oldest-message age with alerts. Queue depth growing while workers are idle means a poison job or a crashed consumer. ## Workflow 1. Define the job: trigger, payload (IDs only), side effects, and the idempotency key for each side effect. 2. Classify failures as retryable vs permanent; set backoff (30 s base, ×2, jitter, 5 attempts) and timeout (≈ p99 runtime + margin). 3. Implement with the project's existing queue library — do not introduce a new one if any queue is already present. 4. Wire the DLQ and its alert; add duration/failure metrics. 5. **Validate:** run the job twice with the same payload and confirm exactly one side effect; kill the worker mid-job and confirm the retry completes cleanly. ## Edge cases & failure modes - **Job depends on uncommitted data** → enqueue after commit (transactional enqueue or on-commit hook), or the worker races the transaction and sees nothing. - **Poison message** (crashes the worker every time) → attempts cap sends it to the DLQ; never retry forever. - **Queue backlog after outage** → workers must tolerate a thundering herd: keep rule 1, add concurrency limits per job type. - **Scheduled (cron-like) jobs double-fire** on overlapping schedules → take a distributed lock keyed by job name + period before running. ## References Deeper recipes and library-specific snippets: see [references/patterns.md](references/patterns.md).