# name: writing-background-jobs description: Designs and implements background jobs and task queues that survive retries, crashes, and duplicate delivery — idempotency, backoff, dead-letter queues, timeouts, and job observability. Use when the user asks to write a background job, worker, or task queue, move work out of a request (send emails, process images, sync data), add retries to a job, or fix duplicate/stuck jobs (Celery, Sidekiq, BullMQ, RQ, or hand-rolled workers). Do not use for inter-service event/messaging architecture (handling-async-messaging) or cron-style system administration.

# Writing Background Jobs

# When to use / when NOT to use

• Use for: designing or reviewing background jobs, workers, and task queues inside one application: enqueueing, retries, failure handling, monitoring.
• Do NOT use for: events between services (→ handling-async-messaging), customer-facing webhook delivery (→ designing-webhooks), or OS-level cron administration.

# Core rules

1. Every job is idempotent. Queues deliver at-least-once; your job WILL run twice. Guard side effects with a natural key or an idempotency record.
   • ✅ INSERT ... ON CONFLICT (payment_id) DO NOTHING, then act only on inserted rows
   • ❌ charge_card(amount) executed unconditionally at the top of the job
2. Payloads carry IDs, not objects. Refetch current state at run time; serialized objects go stale between enqueue and execution.
   • ✅ enqueue(send_invoice, invoice_id=42)
   • ❌ enqueue(send_invoice, invoice=invoice.to_json())
3. Explicit retry policy on every job: exponential backoff with jitter, capped attempts. Default: base 30 s, factor 2, full jitter, max 5 attempts. Distinguish retryable (network, 5xx) from permanent (validation) failures — permanent failures skip retries and go straight to the dead-letter queue.
4. Dead-letter queue with an alert. Exhausted jobs land in a DLQ that pages someone; a silent DLQ is a data-loss buffer.
5. Timeout on every job. No default-infinite jobs: set an explicit per-job timeout slightly above p99 runtime, and make the handler kill-safe (rule 1 covers the rerun).
6. No shared mutable state between jobs. Workers run concurrently across processes and hosts; coordinate through the database (row locks, SELECT ... FOR UPDATE SKIP LOCKED) — never through process memory or files.
7. Long work = chain of short jobs. Split anything over ~1 minute into resumable steps (paginate by cursor, one page per job). Short jobs retry cheaply; hour-long jobs lose an hour per crash.
8. Instrument jobs like endpoints: duration, success/failure counters per job type, queue depth and oldest-message age with alerts. Queue depth growing while workers are idle means a poison job or a crashed consumer.

# Workflow

1. Define the job: trigger, payload (IDs only), side effects, and the idempotency key for each side effect.
2. Classify failures as retryable vs permanent; set backoff (30 s base, ×2, jitter, 5 attempts) and timeout (≈ p99 runtime + margin).
3. Implement with the project's existing queue library — do not introduce a new one if any queue is already present.
4. Wire the DLQ and its alert; add duration/failure metrics.
5. Validate: run the job twice with the same payload and confirm exactly one side effect; kill the worker mid-job and confirm the retry completes cleanly.

# Edge cases & failure modes

• Job depends on uncommitted data → enqueue after commit (transactional enqueue or on-commit hook), or the worker races the transaction and sees nothing.
• Poison message (crashes the worker every time) → attempts cap sends it to the DLQ; never retry forever.
• Queue backlog after outage → workers must tolerate a thundering herd: keep rule 1, add concurrency limits per job type.
• Scheduled (cron-like) jobs double-fire on overlapping schedules → take a distributed lock keyed by job name + period before running.

# References

Deeper recipes and library-specific snippets: see references/patterns.md.