spb/ultra-sharp-agent-skills Public

1---2name: writing-background-jobs3description: Designs and implements background jobs and task queues that survive retries, crashes, and duplicate delivery — idempotency, backoff, dead-letter queues, timeouts, and job observability. Use when the user asks to write a background job, worker, or task queue, move work out of a request (send emails, process images, sync data), add retries to a job, or fix duplicate/stuck jobs (Celery, Sidekiq, BullMQ, RQ, or hand-rolled workers). Do not use for inter-service event/messaging architecture (handling-async-messaging) or cron-style system administration.4---56<!--7Author: Simon-Pierre Boucher8Contact: contact@spboucher.ai9-->1011# Writing Background Jobs1213## When to use / when NOT to use14- **Use for:** designing or reviewing background jobs, workers, and task queues inside one application: enqueueing, retries, failure handling, monitoring.15- **Do NOT use for:** events between services (→ handling-async-messaging), customer-facing webhook delivery (→ designing-webhooks), or OS-level cron administration.1617## Core rules18191. **Every job is idempotent.** Queues deliver at-least-once; your job WILL run twice. Guard side effects with a natural key or an idempotency record.20   - ✅ `INSERT ... ON CONFLICT (payment_id) DO NOTHING`, then act only on inserted rows21   - ❌ `charge_card(amount)` executed unconditionally at the top of the job222. **Payloads carry IDs, not objects.** Refetch current state at run time; serialized objects go stale between enqueue and execution.23   - ✅ `enqueue(send_invoice, invoice_id=42)`24   - ❌ `enqueue(send_invoice, invoice=invoice.to_json())`253. **Explicit retry policy on every job:** exponential backoff with jitter, capped attempts. Default: base 30 s, factor 2, full jitter, max 5 attempts. Distinguish retryable (network, 5xx) from permanent (validation) failures — permanent failures skip retries and go straight to the dead-letter queue.264. **Dead-letter queue with an alert.** Exhausted jobs land in a DLQ that pages someone; a silent DLQ is a data-loss buffer.275. **Timeout on every job.** No default-infinite jobs: set an explicit per-job timeout slightly above p99 runtime, and make the handler kill-safe (rule 1 covers the rerun).286. **No shared mutable state between jobs.** Workers run concurrently across processes and hosts; coordinate through the database (row locks, `SELECT ... FOR UPDATE SKIP LOCKED`) — never through process memory or files.297. **Long work = chain of short jobs.** Split anything over ~1 minute into resumable steps (paginate by cursor, one page per job). Short jobs retry cheaply; hour-long jobs lose an hour per crash.308. **Instrument jobs like endpoints:** duration, success/failure counters per job type, queue depth and oldest-message age with alerts. Queue depth growing while workers are idle means a poison job or a crashed consumer.3132## Workflow33341. Define the job: trigger, payload (IDs only), side effects, and the idempotency key for each side effect.352. Classify failures as retryable vs permanent; set backoff (30 s base, ×2, jitter, 5 attempts) and timeout (≈ p99 runtime + margin).363. Implement with the project's existing queue library — do not introduce a new one if any queue is already present.374. Wire the DLQ and its alert; add duration/failure metrics.385. **Validate:** run the job twice with the same payload and confirm exactly one side effect; kill the worker mid-job and confirm the retry completes cleanly.3940## Edge cases & failure modes41- **Job depends on uncommitted data** → enqueue after commit (transactional enqueue or on-commit hook), or the worker races the transaction and sees nothing.42- **Poison message** (crashes the worker every time) → attempts cap sends it to the DLQ; never retry forever.43- **Queue backlog after outage** → workers must tolerate a thundering herd: keep rule 1, add concurrency limits per job type.44- **Scheduled (cron-like) jobs double-fire** on overlapping schedules → take a distributed lock keyed by job name + period before running.4546## References47Deeper recipes and library-specific snippets: see [references/patterns.md](references/patterns.md).48