---
name: scaling-backend-services
description: Scales backend services to handle load — bottleneck measurement first, stateless app tiers, connection pooling, load balancing, autoscaling on the saturating metric, queue-based load leveling, and read replicas. Use when the user asks how to scale a service or API, handle more traffic or concurrent users, fix connection exhaustion, size a connection pool, add a load balancer or autoscaling, or prepare for a traffic spike. Do not use for SQL query tuning (db-skills/optimizing-sql-performance), cache design (caching-strategies), or rate limiting (limiting-request-rates).
---

<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Scaling Backend Services

## When to use / when NOT to use
- **Use for:** capacity planning, horizontal/vertical scaling decisions, pool sizing, load balancing, autoscaling, spike preparation.
- **Do NOT use for:** query-level tuning (db-skills/optimizing-sql-performance), caching design (caching-strategies), rate limiting (limiting-request-rates), or frontend performance.

## Core rules

1. **Measure the bottleneck before scaling anything.** Load-test with realistic traffic shape and identify the saturating resource (CPU, memory, DB connections, IO, downstream API). Scaling the wrong tier spends money to move the queue.
   - ✅ "p99 collapses at 800 RPS; DB connections pinned at max" → fix pooling.
   - ❌ "It's slow, add more pods."
2. **Vertical first when it's cheaper than complexity.** Doubling instance size is a config change; sharding is a project. Buy headroom while you build the durable fix.
3. **Stateless app tier is the precondition for horizontal scaling.** Sessions, uploads, and locks live in external stores (Redis/object storage/DB) so any instance can serve any request and instances can die freely.
4. **Pool every connection.** App→DB and app→downstream. DB pool starting point: `cores × 2` for CPU-bound work; more only for IO-wait-heavy loads. Many app replicas × pool size must stay under the DB's max — put a server-side pooler (pgbouncer) in front of Postgres once replicas multiply.
5. **Load balance with health checks and connection draining.** Instances that fail readiness stop receiving traffic; deploys drain in-flight requests before termination — zero-downtime is a MUST-pass test, not a hope.
6. **Autoscale on the saturating metric, not CPU by reflex.** Queue depth, p95 latency, or connections — whatever rule 1 found. Scale up fast, down slowly (thrash guard), and cap max instances below what the database can survive (rule 4 arithmetic).
7. **Level spiky writes through a queue.** Accept fast (202 + job id), process at a sustainable rate, make consumers idempotent. The queue absorbs the spike; the worker pool sets the drain rate.
8. **Read replicas for read-heavy loads — with lag eyes open.** Route reads that tolerate staleness to replicas; read-your-own-writes flows stay on the primary.

## Workflow

1. Load-test to current breaking point; record the saturating metric and p95/p99 at each load step.
2. Externalize any instance state found (rule 3).
3. Size pools by rule 4 arithmetic across ALL replicas; add pgbouncer if the sum approaches DB max_connections.
4. Configure LB health checks + draining; set autoscaling on the measured metric with up-fast/down-slow policies and a hard max.
5. Queue spiky write paths (rule 7).
6. Validate: re-run the load test — the previous breaking point passes; kill one instance at full load and confirm zero failed requests; verify autoscaler adds and (slowly) removes instances.

## Edge cases & failure modes
- **Thundering herd after downtime** → LB slow-start / gradual traffic ramp for recovering instances; jittered client retries.
- **Autoscaler scales app until the DB dies** → rule 6 cap; the database's ceiling is the fleet's ceiling.
- **One hot key/tenant saturates a single shard/instance** → consistent hashing with hot-key detection; isolate the noisy tenant.
- **Long-lived connections (websockets) defeat draining** → set max connection age; force periodic reconnect so deploys can complete.

## References
Pool math, autoscaling configs, load-test sketches: see [references/patterns.md](references/patterns.md).