# name: scaling-backend-services description: Scales backend services to handle load — bottleneck measurement first, stateless app tiers, connection pooling, load balancing, autoscaling on the saturating metric, queue-based load leveling, and read replicas. Use when the user asks how to scale a service or API, handle more traffic or concurrent users, fix connection exhaustion, size a connection pool, add a load balancer or autoscaling, or prepare for a traffic spike. Do not use for SQL query tuning (db-skills/optimizing-sql-performance), cache design (caching-strategies), or rate limiting (limiting-request-rates).

# Scaling Backend Services

# When to use / when NOT to use

• Use for: capacity planning, horizontal/vertical scaling decisions, pool sizing, load balancing, autoscaling, spike preparation.
• Do NOT use for: query-level tuning (db-skills/optimizing-sql-performance), caching design (caching-strategies), rate limiting (limiting-request-rates), or frontend performance.

# Core rules

1. Measure the bottleneck before scaling anything. Load-test with realistic traffic shape and identify the saturating resource (CPU, memory, DB connections, IO, downstream API). Scaling the wrong tier spends money to move the queue.
   • ✅ "p99 collapses at 800 RPS; DB connections pinned at max" → fix pooling.
   • ❌ "It's slow, add more pods."
2. Vertical first when it's cheaper than complexity. Doubling instance size is a config change; sharding is a project. Buy headroom while you build the durable fix.
3. Stateless app tier is the precondition for horizontal scaling. Sessions, uploads, and locks live in external stores (Redis/object storage/DB) so any instance can serve any request and instances can die freely.
4. Pool every connection. App→DB and app→downstream. DB pool starting point: cores × 2 for CPU-bound work; more only for IO-wait-heavy loads. Many app replicas × pool size must stay under the DB's max — put a server-side pooler (pgbouncer) in front of Postgres once replicas multiply.
5. Load balance with health checks and connection draining. Instances that fail readiness stop receiving traffic; deploys drain in-flight requests before termination — zero-downtime is a MUST-pass test, not a hope.
6. Autoscale on the saturating metric, not CPU by reflex. Queue depth, p95 latency, or connections — whatever rule 1 found. Scale up fast, down slowly (thrash guard), and cap max instances below what the database can survive (rule 4 arithmetic).
7. Level spiky writes through a queue. Accept fast (202 + job id), process at a sustainable rate, make consumers idempotent. The queue absorbs the spike; the worker pool sets the drain rate.
8. Read replicas for read-heavy loads — with lag eyes open. Route reads that tolerate staleness to replicas; read-your-own-writes flows stay on the primary.

# Workflow

1. Load-test to current breaking point; record the saturating metric and p95/p99 at each load step.
2. Externalize any instance state found (rule 3).
3. Size pools by rule 4 arithmetic across ALL replicas; add pgbouncer if the sum approaches DB max_connections.
4. Configure LB health checks + draining; set autoscaling on the measured metric with up-fast/down-slow policies and a hard max.
5. Queue spiky write paths (rule 7).
6. Validate: re-run the load test — the previous breaking point passes; kill one instance at full load and confirm zero failed requests; verify autoscaler adds and (slowly) removes instances.

# Edge cases & failure modes

• Thundering herd after downtime → LB slow-start / gradual traffic ramp for recovering instances; jittered client retries.
• Autoscaler scales app until the DB dies → rule 6 cap; the database's ceiling is the fleet's ceiling.
• One hot key/tenant saturates a single shard/instance → consistent hashing with hot-key detection; isolate the noisy tenant.
• Long-lived connections (websockets) defeat draining → set max connection age; force periodic reconnect so deploys can complete.

# References

Pool math, autoscaling configs, load-test sketches: see references/patterns.md.