# Patterns — Scaling Backend Services

# Contents

Connection-pool arithmetic
pgbouncer minimal config
Kubernetes HPA on a custom metric
Load balancer health + draining (nginx)
Queue-based load leveling sketch
Load-test skeleton (k6)
Gotchas

# Connection-pool arithmetic

text

per-instance pool  = cores × 2                  # CPU-bound starting point
fleet demand       = replicas × per-instance pool
must satisfy       : fleet demand < db max_connections − superuser_reserved

Example: 12 replicas × 10 pool = 120 > Postgres default 100 → pgbouncer required.

IO-wait-heavy workloads (slow downstreams inside transactions) justify larger pools — but first shorten the transaction, don't widen the pool.

# pgbouncer minimal config

ini

[databases]
app = host=10.0.0.5 port=5432 dbname=app

[pgbouncer]
listen_port = 6432
auth_type = scram-sha-256
pool_mode = transaction        ; multiplexes: many clients, few server conns
default_pool_size = 20         ; server-side connections per db/user pair
max_client_conn = 2000
server_idle_timeout = 60

pool_mode = transaction breaks session state (prepared statements, advisory locks, SET) — verify the driver's compatibility mode before enabling.

# Kubernetes HPA on a custom metric

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3
  maxReplicas: 24            # capped by DB pool arithmetic, not by budget alone
  metrics:
    - type: Pods
      pods:
        metric: { name: http_p95_latency_ms }
        target: { type: AverageValue, averageValue: "250" }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # react to spikes immediately
      policies: [{ type: Percent, value: 100, periodSeconds: 60 }]
    scaleDown:
      stabilizationWindowSeconds: 300    # shrink slowly — thrash guard
      policies: [{ type: Pods, value: 1, periodSeconds: 120 }]

# Load balancer health + draining (nginx)

nginx

upstream api {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8000 slow_start=30s;   # ramp recovered instances gradually
}

App side (draining): on SIGTERM, fail the readiness endpoint, keep serving in-flight requests, exit after they finish or a 30 s deadline:

python

def handle_sigterm(*_):
    ready.set_unhealthy()        # LB stops sending new traffic
    server.shutdown(grace=30)    # finish in-flight, then exit

# Queue-based load leveling sketch

python

# ingest: accept fast, defer work
@app.post("/imports")
def create_import(req):
    job_id = queue.enqueue("imports", req.body, idempotency_key=req.headers["Idempotency-Key"])
    return 202, {"job_id": job_id, "status_url": f"/imports/{job_id}"}

# worker pool drains at a sustainable rate; consumers are idempotent
def worker():
    for job in queue.consume("imports", prefetch=1):
        if already_processed(job.idempotency_key):   # replay-safe
            job.ack(); continue
        process(job)
        mark_processed(job.idempotency_key)
        job.ack()

Worker count — not producer rate — sets DB write pressure; scale workers only while the DB stays under its ceiling.

# Load-test skeleton (k6)

javascript

import http from 'k6/http';
export const options = {
  stages: [
    { duration: '2m', target: 200 },   // ramp
    { duration: '5m', target: 200 },   // steady — realistic think time below
    { duration: '2m', target: 800 },   // find the knee
  ],
  thresholds: { http_req_duration: ['p(95)<300'] },
};
export default function () {
  http.get('https://staging.example.com/api/orders');
}

Test against staging with production-shaped data; an empty database lies.

# Gotchas

Sticky sessions are hidden state — they break draining and uneven-load the fleet; externalize the session instead.
Autoscaling on CPU while blocked on IO does nothing: instances idle at 20% CPU while every request waits on the pool. Scale on the metric that saturates.
least_conn beats round-robin once request costs vary, but health checks matter more than the algorithm.
Replica lag is not constant — it spikes exactly when you're overloaded, i.e., when you rerouted reads there. Monitor lag and fail reads back gracefully.
Local caches per instance multiply cold starts during scale-up; a scale-out event can stampede the DB (see caching-strategies for stampede locks).
Max-instance caps get deleted in incidents — document WHY the cap exists (DB arithmetic) next to the setting.