<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Patterns — Scaling Backend Services

## Contents
- Connection-pool arithmetic
- pgbouncer minimal config
- Kubernetes HPA on a custom metric
- Load balancer health + draining (nginx)
- Queue-based load leveling sketch
- Load-test skeleton (k6)
- Gotchas

## Connection-pool arithmetic

```
per-instance pool  = cores × 2                  # CPU-bound starting point
fleet demand       = replicas × per-instance pool
must satisfy       : fleet demand < db max_connections − superuser_reserved

Example: 12 replicas × 10 pool = 120 > Postgres default 100 → pgbouncer required.
```

IO-wait-heavy workloads (slow downstreams inside transactions) justify larger
pools — but first shorten the transaction, don't widen the pool.

## pgbouncer minimal config

```ini
[databases]
app = host=10.0.0.5 port=5432 dbname=app

[pgbouncer]
listen_port = 6432
auth_type = scram-sha-256
pool_mode = transaction        ; multiplexes: many clients, few server conns
default_pool_size = 20         ; server-side connections per db/user pair
max_client_conn = 2000
server_idle_timeout = 60
```

`pool_mode = transaction` breaks session state (prepared statements, advisory
locks, `SET`) — verify the driver's compatibility mode before enabling.

## Kubernetes HPA on a custom metric

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3
  maxReplicas: 24            # capped by DB pool arithmetic, not by budget alone
  metrics:
    - type: Pods
      pods:
        metric: { name: http_p95_latency_ms }
        target: { type: AverageValue, averageValue: "250" }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # react to spikes immediately
      policies: [{ type: Percent, value: 100, periodSeconds: 60 }]
    scaleDown:
      stabilizationWindowSeconds: 300    # shrink slowly — thrash guard
      policies: [{ type: Pods, value: 1, periodSeconds: 120 }]
```

## Load balancer health + draining (nginx)

```nginx
upstream api {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8000 slow_start=30s;   # ramp recovered instances gradually
}
```

App side (draining): on SIGTERM, fail the readiness endpoint, keep serving
in-flight requests, exit after they finish or a 30 s deadline:

```python
def handle_sigterm(*_):
    ready.set_unhealthy()        # LB stops sending new traffic
    server.shutdown(grace=30)    # finish in-flight, then exit
```

## Queue-based load leveling sketch

```python
# ingest: accept fast, defer work
@app.post("/imports")
def create_import(req):
    job_id = queue.enqueue("imports", req.body, idempotency_key=req.headers["Idempotency-Key"])
    return 202, {"job_id": job_id, "status_url": f"/imports/{job_id}"}

# worker pool drains at a sustainable rate; consumers are idempotent
def worker():
    for job in queue.consume("imports", prefetch=1):
        if already_processed(job.idempotency_key):   # replay-safe
            job.ack(); continue
        process(job)
        mark_processed(job.idempotency_key)
        job.ack()
```

Worker count — not producer rate — sets DB write pressure; scale workers only
while the DB stays under its ceiling.

## Load-test skeleton (k6)

```javascript
import http from 'k6/http';
export const options = {
  stages: [
    { duration: '2m', target: 200 },   // ramp
    { duration: '5m', target: 200 },   // steady — realistic think time below
    { duration: '2m', target: 800 },   // find the knee
  ],
  thresholds: { http_req_duration: ['p(95)<300'] },
};
export default function () {
  http.get('https://staging.example.com/api/orders');
}
```

Test against staging with production-shaped data; an empty database lies.

## Gotchas

- **Sticky sessions are hidden state** — they break draining and uneven-load the fleet; externalize the session instead.
- **Autoscaling on CPU while blocked on IO** does nothing: instances idle at 20% CPU while every request waits on the pool. Scale on the metric that saturates.
- **`least_conn` beats round-robin** once request costs vary, but health checks matter more than the algorithm.
- **Replica lag is not constant** — it spikes exactly when you're overloaded, i.e., when you rerouted reads there. Monitor lag and fail reads back gracefully.
- **Local caches per instance multiply cold starts** during scale-up; a scale-out event can stampede the DB (see caching-strategies for stampede locks).
- **Max-instance caps get deleted in incidents** — document WHY the cap exists (DB arithmetic) next to the setting.