spb/ultra-sharp-agent-skills Public

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# Patterns — Instrumenting Observability78## Contents9- Structured logging setup (Python)10- Request-ID middleware and propagation11- RED metrics (Prometheus)12- OpenTelemetry tracing setup13- Symptom alerts (SLO burn)14- Gotchas1516## Structured logging setup (Python)1718```python19import structlog2021structlog.configure(22    processors=[23        structlog.contextvars.merge_contextvars,   # injects request_id24        structlog.processors.add_log_level,25        structlog.processors.TimeStamper(fmt="iso", key="ts"),26        structlog.processors.JSONRenderer(),27    ],28)29logger = structlog.get_logger(service="orders")30logger.info("order_created", order_id=123, total_cents=4999)31```3233Scrub secrets before rendering:3435```python36DENYLIST = {"password", "token", "authorization", "card_number", "cvv"}37def scrub(logger, method, event_dict):38    for k in list(event_dict):39        if k.lower() in DENYLIST:40            event_dict[k] = "[redacted]"41    return event_dict42# add `scrub` before JSONRenderer in processors43```4445## Request-ID middleware and propagation4647```python48import uuid, structlog4950@app.middleware("http")51async def request_id(request, call_next):52    rid = request.headers.get("x-request-id") or f"r-{uuid.uuid4().hex[:12]}"53    structlog.contextvars.bind_contextvars(request_id=rid)54    response = await call_next(request)55    response.headers["x-request-id"] = rid      # echo for the caller56    return response5758# outbound: always forward59httpx.get(url, headers={"x-request-id": rid})60```6162## RED metrics (Prometheus)6364```python65from prometheus_client import Counter, Histogram6667REQS = Counter("http_requests_total", "requests", ["route", "method", "status"])68# Buckets bracket the SLO (e.g. 300ms target): resolution where it matters.69LAT = Histogram("http_request_seconds", "latency", ["route"],70                buckets=[.025, .05, .1, .2, .3, .5, 1, 2, 5])7172@app.middleware("http")73async def metrics(request, call_next):74    with LAT.labels(request.url.path).time():75        resp = await call_next(request)76    REQS.labels(request.url.path, request.method, resp.status_code).inc()77    return resp78```7980Label values must be low-cardinality: route *templates* (`/orders/{id}`), never raw paths.8182## OpenTelemetry tracing setup8384```python85from opentelemetry import trace86from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor87from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor8889FastAPIInstrumentor.instrument_app(app)     # inbound spans + context extraction90HTTPXClientInstrumentor().instrument()      # outbound propagation9192tracer = trace.get_tracer("orders")93with tracer.start_as_current_span("price_cart") as span:   # manual span: meaningful unit only94    span.set_attribute("cart.items", len(items))95    total = price(items)96```9798Export via OTLP to the collector; configure endpoint with `OTEL_EXPORTER_OTLP_ENDPOINT`.99100## Symptom alerts (SLO burn)101102```yaml103# Page when error budget burns 14.4x too fast over 1h AND 5m (multiwindow).104- alert: HighErrorBurn105  expr: >106    (sum(rate(http_requests_total{status=~"5.."}[5m]))107     / sum(rate(http_requests_total[5m]))) > 14.4 * 0.001108    and109    (sum(rate(http_requests_total{status=~"5.."}[1h]))110     / sum(rate(http_requests_total[1h]))) > 14.4 * 0.001111  labels: {severity: page}112```113114p99 latency against SLO: `histogram_quantile(0.99, sum(rate(http_request_seconds_bucket[5m])) by (le))`.115116## Gotchas117- Averages hide pain: a 50 ms mean can coexist with a 5 s p99 — always use histograms/percentiles.118- `user_id` as a metric label = cardinality bomb; it belongs in log fields and span attributes.119- Logging inside a hot retry loop can 100x volume — first failure + final outcome, counter for the middle.120- Forgetting to *echo* X-Request-ID back means clients can't report the ID for support.121- OTel context does not cross thread/process pools automatically — use the context-propagation helpers.122- DEBUG left on in prod both leaks payloads and doubles log cost; enforce level via env config.123