# name: instrumenting-observability description: Instruments backend services with structured JSON logs, correlation IDs, RED metrics, and OpenTelemetry traces, plus alerting on symptoms and golden-signal dashboards. Use when the user asks to add or improve logging, metrics, tracing, monitoring, alerts, or dashboards for a service, propagate request IDs, or pick log levels. Do not use for structuring error-handling code itself (handling-errors) or for incident-response process and runbooks.

# Instrumenting Observability

# When to use / when NOT to use

• Use for: logging strategy, metrics, distributed tracing, alert and dashboard design for backend services.
• Do NOT use for: how code raises/maps errors (handling-errors) or writing incident runbooks/postmortems.

# Core rules

1. Logs are structured JSON, one event per line, with consistent field names across all services (ts, level, service, request_id, event, then context).
   • ✅ {"ts":"2026-08-05T14:02:11Z","level":"info","service":"orders","request_id":"r-9f2","event":"order_created","order_id":123}
   • ❌ print(f"created order {id}!!")
2. Every request gets a correlation ID — accept inbound X-Request-ID (else generate one), attach it to every log line via context, and forward it on every outbound call.
3. Log levels have contracts. ERROR = a human should act; WARN = degraded but self-healing; INFO = significant state change; DEBUG = development detail, disabled in production. If nobody would act on it, it is not ERROR.
4. Emit RED metrics per endpoint — Rate, Errors, Duration (as a histogram, not an average) — plus the handful of business metrics that matter (orders_created, payments_failed).
5. OpenTelemetry is the default for traces (and its semantic conventions for names). Auto-instrument HTTP/DB clients first; add manual spans only around meaningful units of work.
6. Never log secrets or PII. Maintain a denylist (password, token, authorization, card fields), scrub at the logger layer, and review new log statements for payload dumps.
7. Alert on symptoms, not causes — SLO burn rate, error ratio, p99 latency. CPU at 80% is not a page; users receiving 500s is.
8. One dashboard per service, golden signals first (latency, traffic, errors, saturation), business metrics second. If a panel never changed a decision, delete it.

# Workflow

1. Add the shared logging setup (JSON formatter + context injection) and the request-ID middleware.
2. Instrument RED metrics on every route and consumer; histogram buckets sized to the SLO.
3. Enable OpenTelemetry auto-instrumentation; verify trace context propagates across one full request path.
4. Define 2–4 symptom alerts tied to SLOs; wire dashboards with golden signals.
5. Validate: make one request and confirm the same request_id appears in every service's logs and on the trace; grep a log sample for denylisted keys (grep -iE "password|token|authorization" must return nothing).

# Edge cases & failure modes

• High-cardinality label explosion (user_id, URL-with-ID as metric labels) → metrics store meltdown; keep IDs in logs/traces, out of metric labels.
• Log volume spikes (tight retry loop logging per attempt) → log the first failure and the final outcome, count the rest in a metric.
• Sampling → 100% traces is fine at low traffic; above ~100 rps, head-sample (e.g., 10%) but always keep error traces.
• Clock skew across services → rely on trace spans for ordering, not log timestamps.

# References

Deeper recipes and gotchas: see references/patterns.md