---
name: instrumenting-observability
description: Instruments backend services with structured JSON logs, correlation IDs, RED metrics, and OpenTelemetry traces, plus alerting on symptoms and golden-signal dashboards. Use when the user asks to add or improve logging, metrics, tracing, monitoring, alerts, or dashboards for a service, propagate request IDs, or pick log levels. Do not use for structuring error-handling code itself (handling-errors) or for incident-response process and runbooks.
---

<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Instrumenting Observability

## When to use / when NOT to use
- **Use for:** logging strategy, metrics, distributed tracing, alert and dashboard design for backend services.
- **Do NOT use for:** how code raises/maps errors (handling-errors) or writing incident runbooks/postmortems.

## Core rules

1. **Logs are structured JSON, one event per line,** with consistent field names across all services (`ts`, `level`, `service`, `request_id`, `event`, then context).
   - ✅ `{"ts":"2026-08-05T14:02:11Z","level":"info","service":"orders","request_id":"r-9f2","event":"order_created","order_id":123}`
   - ❌ `print(f"created order {id}!!")`
2. **Every request gets a correlation ID** — accept inbound `X-Request-ID` (else generate one), attach it to every log line via context, and forward it on every outbound call.
3. **Log levels have contracts.** ERROR = a human should act; WARN = degraded but self-healing; INFO = significant state change; DEBUG = development detail, disabled in production. If nobody would act on it, it is not ERROR.
4. **Emit RED metrics per endpoint** — Rate, Errors, Duration (as a histogram, not an average) — plus the handful of business metrics that matter (orders_created, payments_failed).
5. **OpenTelemetry is the default for traces** (and its semantic conventions for names). Auto-instrument HTTP/DB clients first; add manual spans only around meaningful units of work.
6. **Never log secrets or PII.** Maintain a denylist (password, token, authorization, card fields), scrub at the logger layer, and review new log statements for payload dumps.
7. **Alert on symptoms, not causes** — SLO burn rate, error ratio, p99 latency. CPU at 80% is not a page; users receiving 500s is.
8. **One dashboard per service, golden signals first** (latency, traffic, errors, saturation), business metrics second. If a panel never changed a decision, delete it.

## Workflow

1. Add the shared logging setup (JSON formatter + context injection) and the request-ID middleware.
2. Instrument RED metrics on every route and consumer; histogram buckets sized to the SLO.
3. Enable OpenTelemetry auto-instrumentation; verify trace context propagates across one full request path.
4. Define 2–4 symptom alerts tied to SLOs; wire dashboards with golden signals.
5. Validate: make one request and confirm the same `request_id` appears in every service's logs and on the trace; grep a log sample for denylisted keys (`grep -iE "password|token|authorization"` must return nothing).

## Edge cases & failure modes
- **High-cardinality label explosion** (user_id, URL-with-ID as metric labels) → metrics store meltdown; keep IDs in logs/traces, out of metric labels.
- **Log volume spikes** (tight retry loop logging per attempt) → log the first failure and the final outcome, count the rest in a metric.
- **Sampling** → 100% traces is fine at low traffic; above ~100 rps, head-sample (e.g., 10%) but always keep error traces.
- **Clock skew across services** → rely on trace spans for ordering, not log timestamps.

## References
Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)