spb/ultra-sharp-agent-skills Public

1---2name: instrumenting-observability3description: Instruments backend services with structured JSON logs, correlation IDs, RED metrics, and OpenTelemetry traces, plus alerting on symptoms and golden-signal dashboards. Use when the user asks to add or improve logging, metrics, tracing, monitoring, alerts, or dashboards for a service, propagate request IDs, or pick log levels. Do not use for structuring error-handling code itself (handling-errors) or for incident-response process and runbooks.4---56<!--7Author: Simon-Pierre Boucher8Contact: contact@spboucher.ai9-->1011# Instrumenting Observability1213## When to use / when NOT to use14- **Use for:** logging strategy, metrics, distributed tracing, alert and dashboard design for backend services.15- **Do NOT use for:** how code raises/maps errors (handling-errors) or writing incident runbooks/postmortems.1617## Core rules18191. **Logs are structured JSON, one event per line,** with consistent field names across all services (`ts`, `level`, `service`, `request_id`, `event`, then context).20   - ✅ `{"ts":"2026-08-05T14:02:11Z","level":"info","service":"orders","request_id":"r-9f2","event":"order_created","order_id":123}`21   - ❌ `print(f"created order {id}!!")`222. **Every request gets a correlation ID** — accept inbound `X-Request-ID` (else generate one), attach it to every log line via context, and forward it on every outbound call.233. **Log levels have contracts.** ERROR = a human should act; WARN = degraded but self-healing; INFO = significant state change; DEBUG = development detail, disabled in production. If nobody would act on it, it is not ERROR.244. **Emit RED metrics per endpoint** — Rate, Errors, Duration (as a histogram, not an average) — plus the handful of business metrics that matter (orders_created, payments_failed).255. **OpenTelemetry is the default for traces** (and its semantic conventions for names). Auto-instrument HTTP/DB clients first; add manual spans only around meaningful units of work.266. **Never log secrets or PII.** Maintain a denylist (password, token, authorization, card fields), scrub at the logger layer, and review new log statements for payload dumps.277. **Alert on symptoms, not causes** — SLO burn rate, error ratio, p99 latency. CPU at 80% is not a page; users receiving 500s is.288. **One dashboard per service, golden signals first** (latency, traffic, errors, saturation), business metrics second. If a panel never changed a decision, delete it.2930## Workflow31321. Add the shared logging setup (JSON formatter + context injection) and the request-ID middleware.332. Instrument RED metrics on every route and consumer; histogram buckets sized to the SLO.343. Enable OpenTelemetry auto-instrumentation; verify trace context propagates across one full request path.354. Define 2–4 symptom alerts tied to SLOs; wire dashboards with golden signals.365. Validate: make one request and confirm the same `request_id` appears in every service's logs and on the trace; grep a log sample for denylisted keys (`grep -iE "password|token|authorization"` must return nothing).3738## Edge cases & failure modes39- **High-cardinality label explosion** (user_id, URL-with-ID as metric labels) → metrics store meltdown; keep IDs in logs/traces, out of metric labels.40- **Log volume spikes** (tight retry loop logging per attempt) → log the first failure and the final outcome, count the rest in a metric.41- **Sampling** → 100% traces is fine at low traffic; above ~100 rps, head-sample (e.g., 10%) but always keep error traces.42- **Clock skew across services** → rely on trace spans for ordering, not log timestamps.4344## References45Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)46