spb/internetpressure

1# Architecture23```4 8 probes (Go, launchd/systemd)            RIPE RIS Live (wss)            public status pages (optional)5 Québec · Montréal · Atlanta · Gravelines   23 collectors, ~10k prefixes/s   Statuspage / GCP / AWS RSS / status.io6 Dublin · Istanbul · Uşak · Ayia Napa              │                                  │7        │ signed gzip batches (HTTPS)               │                                  │8        ▼                                           ▼                                  ▼9 ┌──────────────── BHS64b (OVH Beauharnois, Docker Compose) ──────────────────────────────────────┐10 │ edge Caddy :8350 (bound to wg1 10.67.0.61) ── /ingest,/api → api :8352 ── / → web :8351         │11 │ api (FastAPI)  ── ingest → ClickHouse writer + Redis live counters                               │12 │ bgp (RIS Live) ── 10 s per-collector aggregates, per-origin minutes, sampled raw                 │13 │ corroboration  ── vendor_status (Postgres)                                                       │14 │ engine (10 s)  ── baselines (5 min) → robust z → components → scopes → global → events → fronts  │15 │                   → Redis live state + pub/sub → SSE ;  ClickHouse pressure_history/provenance   │16 │ Postgres 17 (registry, config, events)  ClickHouse 25.8 (telemetry, history)  Redis 7 (live)     │17 └──────────────────────────────────────────────────────────────────────────────────────────────────┘18        ▲ WireGuard 10.67.0.0/2419 BHS64 gateway: Caddy TLS  https://www.internetpressure.io → 10.67.0.61:8350   (DNS A → 51.161.112.61)20```2122## Decisions2324- **Python (FastAPI) backend, Go probe.** The spec mandates Go for the agent; the backend language is free. Python matches the25  other MacLustr platforms (AI Atlas, SatelliteIndex, CountryAtlas) and keeps the scoring code readable/testable. All stores are26  spoken to over their wire protocols (asyncpg, ClickHouse HTTP, redis) — no ORM.27- **One backend image, four processes** (`ip api`, `ip engine`, `ip bgp`, `ip corroboration`). They share nothing but the stores.28- **ClickHouse HTTP interface** via httpx (JSONEachRow) instead of a driver: async, dependency-free, trivially inspectable.29- **Baselines from raw measurements** (180 d TTL) recomputed every 5 minutes; MAD estimated from the IQR (0.7413 × IQR) in a30  single pass; hour-of-day seasonality (±1 h) once ≥ 3 days of history exist. `measurements_1m` (AggregatingMergeTree MV, 1 y)31  serves long-range charts.32- **Self-exclusion** (`engine/health.py`): < 2 fresh usable probes, every probe failing at once, or a store down ⇒33  `internal_status = degraded`, score frozen (`stale: true`), no history rows, no incidents. A probe failing ≥ 80 % of its HTTP34  targets while the median probe is fine is excluded (its uplink, not the Internet).35- **Calibration**: until at least one signal has ≥ 20 % of `baseline_min_samples` (12 five-minute buckets), the instrument36  publishes `pressure: null`, `level: calibrating`. Nothing is invented.37- **Config-driven scoring**: `packages/config/pressure.yaml`, optionally overridden by the Postgres `config` row (admin UI); the38  engine re-reads it every cycle. Weights must sum to 1 (validated).39- **Explainability**: every cycle stores `signal_features` (component, signal, scope, current, baseline, MAD, z, samples, stress,40  contribution) and the live payload carries `explain[]` (attribution = component contribution × share of stress mass by region /41  signal / service) and `components[].drivers[]`.4243## Scoring pipeline (engine/compute.py, engine/scoring.py)44451. **Pair signals**: per (probe, target, kind, resolver) in the last 120 s: latency z (ttfb, tcp, rtt), loss excess, failure-rate46   excess, 5xx/TLS/reset rates, DNS failure & latency; per target: corroborated unavailability (≥ 2 probes in ≥ 2 regions) and47   resolver disagreement (one resolver answers, another SERVFAILs); per (probe, target) traceroute: route fingerprint changed vs48   the 7-day dominant route, hop-count z, latency shift on changed paths.492. **Pair stress** 0..1: `z_to_stress` ramps from z = 1 to z = 6; rate signals use `rate_stress` (excess / max(scale, 3 × baseline));50   route churn uses `ratio_stress`.513. **Signal stress** = importance-weighted mean of pair stresses; the weight is damped by baseline coverage52   (`coverage_factor`), so thin baselines cannot move the index.534. **Component score** = `saturate(Σ signal_weight × signal_stress)`, `saturate(x) = 100·(1−e^{−kx})/(1−e^{−k})`, k = 1.2.545. **Routing** (global only, ASN level via `bgp_origin_1m`): weighted-median robust z across collectors of per-minute55   withdrawals / announcements / origin changes vs their 7-day (seasonal) baseline; collector disagreement when a minority of56   collectors spike.576. **Corroboration**: Σ impact(indicator) × service importance / 4, capped at 1 — weight 0.05, optional by design.587. **Global** = Σ weight × component over available components (weights renormalised). Regions/countries/services/ASNs reuse59   the same machinery over their signal subsets (routing null where not attributable).608. **Velocity / acceleration / volatility** from the last hour of `pressure_history`.619. **Events** (`engine/events.py`): scope score ≥ 45 with confidence ≥ 0.45 for 2 cycles ⇒ `detected`; → `developing` → `active`62   (6 cycles); < 30 ⇒ `recovering`; 10 min under 30 ⇒ `resolved`. Rule-based hypotheses with evidence.6310. **Pressure Fronts** (`engine/fronts.py`): ≥ 3 stressed (probe, target) pairs on a source-region → destination-region corridor64    with saturated corridor stress ≥ 35.6566## Ethics & load6768Per probe with ~27 700 targets (tiers 20 s / 45 s / 300 s / 900 s; the long tail is HTTP-only): ~60 HTTP/s, ~26 DNS/s (4 resolvers / 600 s, infrastructure only), ~20 ICMP runs/s (every 120 s, infrastructure only), ≤ 1 traceroute in flight (every 15 min69for the 30 `tr: true` targets). Keep-alive disabled (to measure TCP/TLS), body read capped at 16 KiB, no retries, no parallel70checks to one host, intervals never below 10 s. Total ≈ 105 measurements/s/probe → ~70 M rows/day for 8 probes (~3–4 GB/day71compressed in ClickHouse).7273## Replay7475`ip replay <from> <to> --weights '{…}'` (and `POST /api/admin/replay`) recomputes the global index from stored component76history with alternative weights. Re-deriving components from raw measurements with alternative engine parameters is an77offline job to add when enough history exists (raw is kept 180 days for exactly that purpose).7879## Retention — we never erase (founder decision, 2026-09-12, migration `0002_keep_forever.sql`)8081Every table is kept indefinitely: measurements, measurements_1m, traceroutes, probe_health, bgp_stats_10s, bgp_origin_1m,82pressure_history, signal_features, engine_runs. The single exception is `bgp_events`, the *raw* RIS Live stream (≈ 10 k83prefixes/s at the source, already sampled 1/25 for announcements): 90 days. Nightly backups (`deploy/bin/backup.sh`) copy84Postgres and the irreplaceable ClickHouse tables off-node to BHS128. Disk growth ≈ 3–4 GB/day on BHS64b's 878 GB RAID-1;85plan a second ClickHouse volume (or a cold tier on BHS128) within a year.8687## Registry at scale (2026-09-13)8889`data/targets/targets.yaml` (2 128 hand-curated) + `data/targets/fragments/{government,finance,infrastructure,developer,media,commerce}.yaml`90(26 700 more, every host verified by DNS + HTTP with the probe UA at authoring time). `ip seed` loads the main file then the91fragments, skipping duplicate ids or (host, url, ip) — first occurrence wins. Tier 4 (900 s) carries the long tail; the probes92receive the whole list (≈ 8 MB JSON, cached per config version) and schedule ≈ 90 000 jobs each with 48-way concurrency.93Engine: ≈ 35 000 (probe, target) pairs in each 120 s window, ≈ 100 000 signals, ~2 s per 15 s cycle. The sampling boost is94delivered inside batch responses and never bumps the config version (a bump makes all probes re-download the registry).95