spb/internetpressure

1# InternetPressure.io — product specification (authoritative, from the founder, 2026-09-12)23Tagline: **The real-time pressure gauge for the Internet.**45InternetPressure.io is a global, real-time Internet observability platform that measures the current "pressure" of the6public Internet: a continuously changing numerical representation of how stressed, unstable, congested, degraded,7fragmented or abnormal the global Internet currently is. Not another uptime monitor, status aggregator or threat8dashboard. The system observes the Internet independently with our own distributed measurement infrastructure and9open public telemetry; commercial APIs must never be a critical dependency. The long-term asset is the proprietary10historical dataset generated by the InternetPressure Observability Network.1112## 1. Vision13A weather service for the Internet: pressure, baselines, storm systems (Pressure Fronts), regional maps, forecasts.14Users must immediately understand whether the Internet is behaving normally, where stress is increasing, which15regions/providers are affected, whether routing, latency, DNS, availability or paths are abnormal, and whether an16anomaly is local, regional, provider-specific or global. The interface feels alive (Bloomberg terminal / flight radar17/ weather radar / NOC), not like a monitoring SaaS.1819## 2. Core principle — data hierarchy201. our own direct measurements · 2. open public Internet telemetry · 3. public raw feeds · 4. polling public21infrastructure information · 5. external APIs only as optional corroboration. Target: 70–80 % proprietary, 15–25 %22open feeds, < 10 % external enrichment. Must remain useful if every commercial API disappears.2324## 3. Primary metric — Global Internet Pressure Index (0–100)250–10 exceptionally calm · 10–25 normal · 25–40 elevated · 40–55 stressed · 55–70 highly stressed · 70–85 severe26disruption · 85–100 extreme Internet event. A composite observability index, never "scientific truth"; every score27explainable (click → causes).2829## 4. Components30- **Routing**: BGP announcements/withdrawals per second, churn, origin changes, visibility loss, path instability,31  leaks/hijack suspicion. Sources: RIPE RIS Live, RouteViews, CAIDA BGPStream, MRT dumps. Store raw where practical.32- **Latency** (our probes): ICMP RTT, TCP connect, TLS handshake, HTTP TTFB, inter-region latency, loss, jitter —33  always relative to baseline (20 → 100 ms matters; 100 ms alone does not).34- **DNS**: lookup latency, SERVFAIL/timeouts, NXDOMAIN anomalies, DNSSEC, resolver disagreement, authoritative35  failures, root/TLD anomalies. Probe local, Google, Cloudflare, Quad9, ISP, direct authoritative. No abusive volume.36- **Availability**: representative endpoints across cloud, major services, CDNs, DNS, developer platforms, social,37  search, finance, government, communication, streaming, commerce. Multi-probe corroboration required.38- **HTTP/TLS**: failures, 5xx, TLS/cert failures, timeouts, resets, protocol downgrade, HTTP/2–3 availability.39- **Path**: sampled traceroutes → route hashes, ASN path, hop count, transit changes, rerouting, latency shifts.40- **Infrastructure** (later): cloud region incidents, transit issues, submarine cables, IX congestion, CDN disruptions.4142## 5–6. Score design & normalisation43Provisional weights routing 0.25 · latency 0.20 · dns 0.15 · availability 0.15 · http_tls 0.10 · path 0.10 ·44corroboration 0.05 — **configuration-driven, never hard-coded**. Per signal keep rolling median/mean/std/MAD,45percentiles, hourly & weekday seasonality, regional baseline. Preferred anomaly: `robust_z = (x − median) / MAD`,46winsorised/clipped. Normal daily patterns are not pressure.4748## 7–8. Dynamic weighting & confidence49Network Importance Score (ASN centrality, prefixes, downstream dependency, known services, coverage, transit/cloud/CDN50relevance): `event_pressure = anomaly_strength × affected_scope × network_importance × confidence`. Every incident has a51confidence (probes, geo diversity, signal agreement, BGP corroboration, magnitude, external corroboration, duration).52Language: "Potential degradation", "Probable regional routing issue", "High-confidence DNS disruption".5354## 9–14. Observability network, probe agent, measurements, targets, scheduler, ethics55Own probe network (MacLustr Québec, OVH Québec/Canada/France, cheap VPS; later 18+ world regions). Agent in **Go**56(Rust alternative; not Python): lightweight, low RAM, safe restart/reconnect, self-update, buffering, health metrics,57region/ISP/ASN identity, clock sync, signed telemetry; `probe_id, region, country, provider, ASN, lat/lon approx,58version, capabilities`; never precise private addresses. Batched, compressed payloads. Central target registry in59DB/config (categories DNS, CDN, Cloud, Search, Messaging, Social, Finance, Government, News, Developer, AI, Streaming,60Commerce, Infrastructure) with importance, frequency, protocols. Adaptive scheduler: tier 1 15–30 s, tier 2 30–60 s,61tier 3 2–5 min, traceroute 5–30 min, deep checks hourly; boost sampling during anomalies. Ethics: no scanning, no auth62bypass, no private infra, no exploitation, no excessive traffic, no rate-limit evasion, no personal data — traffic63resembles a normal lightweight client.6465## 15–16. BGP ingestion & features66`bgp-ingestor`: RIS Live (+ RouteViews/BGPStream), normalise, dedupe, prefix→ASN, aggregates, anomaly features. Event67schema: timestamp, collector, peer_asn, prefix, origin_asn, event_type, as_path, community, next_hop, source. Raw kept68temporarily, aggregates indefinitely. Features: announcements/s, withdrawals/s, unique prefixes/ASNs changed, origin69changes, churn, path-length change, path entropy, visibility, collector disagreement; baselines by minute/hour/day/70weekday/region/ASN/prefix.7172## 17–20. Detection engine, events, hypotheses, Pressure Fronts73`pressure-engine` every 5–15 s: baselines, anomaly strength, component/regional/ASN/service/global scores, confidence,74incident candidates. Movement only from measurements — never randomised. Events with states detected → developing →75active → recovering → resolved, evolving over time. Causal hypotheses with evidence, never overstated. **Pressure76Fronts**: connected geographic/network regions simultaneously rising (e.g. "North Atlantic Pressure Front, intensity7774, direction East, North America → Western Europe"), drawn on the map — the signature visual feature.7879## 21–22. Storage & retention80PostgreSQL (registry, config, incidents, ASN/region metadata) · ClickHouse (measurements, telemetry, BGP aggregates,81pressure history — primary analytical store) · Redis (latest values, live state, cache, pub/sub) · object storage82(MinIO/S3) for raw archives/exports/backups. Retention: raw probes 90–180 d; 1-min 1 y; 5-min 3 y; hourly forever;83BGP raw selective; aggregates and pressure history indefinite.8485## 23–25. Architecture & stacks86Probes → Ingestion API → ClickHouse/Redis/Postgres → Pressure Engine → Event Engine + Live State → Public API →87Frontend. BGP: RIS/RouteViews → ingestor → normaliser → feature engine → pressure engine. Monorepo apps/ services/88packages/ infra/ data/ docs/. Frontend: Next.js, TypeScript, React, Tailwind, MapLibre GL, ECharts (or lightweight),89WebSocket/SSE. Few dependencies, no SaaS-template look.9091## 26–36. Design & pages92Dark-first, technical, restrained, premium, dense but readable, minimal cards, data is the design, animated values,93strong typography, subtle grid, smooth real-time transitions. Avoid huge rounded boxes, generic gradients,94glassmorphism, childish icons, oversized marketing, fake visualisations. **Homepage is the product**: above the fold95"GLOBAL INTERNET PRESSURE 42.7 ELEVATED +6.3 / 1h", components list, live world map; live ticker (BGP updates/s,96withdrawals/s, probes active, measurements/s, targets degraded, regions elevated, DNS failures/min, median RTT,97route changes/min, active incidents) from real streams. Map modes: pressure, latency, loss, DNS, routing,98availability, incidents, probe network; drill-down world → continent → country → metro → probe → ASN without99pretending precision. Regional pages `/internet/<region>`, `/country/<cc>`; ASN pages `/asn/<n>`; service pages100`/service/<slug>` showing independent observation vs vendor status (the discrepancy is valuable); route explorer101(probe → ISP → transit → destination; normal vs current; added/removed hops, ASN changes, latency shifts); incident102pages `/event/<slug>` kept forever; `/history`, `/history/<year>`, `/history/<year>/<month>`; Global Internet Clock103("Right now: 28,412 probe measurements/min, 8,441 BGP updates/min, 17 regions normal, 3 elevated, 0 severe").104105## 37–39. API, live stream, admin106Public API `/api/v1/pressure/global|country/<cc>|asn/<n>`, `/incidents`, `/bgp/stats`, `/latency` (rate-limited free107tier; the frontend uses the same API). Live `/api/v1/live` (SSE preferred): global/regional updates, incidents, bgp108stats, probe stats, service degradation. `/admin` (separate from public UI): probe health, targets & frequencies,109regions, incident review, scoring/weight config, baseline diagnostics, pipeline health, BGP collector health, storage,110raw event explorer, manual annotations.111112## 40–45. Data quality, external sources, connectors, provenance, explainability, correlation113Track probe uptime, clock drift, missing measurements, error rate, version; per signal sample size, coverage,114collector count, last update, confidence — no strong conclusions from weak coverage. External sources optional115(RIPE, RouteViews, CAIDA, ISP/cloud status pages, Cloudflare Radar, IXP stats, cable announcements): independent116connectors behind a common interface, cached raw data, provenance stored, graceful disappearance, licences respected,117no business logic in connectors. Provenance for every derived metric (source, ts, id, raw, normalised, baseline,118anomaly, contribution) so "Why is Global Pressure 67?" is answerable: "+14 from BGP churn, +11 from NA packet loss, +8119DNS failures, +6 AWS endpoint degradation, −3 Europe stable". Rule-based correlation first (BGP spike + latency +120path change + failures ⇒ higher confidence); ML later; no unnecessary AI.121122## 46–49. AI, forecast, velocity, stability123AI only for grounded summaries/explanations/parsing/ranking; never fabricated measurements. Forecast (30-min,124severe-risk %) only once enough history exists. Compute pressure, velocity (/h), acceleration (/h²), volatility to125distinguish "high but recovering" from "moderate but worsening". Optional inverse Internet Stability index (not MVP).126127## 50–57. Mobile, performance, SEO, security, deployment, domain, self-observability, self-exclusion128Dedicated mobile layouts (big number, swipeable components, map, incidents, ticker). LCP < 2 s, CLS ≈ 0, partial129re-rendering, lazy map, server-side aggregation (never millions of points). Indexable country/ASN/service/incident/130history pages; programmatic SEO only with real data. Security: strict validation, TLS, probe auth + signed telemetry,131rate limiting, admin MFA (later), secret separation, no secrets in Git, isolated DB creds, allowlists, key rotation,132audit logs; ingestion assumes hostile traffic. Deployment: MacLustr environment, Docker Compose (web, api, ingestor,133engine, bgp, postgres, clickhouse, redis, minio, nginx/caddy), auto-restart. Domain https://www.internetpressure.io134(canonical, apex redirects), HTTPS only. Monitor ourselves (API/ingest latency, queues, probe count/freshness, CH135insert rate, Redis, BGP freshness, engine time, frontend errors). **Self-exclusion: our own infrastructure failure must136never be read as an Internet outage.**137138## 58–60. Phases139**MVP**: 4–8 probes, 100–300 targets, RIS ingestion, latency/DNS/HTTP/basic traceroute, ClickHouse, baseline engine,140global + regional pressure, live homepage, map, incident detection, admin health — no auth/billing/enterprise.141**Phase 2**: 15–25 probes, RouteViews, ASN/service/country pages, richer BGP, path comparison, Pressure Fronts,142history explorer, alerting, API. **Phase 3**: 50+ probes, forecasting, event classification, ISP benchmarking, cloud143route observability, cable correlation, stability rankings, enterprise data.144145## 61–74. Business, differentiation, language, philosophy146Public dashboard stays accessible; later revenue from historical/high-res data, enterprise feeds, webhooks, ASN147monitoring, benchmarking, research datasets. Not Downdetector/Radar/RIPEstat/Pingdom/ThousandEyes/Grafana/Statuspage:148we synthesise independent telemetry into a continuously changing measure of global Internet stress. Vocabulary:149Pressure, Elevated, Rising, Falling, Stable, Developing, Recovering, Routing instability, Latency anomaly, Regional150degradation, Observed disruption, Pressure Front — never "Internet collapse/apocalypse/massive outage" without151evidence. Every chart answers: what changed, where, how large, since when, which signals, what cause, better or worse.152**No fake real-time**: never animate random numbers or synthesise events; if one real update arrives per 30 s, update153every 30 s. UTC storage, local display with UTC toggle, deterministic timelines. Development: inspect before changing,154reuse, modular services, migrations, tests for scoring (unit: normalisation, weights, anomaly, confidence, correlation;155integration: ingest, BGP, ClickHouse, Redis, API; failure: probe/BGP/Redis offline, partial DB outage, region156unavailable, clock skew, duplicates), typed interfaces, documented env vars, no hard-coded secrets, reproducible157deploys, replay capability to validate scoring on historical incidents. Success = a knowledgeable user sees pressure158rise 27 → 61, inspects the evidence and concludes something real is happening. Execution order: repo → Docker → stores159→ probe registry → target registry → Go agent → ingestion → storage → baselines → latency/DNS/availability scores →160RIS ingestion → routing score → global engine → SSE → homepage → live metrics → map → regional scores → event engine →161admin. **An instrument, not a website**: leave it open on a NOC screen and watch the Internet change.162