| 65 | |
65 | |
| 66 | ## Ethics & load |
66 | ## Ethics & load |
| 67 | |
67 | |
| 68 | −Per probe with 326 targets: ~7 HTTP/s, ~11 DNS/s (4 resolvers / 120 s), ~5 ICMP runs/s, ≤ 1 traceroute in flight (every 15 min |
68 | +Per probe with 2 128 targets (tiers 20 s / 45 s / 300 s): ~18 HTTP/s, ~14 DNS/s (4 resolvers / 600 s), ~18 ICMP runs/s (every 120 s), ≤ 1 traceroute in flight (every 15 min |
| 69 | for the 30 `tr: true` targets). Keep-alive disabled (to measure TCP/TLS), body read capped at 16 KiB, no retries, no parallel |
69 | for the 30 `tr: true` targets). Keep-alive disabled (to measure TCP/TLS), body read capped at 16 KiB, no retries, no parallel |
| 70 | −checks to one host, intervals never below 10 s. Total ≈ 25 measurements/s/probe → ~17 M rows/day for 8 probes (~1 GB/day |
70 | +checks to one host, intervals never below 10 s. Total ≈ 50 measurements/s/probe → ~35 M rows/day for 8 probes (~1–2 GB/day |
| 71 | compressed in ClickHouse). |
71 | compressed in ClickHouse). |
| 72 | |
72 | |
| 73 | ## Replay |
73 | ## Replay |
| 76 | history with alternative weights. Re-deriving components from raw measurements with alternative engine parameters is an |
76 | history with alternative weights. Re-deriving components from raw measurements with alternative engine parameters is an |
| 77 | offline job to add when enough history exists (raw is kept 180 days for exactly that purpose). |
77 | offline job to add when enough history exists (raw is kept 180 days for exactly that purpose). |
| 78 | |
78 | |
| 79 | −## Retention |
79 | +## Retention — we never erase (founder decision, 2026-09-12, migration `0002_keep_forever.sql`) |
| 80 | |
80 | |
| 81 | −measurements 180 d · measurements_1m 1 y · traceroutes 1 y · bgp_events 3 d (1/25 sampled announcements, all withdrawals) · |
81 | +Every table is kept indefinitely: measurements, measurements_1m, traceroutes, probe_health, bgp_stats_10s, bgp_origin_1m, |
| 82 | −bgp_stats_10s ∞ · bgp_origin_1m 30 d · pressure_history ∞ · signal_features 1 y · probe_health 180 d · engine_runs 90 d. |
82 | +pressure_history, signal_features, engine_runs. The single exception is `bgp_events`, the *raw* RIS Live stream (≈ 10 k |
| |
83 | +prefixes/s at the source, already sampled 1/25 for announcements): 90 days. Nightly backups (`deploy/bin/backup.sh`) copy |
| |
84 | +Postgres and the irreplaceable ClickHouse tables off-node to BHS128. Disk growth ≈ 1–2 GB/day on BHS64b's 878 GB RAID-1; |
| |
85 | +plan a second ClickHouse volume (or a cold tier on BHS128) within a year. |
| 83 | |
86 | |