SPB Git

spb/earth-now

Public License

earth-now.co — real-time planetary dashboard: live world metrics modeled, not streamed.

earth-now.co v0.1 — real-time planetary dashboard

Monorepo (pnpm + Turborepo): pure CounterModel runtime (packages/counter),
statistical fitting + guardrails (packages/models: PCHIP, Fourier,
Holt-Winters, Kalman), 76-metric YAML registry with real-data fixtures,
Fastify API (REST + SSE + SVG badges + USGS/Open Notify RT pollers),
Next.js 14 live-first dashboard (light theme, tier system, per-metric
pages, country/continent rankings, session strip), 3.2 KB embeddable
widget, ingestion scaffold (USGS + NOAA CO2 parsers), infra (PM2 deploy
+ Docker Compose variant). 142 tests: golden, property-based (fast-check)
and bit-identical client/server consistency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 9, 2026) Browse files

153 changed files +15,859 −0

153 changed files
added .env.example +21 −0
@@ -0,0 +1,21 @@
1 +# earth-now.co
2 +# Author: Simon-Pierre Boucher
3 +# Contact: contact@spboucher.ai
4 +# File: .env.example
5 +# Purpose: Environment template — copy to .env; NEVER commit real secrets (ngrok token, DB password)
6 +
7 +# apps/api
8 +PORT=4000
9 +ENABLE_RT_POLLERS=1
10 +
11 +# apps/web
12 +NEXT_PUBLIC_API_URL=http://localhost:4000
13 +API_URL=http://localhost:4000
14 +
15 +# apps/ingest (Redis-less dev mode when unset)
16 +REDIS_URL=
17 +RAW_ARCHIVE_DIR=data/raw
18 +
19 +# infra (production, node m3u96b)
20 +POSTGRES_PASSWORD=
21 +NGROK_AUTHTOKEN=
added .gitignore +11 −0
@@ -0,0 +1,11 @@
1 +node_modules/
2 +dist/
3 +.next/
4 +.turbo/
5 +coverage/
6 +*.log
7 +.env
8 +.env.*
9 +!.env.example
10 +.DS_Store
11 +data/
added CLAUDE.md +205 −0
@@ -0,0 +1,205 @@
1 +# CLAUDE.md — earth-now.co
2 +
3 +## Project Overview
4 +
5 +**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. The site displays live world metrics (population, births, deaths, CO₂ emissions, temperature anomaly, forest loss, energy production, internet users, etc.) that tick continuously on screen.
6 +
7 +The fundamental difference from Worldometer: our counters are not opaque linear extrapolations. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals.
8 +
9 +Product pillars:
10 +1. **Live everywhere**: dashboard, shareable public pages (`/m/:token`), embeddable widgets (JS, iframe, SVG badge).
11 +2. **Statistical honesty**: every number is traceable (source, observation date, interpolation method, uncertainty).
12 +3. **Robustness**: resilient ingestion pipeline; never a frozen or absurd counter in production.
13 +
14 +---
15 +
16 +## Architecture Principle #1: A Counter Is a Model, Not a Stream
17 +
18 +**NEVER push one value per second over the network.** The server ships a **CounterModel**; the client animates locally.
19 +
20 +```typescript
21 +// The project's central contract — types/counter-model.ts
22 +interface CounterModel {
23 + metricId: string; // e.g. "co2_emissions_ytd"
24 + anchorValue: number; // value at the anchor point
25 + anchorTime: string; // ISO 8601 UTC of the anchor point
26 + rateFn: RateFunction; // how the value evolves from the anchor
27 + uncertainty?: { low: number; high: number }; // 90% CI at time T
28 + observedAt: string; // date of the last REAL observation
29 + sourceId: string; // key in the source registry
30 + modelVersion: string; // e.g. "seasonal-spline-v2"
31 + displayHints: { decimals: number; sigFigs?: number; unit: string };
32 +}
33 +
34 +type RateFunction =
35 + | { kind: "linear"; perSecond: number }
36 + | { kind: "piecewise"; segments: Array<{ from: string; perSecond: number }> }
37 + | { kind: "seasonal"; base: number; harmonics: Harmonic[] } // Fourier
38 + | { kind: "spline"; knots: Array<[time: string, value: number]> };
39 +```
40 +
41 +The client computes `value(t)` at 60 fps with `requestAnimationFrame` from the model. The server only pushes (via SSE) a **new model** when: new data is ingested, a new forecast arrives, or a correction is made. Result: zero per-tick traffic, perfect consistency across all clients (same model ⇒ same value at the same instant t), and the SVG badge uses the exact same function server-side.
42 +
43 +---
44 +
45 +## Statistical Methodology (the heart of the product)
46 +
47 +Every metric declares its model level in the registry. Escalation by level:
48 +
49 +**Level 0 — Linear**: `rate = (forecast_annual_value - last_obs) / remaining_seconds`. Acceptable only as a fallback or for genuinely quasi-linear metrics.
50 +
51 +**Level 1 — Seasonal**: trend + seasonality decomposition (Fourier harmonics or STL). Mandatory for: births/deaths (demographic seasonality), CO₂ concentration (Keeling curve: ±3 ppm annual cycle), energy consumption, fires/deforestation.
52 +
53 +**Level 2 — Observation/forecast fusion**: when a source publishes both delayed observations and projections (e.g. UN WPP, Global Carbon Budget, NOAA), we blend:
54 +- Interpolation via **monotone spline (PCHIP)** between observed points — never a natural cubic spline (possible overshoot ⇒ counters absurdly going backwards).
55 +- Observation→forecast junction via a **Kalman filter** or exponential weighting: near the last observation we track the observed data; further out we converge toward the forecast.
56 +- **Holt-Winters** internally for nowcasting frequently-published metrics (weekly/monthly) when the source provides no forecast.
57 +
58 +**Uncertainty**: propagate source CIs (e.g. UN low/median/high variants) into `uncertainty`. The UI may render a band. Absolute rule: `displayHints.sigFigs` caps significant digits — we NEVER display more precision than the model justifies (world population to the unit is a lie; we animate the unit digit but the tooltip shows the CI).
59 +
60 +**Validity constraints** (enforced by `packages/models/validate.ts`):
61 +- Cumulative counters (YTD births, YTD CO₂ emitted): `rateFn` strictly ≥ 0, reset on Jan 1 UTC via `piecewise`.
62 +- Stocks (population, CO₂ ppm): may move both ways, but |derivative| bounded by a max declared in the registry.
63 +- Every re-fitted model is diffed against the previous one: a jump > declared threshold ⇒ model deployment is blocked and an alert fires (no counter "teleportation" in prod; we catch up with a 60 s client-side smoothing transition).
64 +
65 +---
66 +
67 +## Metric & Source Registry
68 +
69 +Everything goes through `packages/registry/metrics/*.yaml`. A metric does not exist until it is declared there. Example:
70 +
71 +```yaml
72 +id: world_population
73 +name: { fr: "Population mondiale", en: "World population" }
74 +unit: people
75 +kind: stock
76 +model: seasonal-spline-v2 # level 2
77 +sources:
78 + - id: un_wpp_2024
79 + url: https://population.un.org/wpp/
80 + license: CC BY 3.0 IGO
81 + cadence: yearly
82 + variants: [low, median, high]
83 +refresh: on_source_update
84 +display: { decimals: 0, sigFigs: 7 }
85 +constraints: { maxAbsRatePerSec: 10 }
86 +```
87 +
88 +Priority sources: UN WPP (demographics), Global Carbon Project + NOAA GML (CO₂), Copernicus/ERA5 (temperature), FAO & Global Forest Watch (forests), Ember/IEA (energy), USGS (earthquakes — special case: true event-driven real time, no interpolation), OWID as a secondary aggregator. **Always check and honor each source's license; attribution is displayed in the UI and in API responses.**
89 +
90 +---
91 +
92 +## Architecture
93 +
94 +```
95 +apps/
96 + web/ # Next.js 14 (App Router) — dashboard, /m/:token pages, /embed
97 + api/ # Fastify (TS) — REST + SSE
98 + ingest/ # Ingestion workers (cron + BullMQ queue)
99 +packages/
100 + registry/ # Metric + source YAML (product source of truth)
101 + models/ # Interpolators, fitting, validation (pure, no I/O)
102 + counter/ # Shared client runtime: value(model, t) — used by web, widget AND badge
103 + widget/ # Embeddable widget.js (IIFE build < 15 KB gzip, zero dependencies)
104 +infra/ # Docker, Terraform, migrations
105 +```
106 +
107 +Data flow:
108 +1. `ingest` downloads/parses sources on their cadence (exponential retry, checksums, raw archive to S3 — we keep ALL raw data for re-fitting).
109 +2. Normalization → `observations` (append-only, PostgreSQL + TimescaleDB).
110 +3. `models` re-fits the CounterModel of each impacted metric → validation → versioned in DB + Redis cache.
111 +4. Redis pub/sub publish on `metric:{id}` → SSE servers relay the new model to connected clients.
112 +5. SVG/PNG badges: rendered server-side with the same `packages/counter`, CDN cache 60 s.
113 +
114 +Database (key tables): `observations` (append-only), `counter_models` (versioned, never UPDATEd), `metrics_registry_sync`, `share_tokens` (revocable, per-metric visibility), `ingest_runs` (full auditability of every run).
115 +
116 +---
117 +
118 +## Stack
119 +
120 +- **Strict TypeScript everywhere** (`"strict": true`, no unjustified `any`). Node 20+, pnpm workspaces, Turborepo.
121 +- Next.js 14 + Tailwind (web); Fastify (api); BullMQ + Redis (ingest); PostgreSQL 16 + TimescaleDB; Drizzle ORM + SQL migrations.
122 +- Statistical fitting: TS implementations in `packages/models` (PCHIP, least-squares Fourier fit, Holt-Winters, 1D Kalman). If a heavy fit justifies it, an isolated Python service (scipy/statsmodels) under `apps/ingest/fitters-py` called by the orchestrator — but prefer TS whenever possible to share code with the client.
123 +- Real time: **SSE** (EventSource) for model distribution; no WebSocket unless a future bidirectional need arises. Reconnection: `Last-Event-ID` + Redis Streams (15 min buffer).
124 +- Cloudflare CDN in front of `/badge/*` and `widget.js`.
125 +
126 +## Mandatory File Header
127 +
128 +**Every source code file in this repository (`.ts`, `.tsx`, `.js`, `.py`, `.sql`, `.sh`, config files where comments are allowed) MUST start with the following author header:**
129 +
130 +```typescript
131 +/**
132 + * earth-now.co
133 + * Author: Simon-Pierre Boucher
134 + * Contact: contact@spboucher.ai
135 + * File: <relative/path/to/file.ts>
136 + * Purpose: <one-line description of what this file does>
137 + */
138 +```
139 +
140 +Adapt the comment syntax to the language (`#` for Python/YAML/shell, `--` for SQL). Rules:
141 +- The `File:` line must match the file's actual path in the repo — update it if the file is moved/renamed.
142 +- `Purpose:` is one line, kept accurate when the file's role changes.
143 +- CI lint rule (`scripts/check-headers.ts`, run in `pnpm lint`) fails any file missing or with a malformed header. When creating a new file, Claude must add this header first, before any code.
144 +- Generated files (build output, lockfiles, migrations auto-generated by drizzle-kit) are exempt and listed in `scripts/check-headers.ignore`.
145 +
146 +## Deployment
147 +
148 +- Production runs on **node `m3u96b`**, exposed publicly through **ngrok** at **www.earth-now.co**.
149 +- Stack on `m3u96b`: Docker Compose (`infra/docker-compose.prod.yml`) running web, api, ingest, PostgreSQL/Timescale, and Redis; ngrok tunnel (reserved domain `www.earth-now.co`) fronting the web/api reverse proxy on port 8080.
150 +- Deploy flow:
151 + ```bash
152 + pnpm build # turbo build of all apps
153 + pnpm deploy:m3u96b # rsync + docker compose up -d on node m3u96b
154 + pnpm tunnel:status # verify the ngrok tunnel is up and serving www.earth-now.co
155 + ```
156 +- ngrok config lives in `infra/ngrok.yml` (reserved domain + edge). The ngrok authtoken is provided via environment/secret manager — **never commit it**.
157 +- Health check after every deploy: `GET https://www.earth-now.co/api/health` must return 200 and `GET /sse/health` must hold an SSE connection open ≥ 10 s.
158 +- Note: ngrok is the current exposure layer; keep the reverse-proxy config portable so we can swap to Cloudflare Tunnel or a plain LB later without app changes.
159 +
160 +## Commands
161 +
162 +```bash
163 +pnpm dev # whole monorepo (turbo)
164 +pnpm dev:web # frontend only
165 +pnpm test # vitest, all packages
166 +pnpm test:models # interpolator tests (the most critical ones)
167 +pnpm lint && pnpm typecheck # lint includes the file-header check
168 +pnpm db:migrate # drizzle-kit
169 +pnpm ingest:run <sourceId> # force an ingestion locally
170 +pnpm models:refit <metricId> # manual re-fit + diff report
171 +pnpm deploy:m3u96b # deploy to production node m3u96b (ngrok → www.earth-now.co)
172 +```
173 +
174 +## Code Conventions
175 +
176 +- Time: **UTC everywhere internally**, ISO 8601; convert to local time only at display. No hand-rolled date arithmetic — use `date-fns` or `Temporal`.
177 +- Units: SI in the data layer; display conversions (t → Gt, etc.) live in `displayHints`, never in the pipeline.
178 +- `packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit system clock (time is always a parameter). This is what makes them testable and shareable client/server.
179 +- Numbers: never floats for ingested cumulative values (use integers in the base unit when possible); formatting via `Intl.NumberFormat`.
180 +- Every new metric = a PR containing: registry YAML + data fixture + golden model test + source license review.
181 +- i18n: fr + en from day one (keys in the registry and `apps/web/messages`).
182 +
183 +## Testing — Non-negotiable
184 +
185 +- **Golden tests for interpolators**: for each `modelVersion`, fixtures (input data → expected values at precise instants t, defined tolerance). Any output change without a version bump = CI failure.
186 +- Property-based tests (fast-check): monotonicity of cumulatives, derivative bounds, continuity at piecewise junctions, `value(anchorTime) === anchorValue`.
187 +- Client/server consistency test: `packages/counter` run in Node and in jsdom must produce bit-identical values for the same model.
188 +- Ingestion: every source parser has fixtures of the real format (including degraded cases: truncated file, renamed columns).
189 +
190 +## Production Guardrails
191 +
192 +- A counter must **never**: go backwards if cumulative, visibly jump (> 60 s smoothing), display NaN/undefined (fallback = last frozen value + "data pending" indicator), or show precision > sigFigs.
193 +- Ingestion failing > 2× the source's cadence ⇒ "stale" badge on the metric + alert (no silent failure).
194 +- Public endpoints (`/m`, `/embed`, `/badge`, `/sse`): rate limiting per IP + token, read-only open CORS, revocable share tokens, no user data exposed.
195 +- Public /methodology page: generated from the registry — every metric exposes its source, cadence, model, and last observation there. This is a product commitment, not an option.
196 +
197 +## What Claude Must Do in This Repo
198 +
199 +1. Before adding/modifying a metric: read the registry YAML and its associated golden tests.
200 +2. Never modify `packages/models` without bumping `modelVersion` and updating the golden fixtures.
201 +3. Prefer extending an existing `RateFunction` over creating a new one; any new kind requires implementation in `packages/counter` (client + server) AND badge rendering.
202 +4. When creating any file: add the mandatory author header (Simon-Pierre Boucher / contact@spboucher.ai) before any code.
203 +5. When in doubt about source data (format, license, cadence): flag it explicitly rather than assuming.
204 +6. UI text displaying numbers goes through helpers in `packages/counter/format.ts` — never inline `toFixed`.
205 +7. After any production deploy to `m3u96b`, run the health checks against https://www.earth-now.co before declaring the deploy done.
added README.md +139 −0
@@ -0,0 +1,139 @@
1 +# 🌍 earth-now.co — La planète en direct / The planet, live
2 +
3 +<p align="center">
4 + <a href="https://www.earth-now.co"><img src="https://img.shields.io/badge/live-www.earth--now.co-2a78d6?style=flat-square" alt="Live site"></a>
5 + <img src="https://img.shields.io/badge/metrics-76-1baf7a?style=flat-square" alt="76 metrics">
6 + <img src="https://img.shields.io/badge/tests-142%20passing-0ca30c?style=flat-square" alt="142 tests">
7 + <img src="https://img.shields.io/badge/TypeScript-strict-2a78d6?style=flat-square&logo=typescript&logoColor=white" alt="TypeScript strict">
8 + <img src="https://img.shields.io/badge/widget-3.2%20KB%20gzip-eda100?style=flat-square" alt="Widget size">
9 + <img src="https://img.shields.io/badge/monorepo-pnpm%20%2B%20turborepo-4a3aa7?style=flat-square" alt="pnpm + turborepo">
10 + <img src="https://img.shields.io/badge/i18n-fr%20%C2%B7%20en-e87ba4?style=flat-square" alt="fr + en">
11 + <img src="https://img.shields.io/badge/real--time-SSE%20%2B%20USGS-e34948?style=flat-square" alt="SSE real-time">
12 +</p>
13 +
14 +**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. It displays live world metrics (population, births, CO₂, forest loss, energy, heartbeats, GDP…) that tick continuously on screen.
15 +
16 +The fundamental difference from every other live-counter site: **our counters are not opaque linear extrapolations**. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals. Every number on screen is traceable: source, license, observation date, model family, confidence interval — one click away, on every counter.
17 +
18 +---
19 +
20 +## ✨ Core principle: a counter is a model, not a stream
21 +
22 +**We never push one value per second over the network.** The server ships a `CounterModel`; every client animates the same pure function locally at 60 fps:
23 +
24 +```typescript
25 +interface CounterModel {
26 + metricId: string; // e.g. "co2_emissions_ytd"
27 + anchorValue: number; // value at the anchor point
28 + anchorTime: string; // ISO 8601 UTC
29 + rateFn: RateFunction; // linear | piecewise | seasonal (Fourier) | spline (PCHIP)
30 + uncertainty?: { low: number; high: number }; // 90 % CI
31 + observedAt: string; // date of the last REAL observation
32 + sourceId: string; // key in the source registry
33 + modelVersion: string; // e.g. "seasonal-spline-v2"
34 + displayHints: { decimals: number; sigFigs?: number; unit: string; scale?: number };
35 +}
36 +```
37 +
38 +The dashboard, the embeddable widget and the server-rendered SVG badges all evaluate the **exact same function** — bit-identical values, verified by a cross-runtime test (jsdom vs a spawned Node process). SSE only pushes *new models* (new ingestion, new forecast, correction), never per-tick values.
39 +
40 +## 📊 The 76 metrics
41 +
42 +| Domain | Count | Highlights |
43 +|---|---|---|
44 +| 👶 Population | 24 | World population (hero), births/deaths/net growth, **human heartbeats (~9.7 B/s)**, top-10 countries + 6 continents ranked live |
45 +| 💰 Economy | 9 | **World GDP (~$3.7 M/s)**, military spending + its **school-meals juxtaposition**, cars, smartphones, cement, steel, e-waste, garments |
46 +| 💧 Society | 9 | Extreme poverty, safe water, hunger, food waste, freshwater use, animals slaughtered, fish caught, **coffee cups (~26 k/s)**, food produced |
47 +| 🏥 Health | 7 | CVD/cancer/tobacco/malaria/child/road mortality (sober editorial rule), cigarettes smoked (~165 k/s) |
48 +| 📱 Tech | 6 | Emails (~4.35 M/s, day+week cycles), Google searches, data created (~6.3 k TB/s), data-center electricity |
49 +| ⚡ Energy | 5 | Electricity, renewable share, coal, oil, **solar installed while you read (~19 kW/s)** |
50 +| 🌡️ Climate | 4 | CO₂ ppm (Keeling curve fit), temperature anomaly, **1.5 °C carbon budget counting down**, years remaining |
51 +| 🏭 Emissions | 3 | CO₂ YTD / today / per second |
52 +| 🌳 Forest | 3 | Tree-cover loss YTD / today / football-pitch equivalent |
53 +| 🌊 Ocean | 2 | Sea level rise, Arctic sea ice (strong seasonal cycle) |
54 +| ⚡ Real-time | 2 | **USGS earthquakes (true event-driven, no interpolation)**, humans in space |
55 +| 🚀 Space | 2 | Earth's orbital distance (deterministic astronomy), Earth Overshoot Day countdown |
56 +
57 +**Model families in production:** `seasonal-ytd-v1` ×27 (Fourier day/week/year harmonics, Jan-1-UTC reset), `seasonal-spline-v2` ×23 (monotone PCHIP obs/forecast fusion), `derived` ×13 (exact server-side compositions), `linear-ytd-v1` ×8, `keeling-fusion-v1` ×2 (least-squares trend + annual harmonics), `static-rt-v1` ×2, `linear-stock-v1` ×1.
58 +
59 +## 🧪 Statistical honesty — the product
60 +
61 +- **Level 0** linear ⟶ **Level 1** seasonal (Fourier/STL) ⟶ **Level 2** observation/forecast fusion (monotone PCHIP splines, Kalman blending, Holt-Winters nowcasting).
62 +- **PCHIP, never natural cubic splines** — a counter can mathematically never overshoot or tick backwards (property-tested with fast-check).
63 +- `sigFigs` caps displayed precision: world population to the unit is a lie; the ticker animates but tooltips show the honest capped value + 90 % CI.
64 +- Wide-uncertainty metrics (food waste ±15 %, garments ±30 %, AI-era estimates) **always** display their interval and an "estimate" label.
65 +- Guardrails enforced at fit time: cumulative monotonicity, declared rate bounds, anchor identity, and an **anti-teleportation diff** that blocks deployment if a refit would visibly jump (clients smooth over 60 s instead).
66 +- The `/methodology` page — and a per-metric method section on every `/metric/<id>` page — is generated from the registry: source, license, cadence, model family + version, last observation, uncertainty. A product commitment, not an option.
67 +
68 +## 🗂️ Architecture
69 +
70 +```
71 +apps/
72 + web/ # Next.js 14 (App Router) — dashboard, /metric/:id, /methodology, /embed
73 + api/ # Fastify — REST + SSE + SVG badges + RT pollers (USGS, Open Notify)
74 + ingest/ # Ingestion workers (USGS, NOAA CO₂; BullMQ when Redis present)
75 +packages/
76 + registry/ # Metric & source YAML + fixtures — the product source of truth
77 + models/ # PCHIP, Fourier LSQ, Holt-Winters, Kalman, guardrail validation (pure)
78 + counter/ # Shared runtime: value(model, t), formatting, windows (pure, isomorphic)
79 + widget/ # Embeddable widget.js — IIFE, zero deps, 3.2 KB gzip (15 KB budget)
80 +infra/ # Docker Compose, nginx, ngrok, reverse proxy, migrations, deploy scripts
81 +scripts/ # check-headers.ts — mandatory author-header lint (runs in pnpm lint)
82 +```
83 +
84 +`packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit clock — time is always a parameter. That's what makes them testable and shareable client/server.
85 +
86 +**Data flow:** ingest ⟶ append-only observations ⟶ re-fit CounterModels ⟶ guardrail validation ⟶ SSE push ⟶ clients animate locally. Raw source payloads are archived with checksums so every model can be re-fitted from history.
87 +
88 +## 🖥️ The frontend
89 +
90 +- **Light-first token theme** (dark toggle), validated data-viz palette, overflow-proof ticking digits (`FitValue` measures and scales — a 17-digit heartbeat counter can't break the layout on any viewport).
91 +- **Live-first information architecture**: a computed tier system (`secondsPerVisibleTick ≤ 2.5 s` ⟶ live grid, sorted fastest-first) decides what leads the page — not hand-picking.
92 +- **"Since you arrived"** session strip (births, deaths, CO₂, forest, orbital km, solar installed) — the catalog's signature.
93 +- Live-ranked **top-10 countries** and **continents** with animated magnitude bars.
94 +- Year progress meter, pulsing live indicators (tier-driven, `prefers-reduced-motion` respected), fr/en from day one, thousands separators everywhere via `Intl.NumberFormat`.
95 +
96 +## 🧰 Commands
97 +
98 +```bash
99 +pnpm dev # whole monorepo (turbo)
100 +pnpm dev:web # frontend only
101 +pnpm test # all packages — 142 tests
102 +pnpm test:models # interpolators only (the critical ones)
103 +pnpm lint # includes the mandatory file-header check
104 +pnpm typecheck # strict TS across 10 packages
105 +pnpm ingest:run <sourceId> # force an ingestion locally (usgs_fdsn, noaa_gml_mlo)
106 +pnpm models:refit # re-fit + diff report with deployability verdicts
107 +pnpm deploy:m3u96b # deploy to production + mandatory health checks
108 +```
109 +
110 +## ✅ Testing — non-negotiable
111 +
112 +- **Golden/recovery tests** for every interpolator (synthetic Keeling recovered to < 0.05 ppm; output changes without a `modelVersion` bump fail CI).
113 +- **Property-based tests** (fast-check): cumulative monotonicity, PCHIP no-overshoot, piecewise continuity, `value(anchorTime) === anchorValue`.
114 +- **Client/server consistency**: `counterValue` in jsdom vs a separate Node process — `Object.is`-identical results.
115 +- **Ingestion fixtures** including degraded cases (truncated NOAA file, malformed USGS GeoJSON).
116 +- Registry integration test: every declared metric must load, fit, and pass all guardrails — 0 blocked models is enforced.
117 +
118 +## 🚀 Deployment
119 +
120 +Production is exposed at **https://www.earth-now.co** (reverse proxy → Next.js web + Fastify API). `pnpm deploy:m3u96b` performs rsync → install/build on the node → process restart → **mandatory health checks**: `GET /api/health` must return 200 and `/sse/health` must hold an SSE stream ≥ 10 s — the deploy is not "done" until both pass. A Docker Compose stack (TimescaleDB + Redis + nginx) ships in `infra/` for the DB-backed ingestion phase.
121 +
122 +## 📖 Reference documents
123 +
124 +- [`CLAUDE.md`](./CLAUDE.md) — the full engineering contract (architecture principles, guardrails, conventions).
125 +- [`earth-now-metrics-catalog.md`](./earth-now-metrics-catalog.md) — the complete metric catalog by domain with model levels, sources and priorities (🥇 MVP · 🥈 V1 · 🥉 V2+).
126 +
127 +## 👤 Author
128 +
129 +| | |
130 +|---|---|
131 +| **Author** | Simon-Pierre Boucher |
132 +| **Contact** | [contact@spboucher.ai](mailto:contact@spboucher.ai) |
133 +| **Site** | [www.earth-now.co](https://www.earth-now.co) |
134 +
135 +Every source file in this repository carries the mandatory author header (enforced by `scripts/check-headers.ts` in CI). Data sources are credited in the UI, in API responses and on the methodology page — licenses are tracked per source in the registry, with `licenseNote` flags on anything requiring legal review before commercial launch.
136 +
137 +---
138 +
139 +<p align="center"><em>Chaque compteur est un modèle statistique documenté — jamais une extrapolation opaque.</em></p>
added apps/api/package.json +32 −0
@@ -0,0 +1,32 @@
1 +{
2 + "name": "@earth-now/api",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "description": "Fastify REST + SSE API — serves CounterModels (never per-tick values), badges, and RT pollers",
7 + "author": "Simon-Pierre Boucher <contact@spboucher.ai>",
8 + "scripts": {
9 + "dev": "tsx watch src/server.ts",
10 + "build": "tsc -p tsconfig.json",
11 + "start": "node dist/server.js",
12 + "test": "vitest run",
13 + "typecheck": "tsc --noEmit",
14 + "lint": "echo 'lint: covered by root header check'",
15 + "db:migrate": "echo 'migrations live in infra/ — applied via docker entrypoint'",
16 + "models:refit": "tsx src/scripts/refit.ts"
17 + },
18 + "dependencies": {
19 + "@earth-now/counter": "workspace:*",
20 + "@earth-now/models": "workspace:*",
21 + "@earth-now/registry": "workspace:*",
22 + "@fastify/cors": "^9.0.1",
23 + "@fastify/rate-limit": "^9.1.0",
24 + "fastify": "^4.28.1"
25 + },
26 + "devDependencies": {
27 + "@types/node": "^20",
28 + "tsx": "^4.19.0",
29 + "typescript": "^5.5.4",
30 + "vitest": "^2.0.5"
31 + }
32 +}
added apps/api/src/badge.ts +61 −0
@@ -0,0 +1,61 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/badge.ts
6 + * Purpose: Server-side SVG badge rendering using the SAME counterValue/formatValue as every client runtime
7 + */
8 +
9 +import { type CounterModel, counterValue, formatValue } from "@earth-now/counter";
10 +import type { MetricEntry } from "@earth-now/registry";
11 +
12 +const FONT = "Verdana,Geneva,DejaVu Sans,sans-serif";
13 +const ATTRIBUTION = "earth-now.co";
14 +const HEIGHT = 28;
15 +const PAD = 12;
16 +const GAP = 12;
17 +
18 +const XML_ESCAPES: Record<string, string> = {
19 + "&": "&amp;",
20 + "<": "&lt;",
21 + ">": "&gt;",
22 + '"': "&quot;",
23 + "'": "&apos;",
24 +};
25 +
26 +function escapeXml(text: string): string {
27 + return text.replace(/[&<>"']/g, (c) => XML_ESCAPES[c] ?? c);
28 +}
29 +
30 +/**
31 + * Dark pill badge. The value is evaluated with the shared runtime at render
32 + * time; badges are static contexts so the sigFigs honesty cap applies strictly.
33 + * A metric without a deployable model renders "data pending" (never NaN).
34 + */
35 +export function renderBadgeSvg(
36 + entry: MetricEntry,
37 + model: CounterModel | undefined,
38 + nowMs: number,
39 +): string {
40 + const name = entry.name.en;
41 + const valueText = model
42 + ? `${formatValue(counterValue(model, nowMs), model.displayHints, { applySigFigs: true })} ${model.displayHints.unit}`.trim()
43 + : "data pending";
44 +
45 + // Approximate text advance widths (no font metrics server-side by design).
46 + const nameW = Math.ceil(name.length * 6.4);
47 + const valueW = Math.ceil(valueText.length * 7.4);
48 + const attrW = Math.ceil(ATTRIBUTION.length * 4.8);
49 + const width = PAD + nameW + GAP + valueW + GAP + attrW + PAD;
50 + const label = `${name}: ${valueText}`;
51 +
52 + return [
53 + `<svg xmlns="http://www.w3.org/2000/svg" width="${width}" height="${HEIGHT}" role="img" aria-label="${escapeXml(label)}">`,
54 + `<title>${escapeXml(label)} — ${ATTRIBUTION}</title>`,
55 + `<rect width="${width}" height="${HEIGHT}" rx="${HEIGHT / 2}" fill="#0f172a"/>`,
56 + `<text x="${PAD}" y="18" font-family="${FONT}" font-size="11" fill="#94a3b8">${escapeXml(name)}</text>`,
57 + `<text x="${PAD + nameW + GAP}" y="18" font-family="${FONT}" font-size="12" font-weight="bold" fill="${model ? "#f1f5f9" : "#fbbf24"}">${escapeXml(valueText)}</text>`,
58 + `<text x="${width - PAD}" y="18" text-anchor="end" font-family="${FONT}" font-size="8" fill="#64748b">${ATTRIBUTION}</text>`,
59 + `</svg>`,
60 + ].join("");
61 +}
added apps/api/src/pollers/open-notify.ts +67 −0
@@ -0,0 +1,67 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/pollers/open-notify.ts
6 + * Purpose: Open Notify poller (1 h) for humans_in_space — extremely failure-tolerant, fixture value stays as fallback
7 + */
8 +
9 +import { buildStaticRtModel } from "@earth-now/models";
10 +import { type ModelStore, displayHintsOf } from "../store.js";
11 +import { type PollerHandle, NOOP_POLLER, fetchJson, pollersEnabled } from "./poller.js";
12 +
13 +const METRIC_ID = "humans_in_space";
14 +const SOURCE_ID = "open_notify";
15 +const ASTROS_URL = "http://api.open-notify.org/astros.json";
16 +// The API is known to be flaky: only flag the metric stale after repeated failures.
17 +const MAX_CONSECUTIVE_FAILURES = 3;
18 +
19 +export const OPEN_NOTIFY_POLL_INTERVAL_MS = 60 * 60_000;
20 +
21 +async function pollOnce(store: ModelStore): Promise<void> {
22 + const data = await fetchJson(ASTROS_URL);
23 + const count = (data as { number?: unknown }).number;
24 + if (typeof count !== "number" || !Number.isInteger(count) || count < 0 || count > 100) {
25 + throw new Error("astros.json malformed or implausible 'number' field");
26 + }
27 + const entry = store.registry.byId.get(METRIC_ID);
28 + if (!entry) throw new Error(`registry has no '${METRIC_ID}' entry`);
29 + const nowIso = new Date().toISOString();
30 + store.updateRtModel(
31 + METRIC_ID,
32 + buildStaticRtModel(
33 + {
34 + metricId: METRIC_ID,
35 + sourceId: SOURCE_ID,
36 + observedAt: nowIso,
37 + displayHints: displayHintsOf(entry),
38 + },
39 + { value: count, at: nowIso },
40 + ),
41 + );
42 +}
43 +
44 +export function startOpenNotifyPoller(
45 + store: ModelStore,
46 + intervalMs: number = OPEN_NOTIFY_POLL_INTERVAL_MS,
47 +): PollerHandle {
48 + if (!pollersEnabled()) return NOOP_POLLER;
49 + let consecutiveFailures = 0;
50 + const tick = (): void => {
51 + void pollOnce(store)
52 + .then(() => {
53 + consecutiveFailures = 0;
54 + })
55 + .catch((err: unknown) => {
56 + consecutiveFailures += 1;
57 + if (consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) store.markStale(METRIC_ID);
58 + console.error(
59 + `[open-notify] poll failed ${consecutiveFailures}× (fixture/previous value keeps serving): ${String(err)}`,
60 + );
61 + });
62 + };
63 + tick();
64 + const timer = setInterval(tick, intervalMs);
65 + timer.unref();
66 + return { stop: () => clearInterval(timer) };
67 +}
added apps/api/src/pollers/poller.ts +31 −0
@@ -0,0 +1,31 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/pollers/poller.ts
6 + * Purpose: Shared RT-poller plumbing — env gate, timeout-bounded JSON fetch, handle type
7 + */
8 +
9 +export interface PollerHandle {
10 + stop(): void;
11 +}
12 +
13 +export const NOOP_POLLER: PollerHandle = { stop: () => undefined };
14 +
15 +/** RT pollers run unless explicitly disabled (tests, offline dev). */
16 +export function pollersEnabled(): boolean {
17 + return process.env.ENABLE_RT_POLLERS !== "0";
18 +}
19 +
20 +/** GET a JSON document with a hard timeout; throws on HTTP/network/abort errors. */
21 +export async function fetchJson(url: string, timeoutMs = 10_000): Promise<unknown> {
22 + const res = await fetch(url, {
23 + signal: AbortSignal.timeout(timeoutMs),
24 + headers: {
25 + accept: "application/json",
26 + "user-agent": "earth-now.co/0.1 (contact@spboucher.ai)",
27 + },
28 + });
29 + if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`);
30 + return (await res.json()) as unknown;
31 +}
added apps/api/src/pollers/usgs.ts +90 −0
@@ -0,0 +1,90 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/pollers/usgs.ts
6 + * Purpose: USGS FDSN poller (5 min) — true event-driven RT for earthquakes_24h, never interpolated, never crashes the server
7 + */
8 +
9 +import { buildStaticRtModel } from "@earth-now/models";
10 +import { type ModelStore, type QuakesSnapshot, displayHintsOf } from "../store.js";
11 +import { type PollerHandle, NOOP_POLLER, fetchJson, pollersEnabled } from "./poller.js";
12 +
13 +const METRIC_ID = "earthquakes_24h";
14 +const SOURCE_ID = "usgs_fdsn";
15 +const BASE = "https://earthquake.usgs.gov/fdsnws/event/1";
16 +const HOUR_MS = 3_600_000;
17 +
18 +export const USGS_POLL_INTERVAL_MS = 5 * 60_000;
19 +
20 +function readCount(data: unknown): number {
21 + const count = (data as { count?: unknown }).count;
22 + if (typeof count !== "number" || !Number.isFinite(count) || count < 0) {
23 + throw new Error("USGS count response malformed");
24 + }
25 + return count;
26 +}
27 +
28 +function readLastMajor(data: unknown): QuakesSnapshot["lastMajor"] {
29 + const features = (data as { features?: unknown }).features;
30 + if (!Array.isArray(features) || features.length === 0) return null;
31 + const props = (features[0] as { properties?: { mag?: unknown; place?: unknown; time?: unknown } })
32 + .properties;
33 + if (!props || typeof props.mag !== "number" || typeof props.time !== "number") return null;
34 + return {
35 + mag: props.mag,
36 + place: typeof props.place === "string" ? props.place : "unknown location",
37 + timeIso: new Date(props.time).toISOString(),
38 + };
39 +}
40 +
41 +async function pollOnce(store: ModelStore): Promise<void> {
42 + const nowMs = Date.now();
43 + const nowIso = new Date(nowMs).toISOString();
44 + const countUrl = `${BASE}/count?format=geojson&starttime=${new Date(nowMs - 24 * HOUR_MS).toISOString()}&minmagnitude=2.5`;
45 + const majorUrl = `${BASE}/query?format=geojson&starttime=${new Date(nowMs - 7 * 24 * HOUR_MS).toISOString()}&minmagnitude=5&orderby=time&limit=1`;
46 +
47 + const [countRes, majorRes] = await Promise.allSettled([fetchJson(countUrl), fetchJson(majorUrl)]);
48 + if (countRes.status === "rejected") {
49 + throw new Error(`USGS count fetch failed: ${String(countRes.reason)}`);
50 + }
51 + const count24h = readCount(countRes.value);
52 + // The "last major quake" leg is optional — keep the previous one on failure.
53 + const lastMajor =
54 + majorRes.status === "fulfilled" ? readLastMajor(majorRes.value) : store.quakes.lastMajor;
55 +
56 + const entry = store.registry.byId.get(METRIC_ID);
57 + if (!entry) throw new Error(`registry has no '${METRIC_ID}' entry`);
58 +
59 + store.updateRtModel(
60 + METRIC_ID,
61 + buildStaticRtModel(
62 + {
63 + metricId: METRIC_ID,
64 + sourceId: SOURCE_ID,
65 + observedAt: nowIso,
66 + displayHints: displayHintsOf(entry),
67 + },
68 + { value: count24h, at: nowIso },
69 + ),
70 + );
71 + store.setQuakesSnapshot({ count24h, lastMajor, fetchedAt: nowIso });
72 +}
73 +
74 +export function startUsgsPoller(
75 + store: ModelStore,
76 + intervalMs: number = USGS_POLL_INTERVAL_MS,
77 +): PollerHandle {
78 + if (!pollersEnabled()) return NOOP_POLLER;
79 + const tick = (): void => {
80 + void pollOnce(store).catch((err: unknown) => {
81 + // Keep serving the previous model — a counter must never freeze silently or show NaN.
82 + store.markStale(METRIC_ID);
83 + console.error(`[usgs] poll failed (previous model keeps serving): ${String(err)}`);
84 + });
85 + };
86 + tick();
87 + const timer = setInterval(tick, intervalMs);
88 + timer.unref();
89 + return { stop: () => clearInterval(timer) };
90 +}
added apps/api/src/routes.ts +141 −0
@@ -0,0 +1,141 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/routes.ts
6 + * Purpose: Public REST + SSE + badge routes — read-only, CORS-open, rate-limited; ships models, never per-tick values
7 + */
8 +
9 +import cors from "@fastify/cors";
10 +import rateLimit from "@fastify/rate-limit";
11 +import type { CounterModel } from "@earth-now/counter";
12 +import type { FastifyInstance } from "fastify";
13 +import { renderBadgeSvg } from "./badge.js";
14 +import type { ModelStore } from "./store.js";
15 +
16 +const SSE_HEADERS = {
17 + "content-type": "text/event-stream; charset=utf-8",
18 + "cache-control": "no-cache, no-transform",
19 + connection: "keep-alive",
20 + "x-accel-buffering": "no",
21 +} as const;
22 +
23 +function modelsSnapshot(store: ModelStore): { models: Record<string, CounterModel>; generatedAt: string } {
24 + return {
25 + models: Object.fromEntries(store.models),
26 + generatedAt: new Date().toISOString(),
27 + };
28 +}
29 +
30 +export async function registerRoutes(app: FastifyInstance, store: ModelStore): Promise<void> {
31 + // Public read-only endpoints: open CORS, GET only.
32 + await app.register(cors, { origin: "*", methods: ["GET"] });
33 + // 300 req/min per IP; SSE connections are long-lived and exempt.
34 + await app.register(rateLimit, {
35 + max: 300,
36 + timeWindow: "1 minute",
37 + allowList: (req) => req.url.startsWith("/sse/"),
38 + });
39 +
40 + app.get("/api/health", () => ({
41 + status: "ok",
42 + now: new Date().toISOString(),
43 + modelCount: store.models.size,
44 + blockedCount: store.blocked.length,
45 + uptimeSeconds: Math.round(process.uptime()),
46 + }));
47 +
48 + // Full registry (attribution included — sources+licenses are a product commitment)
49 + // plus per-metric serving status.
50 + app.get("/v1/metrics", () =>
51 + store.registry.metrics.map((entry) => {
52 + const model = store.getModel(entry.id);
53 + return {
54 + ...entry,
55 + stale: model === undefined || store.isStale(entry.id),
56 + ...(model !== undefined
57 + ? { modelVersion: model.modelVersion, observedAt: model.observedAt }
58 + : {}),
59 + };
60 + }),
61 + );
62 +
63 + app.get<{ Params: { id: string } }>("/v1/metrics/:id/model", (req, reply) => {
64 + const { id } = req.params;
65 + if (!store.registry.byId.has(id)) {
66 + return reply.code(404).send({ error: "unknown_metric", metricId: id });
67 + }
68 + const model = store.getModel(id);
69 + if (!model) return reply.code(503).send({ error: "stale", metricId: id });
70 + return model;
71 + });
72 +
73 + app.get("/v1/models", () => modelsSnapshot(store));
74 +
75 + app.get("/v1/rt/quakes", () => store.quakes);
76 +
77 + // /badge/:id.svg — same counterValue+formatValue as the widget and dashboard.
78 + app.get<{ Params: { file: string } }>("/badge/:file", (req, reply) => {
79 + const { file } = req.params;
80 + if (!file.endsWith(".svg")) return reply.code(404).send({ error: "not_found" });
81 + const id = file.slice(0, -".svg".length);
82 + const entry = store.registry.byId.get(id);
83 + if (!entry) return reply.code(404).send({ error: "unknown_metric", metricId: id });
84 + return reply
85 + .header("content-type", "image/svg+xml")
86 + .header("cache-control", "public, max-age=60")
87 + .send(renderBadgeSvg(entry, store.getModel(id), Date.now()));
88 + });
89 +
90 + // Model distribution stream: full snapshot on connect, then ONLY new models.
91 + app.get("/sse/stream", (req, reply) => {
92 + reply.hijack();
93 + const raw = reply.raw;
94 + raw.writeHead(200, SSE_HEADERS);
95 + raw.write("retry: 5000\n\n");
96 + raw.write(`event: models\ndata: ${JSON.stringify(modelsSnapshot(store))}\n\n`);
97 +
98 + const onModel = (model: CounterModel): void => {
99 + raw.write(`event: model\ndata: ${JSON.stringify(model)}\n\n`);
100 + };
101 + store.onModel(onModel);
102 +
103 + const heartbeat = setInterval(() => {
104 + raw.write(": ping\n\n");
105 + }, 15_000);
106 + heartbeat.unref();
107 +
108 + req.raw.on("close", () => {
109 + clearInterval(heartbeat);
110 + store.offModel(onModel);
111 + raw.end();
112 + });
113 + });
114 +
115 + // Deploy health check holds this open ≥ 10 s; we cap the connection at 60 s.
116 + app.get("/sse/health", (req, reply) => {
117 + reply.hijack();
118 + const raw = reply.raw;
119 + raw.writeHead(200, SSE_HEADERS);
120 + // 2 KB padding comment: tiny comment-only chunks can sit below the flush
121 + // threshold of tunnel/proxy edges (observed with ngrok) — pad past it, then
122 + // send real data events rather than bare comments.
123 + raw.write(`: ${"p".repeat(2048)}\n\n`);
124 + raw.write(`event: hb\ndata: {"t":"${new Date().toISOString()}"}\n\n`);
125 +
126 + const heartbeat = setInterval(() => {
127 + raw.write(`event: hb\ndata: {"t":"${new Date().toISOString()}"}\n\n`);
128 + }, 2_000);
129 + heartbeat.unref();
130 + const shutdown = setTimeout(() => {
131 + clearInterval(heartbeat);
132 + raw.end();
133 + }, 60_000);
134 + shutdown.unref();
135 +
136 + req.raw.on("close", () => {
137 + clearInterval(heartbeat);
138 + clearTimeout(shutdown);
139 + });
140 + });
141 +}
added apps/api/src/scripts/refit.ts +60 −0
@@ -0,0 +1,60 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/scripts/refit.ts
6 + * Purpose: CLI re-fit + diff report (pnpm models:refit) — old vs new value at now, jump, deployability gate per metric
7 + */
8 +
9 +import { counterValue, formatValue } from "@earth-now/counter";
10 +import { assertDeployable, diffModels } from "@earth-now/models";
11 +import { fitAllModels, loadFixtures, loadRegistry } from "@earth-now/registry";
12 +import { fitHorizon, metricConstraints } from "../store.js";
13 +
14 +const nowMs = Date.now();
15 +const registry = loadRegistry();
16 +const fixtures = loadFixtures();
17 +const { fromMs, toMs } = fitHorizon(nowMs);
18 +const fitOpts = { validateFromMs: fromMs, validateToMs: toMs };
19 +const horizon = { fromMs, toMs };
20 +
21 +// The currently deployed set. TODO: read from the counter_models table once the
22 +// DB layer lands — until then the fixture fit stands in for it, so the reported
23 +// jump is 0 by construction and the report exercises the full gate anyway.
24 +const previousModels = fitAllModels(registry.metrics, fixtures, fitOpts).models;
25 +const next = fitAllModels(registry.metrics, fixtures, fitOpts);
26 +
27 +let failures = 0;
28 +console.log(
29 + `refit report @ ${new Date(nowMs).toISOString()} — ${registry.metrics.length} metrics, horizon ${new Date(fromMs).toISOString()} → ${new Date(toMs).toISOString()}\n`,
30 +);
31 +
32 +for (const entry of registry.metrics) {
33 + const previous = previousModels.get(entry.id) ?? null;
34 + const model = next.models.get(entry.id);
35 +
36 + if (!model) {
37 + failures += 1;
38 + const issues = next.blocked.find((b) => b.metricId === entry.id)?.issues ?? [];
39 + const detail = issues.map((i) => `${i.code}: ${i.message}`).join("; ") || "no model produced";
40 + console.log(`✗ ${entry.id}: BLOCKED — ${detail}`);
41 + continue;
42 + }
43 +
44 + const hints = model.displayHints;
45 + const newValue = counterValue(model, nowMs);
46 + const oldText = previous ? formatValue(counterValue(previous, nowMs), hints) : "(new)";
47 + const jump = previous ? diffModels(previous, model, nowMs) : 0;
48 + const issues = assertDeployable(previous, model, metricConstraints(entry), nowMs, horizon);
49 + const ok = issues.length === 0;
50 + if (!ok) failures += 1;
51 +
52 + console.log(
53 + `${ok ? "✓" : "✗"} ${entry.id}: old=${oldText} new=${formatValue(newValue, hints)} ${hints.unit} jump=${jump.toPrecision(3)}${ok ? "" : ` — NOT DEPLOYABLE: ${issues.map((i) => i.code).join(", ")}`}`,
54 + );
55 +}
56 +
57 +console.log(
58 + `\n${failures === 0 ? "all models deployable" : `${failures} metric(s) NOT deployable — deployment must be blocked`}`,
59 +);
60 +if (failures > 0) process.exit(1);
added apps/api/src/server.ts +51 −0
@@ -0,0 +1,51 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/server.ts
6 + * Purpose: Fastify app factory (buildApp for tests) + production listener on PORT (default 4000)
7 + */
8 +
9 +import { pathToFileURL } from "node:url";
10 +import Fastify, { type FastifyInstance } from "fastify";
11 +import { startOpenNotifyPoller } from "./pollers/open-notify.js";
12 +import { startUsgsPoller } from "./pollers/usgs.js";
13 +import { registerRoutes } from "./routes.js";
14 +import { ModelStore } from "./store.js";
15 +
16 +export interface BuildAppOptions {
17 + logger?: boolean;
18 + /** Start the RT pollers (additionally gated by ENABLE_RT_POLLERS !== "0"). */
19 + pollers?: boolean;
20 +}
21 +
22 +export async function buildApp(options: BuildAppOptions = {}): Promise<FastifyInstance> {
23 + const app = Fastify({ logger: options.logger ?? true });
24 +
25 + const store = new ModelStore();
26 + store.boot();
27 + await registerRoutes(app, store);
28 +
29 + if (options.pollers ?? true) {
30 + const handles = [startUsgsPoller(store), startOpenNotifyPoller(store)];
31 + app.addHook("onClose", async () => {
32 + for (const handle of handles) handle.stop();
33 + });
34 + }
35 +
36 + return app;
37 +}
38 +
39 +const isMain =
40 + process.argv[1] !== undefined && import.meta.url === pathToFileURL(process.argv[1]).href;
41 +
42 +if (isMain) {
43 + const app = await buildApp();
44 + const port = Number(process.env.PORT ?? 4000);
45 + try {
46 + await app.listen({ port, host: "0.0.0.0" });
47 + } catch (err) {
48 + app.log.error(err);
49 + process.exit(1);
50 + }
51 +}
added apps/api/src/store.ts +171 −0
@@ -0,0 +1,171 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/src/store.ts
6 + * Purpose: In-memory model store — boots from registry+fixtures, tracks blocked/stale metrics, emits "model" events for SSE
7 + */
8 +
9 +import { EventEmitter } from "node:events";
10 +import type { CounterModel, DisplayHints } from "@earth-now/counter";
11 +import type { MetricConstraints, ValidationIssue } from "@earth-now/models";
12 +import {
13 + type MetricEntry,
14 + type Registry,
15 + fitAllModels,
16 + loadFixtures,
17 + loadRegistry,
18 +} from "@earth-now/registry";
19 +
20 +export interface BlockedMetric {
21 + metricId: string;
22 + issues: ValidationIssue[];
23 +}
24 +
25 +export interface QuakesSnapshot {
26 + count24h: number;
27 + lastMajor: { mag: number; place: string; timeIso: string } | null;
28 + fetchedAt: string;
29 +}
30 +
31 +export type ModelListener = (model: CounterModel) => void;
32 +
33 +/** Registry display block → runtime DisplayHints (same mapping the fitter uses). */
34 +export function displayHintsOf(entry: MetricEntry, locale: "fr" | "en" = "en"): DisplayHints {
35 + const hints: DisplayHints = {
36 + decimals: entry.display.decimals,
37 + unit: entry.display.unit[locale],
38 + };
39 + if (entry.display.sigFigs !== undefined) hints.sigFigs = entry.display.sigFigs;
40 + if (entry.display.scale !== undefined) hints.scale = entry.display.scale;
41 + return hints;
42 +}
43 +
44 +/** Registry constraints block → validation MetricConstraints. */
45 +export function metricConstraints(entry: MetricEntry): MetricConstraints {
46 + const c: MetricConstraints = {
47 + kind: entry.kind === "cumulative" ? "cumulative" : "stock",
48 + };
49 + if (entry.constraints.maxAbsRatePerSec !== undefined)
50 + c.maxAbsRatePerSec = entry.constraints.maxAbsRatePerSec;
51 + if (entry.constraints.maxJumpOnRefit !== undefined)
52 + c.maxJumpOnRefit = entry.constraints.maxJumpOnRefit;
53 + return c;
54 +}
55 +
56 +/** Fit/validation horizon: Jan 1 of the current UTC year → Dec 31 of the next year. */
57 +export function fitHorizon(nowMs: number): { fromMs: number; toMs: number } {
58 + const year = new Date(nowMs).getUTCFullYear();
59 + return { fromMs: Date.UTC(year, 0, 1), toMs: Date.UTC(year + 2, 0, 1) };
60 +}
61 +
62 +/**
63 + * Holds the currently deployable CounterModel of every metric. The API serves
64 + * models, never per-tick values; SSE subscribers listen to the "model" event
65 + * and only receive a payload when a NEW model is deployed.
66 + */
67 +export class ModelStore extends EventEmitter {
68 + private readonly modelMap = new Map<string, CounterModel>();
69 + private blockedList: BlockedMetric[] = [];
70 + private readonly staleSinceMap = new Map<string, string>();
71 + private loadedRegistry: Registry | null = null;
72 + private quakesSnapshot: QuakesSnapshot = {
73 + count24h: 0,
74 + lastMajor: null,
75 + fetchedAt: new Date(0).toISOString(),
76 + };
77 +
78 + /** Load registry + fixtures and fit every model. Blocked metrics are served as stale. */
79 + boot(nowMs: number = Date.now()): void {
80 + const registry = loadRegistry();
81 + const fixtures = loadFixtures();
82 + const { fromMs, toMs } = fitHorizon(nowMs);
83 + const { models, blocked } = fitAllModels(registry.metrics, fixtures, {
84 + validateFromMs: fromMs,
85 + validateToMs: toMs,
86 + });
87 +
88 + this.loadedRegistry = registry;
89 + this.modelMap.clear();
90 + for (const [id, model] of models) this.modelMap.set(id, model);
91 + this.blockedList = blocked;
92 +
93 + // Loud by design: a blocked metric must never fail silently (stale badge instead).
94 + for (const b of blocked) {
95 + console.error(`[store] BLOCKED metric '${b.metricId}' — no model served (stale):`);
96 + for (const issue of b.issues) console.error(`[store] ${issue.code}: ${issue.message}`);
97 + }
98 +
99 + // Fixture-backed defaults for /v1/rt/quakes until the USGS poller first succeeds.
100 + const quakesModel = models.get("earthquakes_24h");
101 + if (quakesModel) {
102 + this.quakesSnapshot = {
103 + count24h: quakesModel.anchorValue,
104 + lastMajor: null,
105 + fetchedAt: quakesModel.observedAt,
106 + };
107 + }
108 + }
109 +
110 + get registry(): Registry {
111 + if (!this.loadedRegistry) throw new Error("ModelStore used before boot()");
112 + return this.loadedRegistry;
113 + }
114 +
115 + get models(): ReadonlyMap<string, CounterModel> {
116 + return this.modelMap;
117 + }
118 +
119 + get blocked(): readonly BlockedMetric[] {
120 + return this.blockedList;
121 + }
122 +
123 + get quakes(): QuakesSnapshot {
124 + return this.quakesSnapshot;
125 + }
126 +
127 + getModel(metricId: string): CounterModel | undefined {
128 + return this.modelMap.get(metricId);
129 + }
130 +
131 + isStale(metricId: string): boolean {
132 + return !this.modelMap.has(metricId) || this.staleSinceMap.has(metricId);
133 + }
134 +
135 + staleSince(metricId: string): string | undefined {
136 + return this.staleSinceMap.get(metricId);
137 + }
138 +
139 + /**
140 + * Swap the model of a true-RT (event) metric and notify SSE subscribers.
141 + * Event-driven jumps are legitimate (no interpolation, no anti-teleportation
142 + * diff); every other model family MUST go through fitAllModels + validation.
143 + */
144 + updateRtModel(metricId: string, model: CounterModel): void {
145 + const entry = this.registry.byId.get(metricId);
146 + if (!entry) throw new Error(`updateRtModel: unknown metric '${metricId}'`);
147 + if (entry.level !== "rt") {
148 + throw new Error(`updateRtModel: metric '${metricId}' is not event-driven (level ${entry.level})`);
149 + }
150 + this.modelMap.set(metricId, model);
151 + this.staleSinceMap.delete(metricId);
152 + this.emit("model", model);
153 + }
154 +
155 + /** Record when a metric's ingestion started failing (previous model keeps serving). */
156 + markStale(metricId: string, atIso: string = new Date().toISOString()): void {
157 + if (!this.staleSinceMap.has(metricId)) this.staleSinceMap.set(metricId, atIso);
158 + }
159 +
160 + setQuakesSnapshot(snapshot: QuakesSnapshot): void {
161 + this.quakesSnapshot = snapshot;
162 + }
163 +
164 + onModel(listener: ModelListener): void {
165 + this.on("model", listener);
166 + }
167 +
168 + offModel(listener: ModelListener): void {
169 + this.off("model", listener);
170 + }
171 +}
added apps/api/test/api.test.ts +123 −0
@@ -0,0 +1,123 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/api/test/api.test.ts
6 + * Purpose: API integration tests via app.inject() — health, registry, models, 404s, badge SVG (pollers disabled)
7 + */
8 +
9 +// Pollers read the env at buildApp() time, so setting it here (after hoisted
10 +// imports evaluate) is safe; buildApp({ pollers: false }) is the second guard.
11 +process.env.ENABLE_RT_POLLERS = "0";
12 +
13 +import { afterAll, beforeAll, describe, expect, it } from "vitest";
14 +import { type CounterModel, counterValue } from "@earth-now/counter";
15 +import { buildApp } from "../src/server";
16 +
17 +type App = Awaited<ReturnType<typeof buildApp>>;
18 +
19 +interface HealthBody {
20 + status: string;
21 + now: string;
22 + modelCount: number;
23 + blockedCount: number;
24 + uptimeSeconds: number;
25 +}
26 +
27 +interface MetricSummary {
28 + id: string;
29 + stale: boolean;
30 + modelVersion?: string;
31 + observedAt?: string;
32 + sources: Array<{ id: string; license: string }>;
33 +}
34 +
35 +let app: App;
36 +
37 +beforeAll(async () => {
38 + app = await buildApp({ logger: false, pollers: false });
39 +});
40 +
41 +afterAll(async () => {
42 + await app.close();
43 +});
44 +
45 +describe("GET /api/health", () => {
46 + it("returns ok with model/blocked counts and uptime", async () => {
47 + const res = await app.inject({ method: "GET", url: "/api/health" });
48 + expect(res.statusCode).toBe(200);
49 + const body = JSON.parse(res.body) as HealthBody;
50 + expect(body.status).toBe("ok");
51 + expect(Number.isNaN(Date.parse(body.now))).toBe(false);
52 + expect(body.modelCount).toBeGreaterThan(0);
53 + expect(body.blockedCount).toBeGreaterThanOrEqual(0);
54 + expect(body.uptimeSeconds).toBeGreaterThanOrEqual(0);
55 + });
56 +});
57 +
58 +describe("GET /v1/metrics", () => {
59 + it("returns the full registry with source attribution and stale flags", async () => {
60 + const res = await app.inject({ method: "GET", url: "/v1/metrics" });
61 + expect(res.statusCode).toBe(200);
62 + const metrics = JSON.parse(res.body) as MetricSummary[];
63 + expect(metrics.length).toBeGreaterThanOrEqual(30);
64 + expect(metrics.map((m) => m.id)).toContain("world_population");
65 + for (const metric of metrics) {
66 + expect(typeof metric.stale).toBe("boolean");
67 + expect(metric.sources.length).toBeGreaterThan(0);
68 + for (const source of metric.sources) expect(source.license.length).toBeGreaterThan(0);
69 + }
70 + });
71 +});
72 +
73 +describe("GET /v1/metrics/:id/model", () => {
74 + it("serves a world_population model evaluating to a sane live value", async () => {
75 + const res = await app.inject({ method: "GET", url: "/v1/metrics/world_population/model" });
76 + expect(res.statusCode).toBe(200);
77 + const model = JSON.parse(res.body) as CounterModel;
78 + expect(model.metricId).toBe("world_population");
79 + const value = counterValue(model, Date.now());
80 + expect(value).toBeGreaterThan(8.2e9);
81 + expect(value).toBeLessThan(8.4e9);
82 + });
83 +
84 + it("returns 404 for an unknown metric id", async () => {
85 + const res = await app.inject({ method: "GET", url: "/v1/metrics/nope_not_a_metric/model" });
86 + expect(res.statusCode).toBe(404);
87 + });
88 +});
89 +
90 +describe("GET /badge/:id.svg", () => {
91 + it("renders an SVG badge for co2_ppm with CDN caching headers", async () => {
92 + const res = await app.inject({ method: "GET", url: "/badge/co2_ppm.svg" });
93 + expect(res.statusCode).toBe(200);
94 + expect(res.headers["content-type"]).toContain("image/svg+xml");
95 + expect(res.headers["cache-control"]).toBe("public, max-age=60");
96 + expect(res.body).toContain("<svg");
97 + expect(res.body).toContain("earth-now.co");
98 + });
99 +
100 + it("returns 404 for an unknown metric badge", async () => {
101 + const res = await app.inject({ method: "GET", url: "/badge/nope_not_a_metric.svg" });
102 + expect(res.statusCode).toBe(404);
103 + });
104 +});
105 +
106 +describe("GET /v1/models", () => {
107 + it("serves every fitted model (registry count minus blocked)", async () => {
108 + const [modelsRes, metricsRes, healthRes] = await Promise.all([
109 + app.inject({ method: "GET", url: "/v1/models" }),
110 + app.inject({ method: "GET", url: "/v1/metrics" }),
111 + app.inject({ method: "GET", url: "/api/health" }),
112 + ]);
113 + expect(modelsRes.statusCode).toBe(200);
114 + const { models, generatedAt } = JSON.parse(modelsRes.body) as {
115 + models: Record<string, CounterModel>;
116 + generatedAt: string;
117 + };
118 + const metrics = JSON.parse(metricsRes.body) as MetricSummary[];
119 + const health = JSON.parse(healthRes.body) as HealthBody;
120 + expect(Number.isNaN(Date.parse(generatedAt))).toBe(false);
121 + expect(Object.keys(models).length).toBe(metrics.length - health.blockedCount);
122 + });
123 +});
added apps/api/tsconfig.json +10 −0
@@ -0,0 +1,10 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "outDir": "dist",
5 + "rootDir": "src",
6 + "module": "NodeNext",
7 + "moduleResolution": "NodeNext"
8 + },
9 + "include": ["src"]
10 +}
added apps/ingest/package.json +29 −0
@@ -0,0 +1,29 @@
1 +{
2 + "name": "@earth-now/ingest",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "description": "Ingestion workers — source fetchers/parsers, raw archiving, cron + BullMQ scheduling",
7 + "author": "Simon-Pierre Boucher <contact@spboucher.ai>",
8 + "scripts": {
9 + "dev": "tsx watch src/index.ts",
10 + "build": "tsc -p tsconfig.json",
11 + "test": "vitest run",
12 + "typecheck": "tsc -p tsconfig.json --noEmit",
13 + "lint": "echo 'lint: covered by root header check'",
14 + "run:source": "tsx src/run-source.ts"
15 + },
16 + "dependencies": {
17 + "@earth-now/counter": "workspace:*",
18 + "@earth-now/models": "workspace:*",
19 + "@earth-now/registry": "workspace:*",
20 + "bullmq": "^5.12.0",
21 + "ioredis": "^5.4.1"
22 + },
23 + "devDependencies": {
24 + "@types/node": "^20",
25 + "tsx": "^4.19.0",
26 + "typescript": "^5.5.4",
27 + "vitest": "^2.0.5"
28 + }
29 +}
added apps/ingest/src/archive.ts +53 −0
@@ -0,0 +1,53 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/ingest/src/archive.ts
6 + * Purpose: Raw payload archive — every ingested byte is kept (with sha256) for future re-fitting; S3-shaped interface, local FS for now
7 + */
8 +
9 +import { createHash } from "node:crypto";
10 +import { mkdir, writeFile } from "node:fs/promises";
11 +import { join } from "node:path";
12 +
13 +export interface ArchiveOptions {
14 + /** Archive root. Defaults to $RAW_ARCHIVE_DIR, then "data/raw". */
15 + dir?: string;
16 + /** Timestamp override for deterministic tests. */
17 + now?: Date;
18 +}
19 +
20 +export interface ArchiveResult {
21 + rawPath: string;
22 + checksumPath: string;
23 + sha256: string;
24 +}
25 +
26 +/**
27 + * Write `${dir}/${sourceId}/${timestamp}.raw` plus a sibling `.sha256` checksum file.
28 + * We keep ALL raw data for re-fitting; when this moves to S3 only this module changes.
29 + */
30 +export async function archiveRaw(
31 + sourceId: string,
32 + payload: string | Uint8Array,
33 + options: ArchiveOptions = {},
34 +): Promise<ArchiveResult> {
35 + if (!/^[a-z0-9_-]+$/i.test(sourceId))
36 + throw new Error(`archiveRaw: invalid sourceId '${sourceId}' (path-safe [a-z0-9_-] only)`);
37 + const dir = options.dir ?? process.env.RAW_ARCHIVE_DIR ?? "data/raw";
38 + const now = options.now ?? new Date();
39 + // ISO timestamp made filename-safe: 2026-08-09T12:00:00.000Z → 2026-08-09T12-00-00-000Z
40 + const stamp = now.toISOString().replace(/[:.]/g, "-");
41 + const targetDir = join(dir, sourceId);
42 + await mkdir(targetDir, { recursive: true });
43 +
44 + const data = typeof payload === "string" ? Buffer.from(payload, "utf8") : Buffer.from(payload);
45 + const sha256 = createHash("sha256").update(data).digest("hex");
46 + const fileName = `${stamp}.raw`;
47 + const rawPath = join(targetDir, fileName);
48 + const checksumPath = `${rawPath}.sha256`;
49 +
50 + await writeFile(rawPath, data);
51 + await writeFile(checksumPath, `${sha256} ${fileName}\n`, "utf8");
52 + return { rawPath, checksumPath, sha256 };
53 +}
added apps/ingest/src/fetch-like.ts +18 −0
@@ -0,0 +1,18 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/ingest/src/fetch-like.ts
6 + * Purpose: Minimal structural fetch interface so tests inject fixture-backed fakes and never hit the network
7 + */
8 +
9 +export interface FetchResponseLike {
10 + ok: boolean;
11 + status: number;
12 + text(): Promise<string>;
13 +}
14 +
15 +export type FetchLike = (url: string) => Promise<FetchResponseLike>;
16 +
17 +/** Default implementation: the global fetch (Node 20+). Response satisfies FetchResponseLike structurally. */
18 +export const defaultFetch: FetchLike = (url) => fetch(url);
added apps/ingest/src/index.ts +114 −0
@@ -0,0 +1,114 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/ingest/src/index.ts
6 + * Purpose: Ingestion orchestrator — BullMQ repeatable jobs when REDIS_URL is set, plain setInterval scheduler otherwise; never crashes on a failing source
7 + */
8 +
9 +import { SOURCES, runSource } from "./runners.js";
10 +
11 +const log = {
12 + info: (msg: string): void => console.log(`${new Date().toISOString()} [ingest] ${msg}`),
13 + warn: (msg: string): void => console.warn(`${new Date().toISOString()} [ingest] WARN ${msg}`),
14 + error: (msg: string): void => console.error(`${new Date().toISOString()} [ingest] ERROR ${msg}`),
15 +};
16 +
17 +interface SourceState {
18 + lastSuccessMs: number | null;
19 + failureCount: number;
20 + startedMs: number;
21 +}
22 +
23 +const states = new Map<string, SourceState>(
24 + SOURCES.map((s) => [s.id, { lastSuccessMs: null, failureCount: 0, startedMs: Date.now() }]),
25 +);
26 +
27 +/**
28 + * Production guardrail: a source failing for longer than 2× its cadence must raise the
29 + * "stale" badge + alert. Stub for now — wired to the API alerting later.
30 + */
31 +function checkStale(sourceId: string, cadenceMs: number): void {
32 + const state = states.get(sourceId);
33 + if (!state) return;
34 + const reference = state.lastSuccessMs ?? state.startedMs;
35 + if (Date.now() - reference > 2 * cadenceMs) {
36 + log.warn(
37 + `[stale] ${sourceId} has had no successful ingestion for > 2× its cadence ` +
38 + `(${state.failureCount} consecutive failure(s)) — would trigger the stale badge + alert (stub)`,
39 + );
40 + }
41 +}
42 +
43 +/** Run one source, absorbing every failure: log + count, never crash the scheduler. */
44 +async function tick(sourceId: string, cadenceMs: number): Promise<void> {
45 + const state = states.get(sourceId);
46 + try {
47 + const summary = await runSource(sourceId);
48 + if (state) {
49 + state.lastSuccessMs = Date.now();
50 + state.failureCount = 0;
51 + }
52 + log.info(
53 + `${sourceId}: ok — ${summary.observationCount} observation(s), ` +
54 + `last=${summary.last ? `${summary.last.time} → ${summary.last.value}` : "n/a"}, raw=${summary.rawPath}`,
55 + );
56 + } catch (err) {
57 + if (state) state.failureCount += 1;
58 + log.error(`${sourceId}: run failed — ${err instanceof Error ? err.message : String(err)}`);
59 + checkStale(sourceId, cadenceMs);
60 + }
61 +}
62 +
63 +async function startWithRedis(redisUrl: string): Promise<void> {
64 + // bullmq/ioredis are imported ONLY when REDIS_URL is set — dev stays dependency-light.
65 + const { Queue, Worker } = await import("bullmq");
66 + const { Redis } = await import("ioredis");
67 + const connection = new Redis(redisUrl, { maxRetriesPerRequest: null });
68 +
69 + const queue = new Queue("ingest", { connection });
70 + for (const source of SOURCES) {
71 + await queue.add(
72 + source.id,
73 + {},
74 + {
75 + repeat: { every: source.cadenceMs },
76 + jobId: `repeat:${source.id}`,
77 + removeOnComplete: 100,
78 + removeOnFail: 500,
79 + },
80 + );
81 + log.info(`scheduled '${source.id}' every ${source.cadenceMs / 1000}s (BullMQ repeatable)`);
82 + }
83 +
84 + const worker = new Worker(
85 + "ingest",
86 + async (job) => {
87 + const source = SOURCES.find((s) => s.id === job.name);
88 + await tick(job.name, source?.cadenceMs ?? 0);
89 + },
90 + { connection },
91 + );
92 + worker.on("error", (err) => log.error(`worker error: ${err.message}`));
93 + log.info(`BullMQ mode: queue "ingest" on ${redisUrl}`);
94 +}
95 +
96 +function startRedisLess(): void {
97 + log.info("REDIS_URL not set — Redis-less dev mode: plain setInterval scheduler");
98 + for (const source of SOURCES) {
99 + void tick(source.id, source.cadenceMs);
100 + setInterval(() => void tick(source.id, source.cadenceMs), source.cadenceMs);
101 + log.info(`scheduled '${source.id}' every ${source.cadenceMs / 1000}s (setInterval)`);
102 + }
103 +}
104 +
105 +async function main(): Promise<void> {
106 + const redisUrl = process.env.REDIS_URL;
107 + if (redisUrl) await startWithRedis(redisUrl);
108 + else startRedisLess();
109 +}
110 +
111 +main().catch((err: unknown) => {
112 + log.error(`fatal startup error: ${err instanceof Error ? err.message : String(err)}`);
113 + process.exit(1);
114 +});
added apps/ingest/src/run-source.ts +32 −0
@@ -0,0 +1,32 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/ingest/src/run-source.ts
6 + * Purpose: CLI for `pnpm ingest:run <sourceId>` — force one ingestion locally and print a summary
7 + */
8 +
9 +import { KNOWN_SOURCE_IDS, runSource } from "./runners.js";
10 +
11 +const sourceId = process.argv[2];
12 +
13 +if (!sourceId || !KNOWN_SOURCE_IDS.includes(sourceId)) {
14 + console.error(
15 + sourceId ? `Unknown source '${sourceId}'.` : "Usage: pnpm ingest:run <sourceId>",
16 + );
17 + console.error("Known sources:");
18 + for (const id of KNOWN_SOURCE_IDS) console.error(` - ${id}`);
19 + process.exit(1);
20 +}
21 +
22 +runSource(sourceId)
23 + .then((summary) => {
24 + console.log(`✓ ${summary.sourceId}: ${summary.observationCount} observation(s)`);
25 + if (summary.first) console.log(` first: ${summary.first.time} → ${summary.first.value}`);
26 + if (summary.last) console.log(` last: ${summary.last.time} → ${summary.last.value}`);
27 + console.log(` raw: ${summary.rawPath} (sha256 ${summary.sha256.slice(0, 12)}…)`);
28 + })
29 + .catch((err: unknown) => {
30 + console.error(`✗ ${sourceId} ingestion failed:`, err instanceof Error ? err.message : err);
31 + process.exit(1);
32 + });
added apps/ingest/src/runners.ts +94 −0
@@ -0,0 +1,94 @@
1 +/**
2 + * earth-now.co
3 + * Author: Simon-Pierre Boucher
4 + * Contact: contact@spboucher.ai
5 + * File: apps/ingest/src/runners.ts
6 + * Purpose: Per-source run functions (fetch → archive raw → parse → summary) shared by the CLI and the queue worker
7 + */
8 +
9 +import { archiveRaw } from "./archive.js";
10 +import { type FetchLike, defaultFetch } from "./fetch-like.js";
11 +import { NOAA_CO2_URL, parseNoaaMonthlyCo2 } from "./sources/noaa-co2.js";
12 +import { parseUsgsCount, usgsCountUrl } from "./sources/usgs.js";
13 +
14 +/** USGS live counter parameters: quakes M ≥ 4.5 over the trailing 24 h. */
15 +export const USGS_WINDOW_HOURS = 24;
16 +// Must match the registry definition of earthquakes_24h (catalog: M ≥ 2.5).
17 +export const USGS_MIN_MAGNITUDE = 2.5;
18 +
19 +export interface ObservationPoint {
20 + time: string;
21 + value: number;
22 +}
23 +
24 +export interface RunSummary {
25 + sourceId: string;
26 + observationCount: number;
27 + first: ObservationPoint | null;
28 + last: ObservationPoint | null;
29 + rawPath: string;
30 + sha256: string;
31 +}
32 +
33 +async function fetchText(url: string, fetchImpl: FetchLike): Promise<string> {
34 + const res = await fetchImpl(url);
35 + if (!res.ok) throw new Error(`fetch failed: HTTP ${res.status} for ${url}`);
36 + return res.text();
37 +}
38 +
39 +/** USGS FDSN: one observation — the M≥4.5 quake count over the trailing 24 h window. */
40 +export async function runUsgsFdsn(fetchImpl: FetchLike = defaultFetch): Promise<RunSummary> {
41 + const url = usgsCountUrl(USGS_WINDOW_HOURS, USGS_MIN_MAGNITUDE);
42 + const text = await fetchText(url, fetchImpl);
43 + const archived = await archiveRaw("usgs_fdsn", text);
44 + const count = parseUsgsCount(JSON.parse(text));
45 + const point: ObservationPoint = { time: new Date().toISOString(), value: count };
46 + return {
47 + sourceId: "usgs_fdsn",
48 + observationCount: 1,
49 + first: point,
50 + last: point,
51 + rawPath: archived.rawPath,
52 + sha256: archived.sha256,
53 + };
54 +}
55 +
56 +/** NOAA GML Mauna Loa: full monthly CO₂ series (Keeling curve). */
57 +export async function runNoaaGmlMlo(fetchImpl: FetchLike = defaultFetch): Promise<RunSummary> {
58 + const text = await fetchText(NOAA_CO2_URL, fetchImpl);
59 + const archived = await archiveRaw("noaa_gml_mlo", text);
60 + const observations = parseNoaaMonthlyCo2(text);
61 + return {
62 + sourceId: "noaa_gml_mlo",
63 + observationCount: observations.length,
64 + first: observations[0] ?? null,
65 + last: observations[observations.length - 1] ?? null,
66 + rawPath: archived.rawPath,
67 + sha256: archived.sha256,
68 + };
69 +}
70 +
71 +export interface SourceSpec {
72 + id: string;
73 + /** Scheduling cadence in ms (usgs 5 min, noaa daily). */
74 + cadenceMs: number;
75 + run: (fetchImpl?: FetchLike) => Promise<RunSummary>;
76 +}
77 +
78 +/** Every source the ingest service knows how to run, with its scheduling cadence. */
79 +export const SOURCES: readonly SourceSpec[] = [
80 + { id: "usgs_fdsn", cadenceMs: 5 * 60_000, run: runUsgsFdsn },
81 + { id: "noaa_gml_mlo", cadenceMs: 24 * 3_600_000, run: runNoaaGmlMlo },
82 +];
83 +
84 +export const KNOWN_SOURCE_IDS: readonly string[] = SOURCES.map((s) => s.id);
85 +
86 +/** Run one source by id. Throws on unknown id or on any fetch/parse failure. */
87 +export async function runSource(
88 + sourceId: string,
89 + fetchImpl: FetchLike = defaultFetch,
90 +): Promise<RunSummary> {
91 + const spec = SOURCES.find((s) => s.id === sourceId);
92 + if (!spec) throw new Error(`Unknown source '${sourceId}'. Known: ${KNOWN_SOURCE_IDS.join(", ")}`);
93 + return spec.run(fetchImpl);
94 +}
added apps/ingest/src/sources/noaa-co2.ts +71 −0
Large diff (71 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/src/sources/usgs.ts +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/archive.test.ts +46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/co2_mm_mlo_sample.txt +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/co2_mm_mlo_truncated.txt +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/usgs_count_malformed.json +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/usgs_count_sample.json +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/usgs_query_malformed.json +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/fixtures/usgs_query_sample.json +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/noaa-co2.test.ts +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/runners.test.ts +74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/test/usgs.test.ts +99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added apps/ingest/tsconfig.json +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/embed/[id]/page.tsx +58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/globals.css +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/layout.tsx +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/methodology/page.tsx +136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/metric/[id]/page.tsx +47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/app/page.tsx +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/CountryRanking.tsx +112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/Dashboard.tsx +418 −0
Large diff (418 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/EmbedCounter.tsx +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/FitValue.tsx +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/LiveCounter.tsx +327 −0
Large diff (327 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/MetricDetail.tsx +175 −0
Large diff (175 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/components/YearProgress.tsx +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/api.ts +129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/derived.ts +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/i18n.tsx +45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/messages.ts +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/smoothing.ts +57 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/lib/tiers.ts +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/messages/en.json +87 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/messages/fr.json +87 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/next-env.d.ts +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/next.config.mjs +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/package.json +32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/postcss.config.mjs +14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/tailwind.config.ts +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/consistency-shared.ts +83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/consistency.jsdom.test.ts +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/consistency.node.test.ts +52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/derived.test.ts +202 −0
Large diff (202 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/smoothing.test.ts +95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/test/tiers.test.ts +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/tsconfig.json +17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/tsconfig.tsbuildinfo +1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/vitest.config.ts +23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added earth-now-metrics-catalog.md +234 −0
Large diff (234 lines) is not loaded by default to keep the page fast.Load diff
added infra/deploy-m3u96b-docker.sh +63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added infra/deploy-m3u96b.sh +67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added infra/docker-compose.prod.yml +93 −0
Large diff (93 lines) is not loaded by default to keep the page fast.Load diff
added infra/docker/api.Dockerfile +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added infra/docker/ingest.Dockerfile +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added infra/docker/web.Dockerfile +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added infra/migrations/001_initial.sql +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added infra/nginx.conf +68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added infra/ngrok.yml +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added infra/proxy.mjs +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added infra/tunnel-status.sh +45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added package.json +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/package.json +29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/counter-model.ts +96 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/format.ts +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/index.ts +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/pchip.ts +122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/value.ts +129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/src/windows.ts +49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/test/consistency.test.ts +107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/test/format.test.ts +85 −0
Large diff (85 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/test/property.test.ts +141 −0
Large diff (141 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/test/value.test.ts +183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added packages/counter/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/package.json +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/builders.ts +255 −0
Large diff (255 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/fourier.ts +111 −0
Large diff (111 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/holt-winters.ts +74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/index.ts +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/kalman.ts +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/linalg.ts +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/src/validate.ts +127 −0
Large diff (127 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/test/builders-validate.test.ts +183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/test/filters.test.ts +62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/test/fourier.test.ts +81 −0
Large diff (81 lines) is not loaded by default to keep the page fast.Load diff
added packages/models/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/climate.json +66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/continents.json +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/economy.json +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/emissions.json +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/energy.json +100 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/forest.json +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/health.json +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/ocean.json +67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/population-countries.json +152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/population.json +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/realtime.json +14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/society.json +188 −0
Large diff (189 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/space.json +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/fixtures/tech.json +54 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/climate.yaml +120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/continents.yaml +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/economy.yaml +231 −0
Large diff (231 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/emissions.yaml +79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/energy.yaml +132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/forest.yaml +82 −0
Large diff (82 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/health.yaml +112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/ocean.yaml +57 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/population-countries.yaml +66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/population.yaml +207 −0
Large diff (207 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/realtime.yaml +52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/society.yaml +237 −0
Large diff (237 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/space.yaml +62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/metrics/tech.yaml +163 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/package.json +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/src/fit.ts +164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/src/index.ts +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/src/load.ts +79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/src/schema.ts +136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/test/registry.test.ts +131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added packages/registry/tsconfig.json +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/package.json +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/scripts/check-size.mjs +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/src/render.ts +103 −0
Large diff (103 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/src/widget.ts +168 −0
Large diff (168 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/test/widget.test.ts +138 −0
Large diff (138 lines) is not loaded by default to keep the page fast.Load diff
added packages/widget/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-lock.yaml +3939 −0
Large diff (3,939 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-workspace.yaml +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added scripts/check-headers.ignore +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added scripts/check-headers.ts +116 −0
Large diff (116 lines) is not loaded by default to keep the page fast.Load diff
added tsconfig.base.json +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added turbo.json +24 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme