earth-now.co v0.1 — real-time planetary dashboard
Monorepo (pnpm + Turborepo): pure CounterModel runtime (packages/counter), statistical fitting + guardrails (packages/models: PCHIP, Fourier, Holt-Winters, Kalman), 76-metric YAML registry with real-data fixtures, Fastify API (REST + SSE + SVG badges + USGS/Open Notify RT pollers), Next.js 14 live-first dashboard (light theme, tier system, per-metric pages, country/continent rankings, session strip), 3.2 KB embeddable widget, ingestion scaffold (USGS + NOAA CO2 parsers), infra (PM2 deploy + Docker Compose variant). 142 tests: golden, property-based (fast-check) and bit-identical client/server consistency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
153 changed files +15,859 −0
added
.env.example
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +# earth-now.co | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# File: .env.example | |
| 5 | +# Purpose: Environment template — copy to .env; NEVER commit real secrets (ngrok token, DB password) | |
| 6 | + | |
| 7 | +# apps/api | |
| 8 | +PORT=4000 | |
| 9 | +ENABLE_RT_POLLERS=1 | |
| 10 | + | |
| 11 | +# apps/web | |
| 12 | +NEXT_PUBLIC_API_URL=http://localhost:4000 | |
| 13 | +API_URL=http://localhost:4000 | |
| 14 | + | |
| 15 | +# apps/ingest (Redis-less dev mode when unset) | |
| 16 | +REDIS_URL= | |
| 17 | +RAW_ARCHIVE_DIR=data/raw | |
| 18 | + | |
| 19 | +# infra (production, node m3u96b) | |
| 20 | +POSTGRES_PASSWORD= | |
| 21 | +NGROK_AUTHTOKEN= | |
added
.gitignore
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +node_modules/ | |
| 2 | +dist/ | |
| 3 | +.next/ | |
| 4 | +.turbo/ | |
| 5 | +coverage/ | |
| 6 | +*.log | |
| 7 | +.env | |
| 8 | +.env.* | |
| 9 | +!.env.example | |
| 10 | +.DS_Store | |
| 11 | +data/ | |
added
CLAUDE.md
+205 −0
@@ -0,0 +1,205 @@ | ||
| 1 | +# CLAUDE.md — earth-now.co | |
| 2 | + | |
| 3 | +## Project Overview | |
| 4 | + | |
| 5 | +**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. The site displays live world metrics (population, births, deaths, CO₂ emissions, temperature anomaly, forest loss, energy production, internet users, etc.) that tick continuously on screen. | |
| 6 | + | |
| 7 | +The fundamental difference from Worldometer: our counters are not opaque linear extrapolations. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals. | |
| 8 | + | |
| 9 | +Product pillars: | |
| 10 | +1. **Live everywhere**: dashboard, shareable public pages (`/m/:token`), embeddable widgets (JS, iframe, SVG badge). | |
| 11 | +2. **Statistical honesty**: every number is traceable (source, observation date, interpolation method, uncertainty). | |
| 12 | +3. **Robustness**: resilient ingestion pipeline; never a frozen or absurd counter in production. | |
| 13 | + | |
| 14 | +--- | |
| 15 | + | |
| 16 | +## Architecture Principle #1: A Counter Is a Model, Not a Stream | |
| 17 | + | |
| 18 | +**NEVER push one value per second over the network.** The server ships a **CounterModel**; the client animates locally. | |
| 19 | + | |
| 20 | +```typescript | |
| 21 | +// The project's central contract — types/counter-model.ts | |
| 22 | +interface CounterModel { | |
| 23 | + metricId: string; // e.g. "co2_emissions_ytd" | |
| 24 | + anchorValue: number; // value at the anchor point | |
| 25 | + anchorTime: string; // ISO 8601 UTC of the anchor point | |
| 26 | + rateFn: RateFunction; // how the value evolves from the anchor | |
| 27 | + uncertainty?: { low: number; high: number }; // 90% CI at time T | |
| 28 | + observedAt: string; // date of the last REAL observation | |
| 29 | + sourceId: string; // key in the source registry | |
| 30 | + modelVersion: string; // e.g. "seasonal-spline-v2" | |
| 31 | + displayHints: { decimals: number; sigFigs?: number; unit: string }; | |
| 32 | +} | |
| 33 | + | |
| 34 | +type RateFunction = | |
| 35 | + | { kind: "linear"; perSecond: number } | |
| 36 | + | { kind: "piecewise"; segments: Array<{ from: string; perSecond: number }> } | |
| 37 | + | { kind: "seasonal"; base: number; harmonics: Harmonic[] } // Fourier | |
| 38 | + | { kind: "spline"; knots: Array<[time: string, value: number]> }; | |
| 39 | +``` | |
| 40 | + | |
| 41 | +The client computes `value(t)` at 60 fps with `requestAnimationFrame` from the model. The server only pushes (via SSE) a **new model** when: new data is ingested, a new forecast arrives, or a correction is made. Result: zero per-tick traffic, perfect consistency across all clients (same model ⇒ same value at the same instant t), and the SVG badge uses the exact same function server-side. | |
| 42 | + | |
| 43 | +--- | |
| 44 | + | |
| 45 | +## Statistical Methodology (the heart of the product) | |
| 46 | + | |
| 47 | +Every metric declares its model level in the registry. Escalation by level: | |
| 48 | + | |
| 49 | +**Level 0 — Linear**: `rate = (forecast_annual_value - last_obs) / remaining_seconds`. Acceptable only as a fallback or for genuinely quasi-linear metrics. | |
| 50 | + | |
| 51 | +**Level 1 — Seasonal**: trend + seasonality decomposition (Fourier harmonics or STL). Mandatory for: births/deaths (demographic seasonality), CO₂ concentration (Keeling curve: ±3 ppm annual cycle), energy consumption, fires/deforestation. | |
| 52 | + | |
| 53 | +**Level 2 — Observation/forecast fusion**: when a source publishes both delayed observations and projections (e.g. UN WPP, Global Carbon Budget, NOAA), we blend: | |
| 54 | +- Interpolation via **monotone spline (PCHIP)** between observed points — never a natural cubic spline (possible overshoot ⇒ counters absurdly going backwards). | |
| 55 | +- Observation→forecast junction via a **Kalman filter** or exponential weighting: near the last observation we track the observed data; further out we converge toward the forecast. | |
| 56 | +- **Holt-Winters** internally for nowcasting frequently-published metrics (weekly/monthly) when the source provides no forecast. | |
| 57 | + | |
| 58 | +**Uncertainty**: propagate source CIs (e.g. UN low/median/high variants) into `uncertainty`. The UI may render a band. Absolute rule: `displayHints.sigFigs` caps significant digits — we NEVER display more precision than the model justifies (world population to the unit is a lie; we animate the unit digit but the tooltip shows the CI). | |
| 59 | + | |
| 60 | +**Validity constraints** (enforced by `packages/models/validate.ts`): | |
| 61 | +- Cumulative counters (YTD births, YTD CO₂ emitted): `rateFn` strictly ≥ 0, reset on Jan 1 UTC via `piecewise`. | |
| 62 | +- Stocks (population, CO₂ ppm): may move both ways, but |derivative| bounded by a max declared in the registry. | |
| 63 | +- Every re-fitted model is diffed against the previous one: a jump > declared threshold ⇒ model deployment is blocked and an alert fires (no counter "teleportation" in prod; we catch up with a 60 s client-side smoothing transition). | |
| 64 | + | |
| 65 | +--- | |
| 66 | + | |
| 67 | +## Metric & Source Registry | |
| 68 | + | |
| 69 | +Everything goes through `packages/registry/metrics/*.yaml`. A metric does not exist until it is declared there. Example: | |
| 70 | + | |
| 71 | +```yaml | |
| 72 | +id: world_population | |
| 73 | +name: { fr: "Population mondiale", en: "World population" } | |
| 74 | +unit: people | |
| 75 | +kind: stock | |
| 76 | +model: seasonal-spline-v2 # level 2 | |
| 77 | +sources: | |
| 78 | + - id: un_wpp_2024 | |
| 79 | + url: https://population.un.org/wpp/ | |
| 80 | + license: CC BY 3.0 IGO | |
| 81 | + cadence: yearly | |
| 82 | + variants: [low, median, high] | |
| 83 | +refresh: on_source_update | |
| 84 | +display: { decimals: 0, sigFigs: 7 } | |
| 85 | +constraints: { maxAbsRatePerSec: 10 } | |
| 86 | +``` | |
| 87 | + | |
| 88 | +Priority sources: UN WPP (demographics), Global Carbon Project + NOAA GML (CO₂), Copernicus/ERA5 (temperature), FAO & Global Forest Watch (forests), Ember/IEA (energy), USGS (earthquakes — special case: true event-driven real time, no interpolation), OWID as a secondary aggregator. **Always check and honor each source's license; attribution is displayed in the UI and in API responses.** | |
| 89 | + | |
| 90 | +--- | |
| 91 | + | |
| 92 | +## Architecture | |
| 93 | + | |
| 94 | +``` | |
| 95 | +apps/ | |
| 96 | + web/ # Next.js 14 (App Router) — dashboard, /m/:token pages, /embed | |
| 97 | + api/ # Fastify (TS) — REST + SSE | |
| 98 | + ingest/ # Ingestion workers (cron + BullMQ queue) | |
| 99 | +packages/ | |
| 100 | + registry/ # Metric + source YAML (product source of truth) | |
| 101 | + models/ # Interpolators, fitting, validation (pure, no I/O) | |
| 102 | + counter/ # Shared client runtime: value(model, t) — used by web, widget AND badge | |
| 103 | + widget/ # Embeddable widget.js (IIFE build < 15 KB gzip, zero dependencies) | |
| 104 | +infra/ # Docker, Terraform, migrations | |
| 105 | +``` | |
| 106 | + | |
| 107 | +Data flow: | |
| 108 | +1. `ingest` downloads/parses sources on their cadence (exponential retry, checksums, raw archive to S3 — we keep ALL raw data for re-fitting). | |
| 109 | +2. Normalization → `observations` (append-only, PostgreSQL + TimescaleDB). | |
| 110 | +3. `models` re-fits the CounterModel of each impacted metric → validation → versioned in DB + Redis cache. | |
| 111 | +4. Redis pub/sub publish on `metric:{id}` → SSE servers relay the new model to connected clients. | |
| 112 | +5. SVG/PNG badges: rendered server-side with the same `packages/counter`, CDN cache 60 s. | |
| 113 | + | |
| 114 | +Database (key tables): `observations` (append-only), `counter_models` (versioned, never UPDATEd), `metrics_registry_sync`, `share_tokens` (revocable, per-metric visibility), `ingest_runs` (full auditability of every run). | |
| 115 | + | |
| 116 | +--- | |
| 117 | + | |
| 118 | +## Stack | |
| 119 | + | |
| 120 | +- **Strict TypeScript everywhere** (`"strict": true`, no unjustified `any`). Node 20+, pnpm workspaces, Turborepo. | |
| 121 | +- Next.js 14 + Tailwind (web); Fastify (api); BullMQ + Redis (ingest); PostgreSQL 16 + TimescaleDB; Drizzle ORM + SQL migrations. | |
| 122 | +- Statistical fitting: TS implementations in `packages/models` (PCHIP, least-squares Fourier fit, Holt-Winters, 1D Kalman). If a heavy fit justifies it, an isolated Python service (scipy/statsmodels) under `apps/ingest/fitters-py` called by the orchestrator — but prefer TS whenever possible to share code with the client. | |
| 123 | +- Real time: **SSE** (EventSource) for model distribution; no WebSocket unless a future bidirectional need arises. Reconnection: `Last-Event-ID` + Redis Streams (15 min buffer). | |
| 124 | +- Cloudflare CDN in front of `/badge/*` and `widget.js`. | |
| 125 | + | |
| 126 | +## Mandatory File Header | |
| 127 | + | |
| 128 | +**Every source code file in this repository (`.ts`, `.tsx`, `.js`, `.py`, `.sql`, `.sh`, config files where comments are allowed) MUST start with the following author header:** | |
| 129 | + | |
| 130 | +```typescript | |
| 131 | +/** | |
| 132 | + * earth-now.co | |
| 133 | + * Author: Simon-Pierre Boucher | |
| 134 | + * Contact: contact@spboucher.ai | |
| 135 | + * File: <relative/path/to/file.ts> | |
| 136 | + * Purpose: <one-line description of what this file does> | |
| 137 | + */ | |
| 138 | +``` | |
| 139 | + | |
| 140 | +Adapt the comment syntax to the language (`#` for Python/YAML/shell, `--` for SQL). Rules: | |
| 141 | +- The `File:` line must match the file's actual path in the repo — update it if the file is moved/renamed. | |
| 142 | +- `Purpose:` is one line, kept accurate when the file's role changes. | |
| 143 | +- CI lint rule (`scripts/check-headers.ts`, run in `pnpm lint`) fails any file missing or with a malformed header. When creating a new file, Claude must add this header first, before any code. | |
| 144 | +- Generated files (build output, lockfiles, migrations auto-generated by drizzle-kit) are exempt and listed in `scripts/check-headers.ignore`. | |
| 145 | + | |
| 146 | +## Deployment | |
| 147 | + | |
| 148 | +- Production runs on **node `m3u96b`**, exposed publicly through **ngrok** at **www.earth-now.co**. | |
| 149 | +- Stack on `m3u96b`: Docker Compose (`infra/docker-compose.prod.yml`) running web, api, ingest, PostgreSQL/Timescale, and Redis; ngrok tunnel (reserved domain `www.earth-now.co`) fronting the web/api reverse proxy on port 8080. | |
| 150 | +- Deploy flow: | |
| 151 | + ```bash | |
| 152 | + pnpm build # turbo build of all apps | |
| 153 | + pnpm deploy:m3u96b # rsync + docker compose up -d on node m3u96b | |
| 154 | + pnpm tunnel:status # verify the ngrok tunnel is up and serving www.earth-now.co | |
| 155 | + ``` | |
| 156 | +- ngrok config lives in `infra/ngrok.yml` (reserved domain + edge). The ngrok authtoken is provided via environment/secret manager — **never commit it**. | |
| 157 | +- Health check after every deploy: `GET https://www.earth-now.co/api/health` must return 200 and `GET /sse/health` must hold an SSE connection open ≥ 10 s. | |
| 158 | +- Note: ngrok is the current exposure layer; keep the reverse-proxy config portable so we can swap to Cloudflare Tunnel or a plain LB later without app changes. | |
| 159 | + | |
| 160 | +## Commands | |
| 161 | + | |
| 162 | +```bash | |
| 163 | +pnpm dev # whole monorepo (turbo) | |
| 164 | +pnpm dev:web # frontend only | |
| 165 | +pnpm test # vitest, all packages | |
| 166 | +pnpm test:models # interpolator tests (the most critical ones) | |
| 167 | +pnpm lint && pnpm typecheck # lint includes the file-header check | |
| 168 | +pnpm db:migrate # drizzle-kit | |
| 169 | +pnpm ingest:run <sourceId> # force an ingestion locally | |
| 170 | +pnpm models:refit <metricId> # manual re-fit + diff report | |
| 171 | +pnpm deploy:m3u96b # deploy to production node m3u96b (ngrok → www.earth-now.co) | |
| 172 | +``` | |
| 173 | + | |
| 174 | +## Code Conventions | |
| 175 | + | |
| 176 | +- Time: **UTC everywhere internally**, ISO 8601; convert to local time only at display. No hand-rolled date arithmetic — use `date-fns` or `Temporal`. | |
| 177 | +- Units: SI in the data layer; display conversions (t → Gt, etc.) live in `displayHints`, never in the pipeline. | |
| 178 | +- `packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit system clock (time is always a parameter). This is what makes them testable and shareable client/server. | |
| 179 | +- Numbers: never floats for ingested cumulative values (use integers in the base unit when possible); formatting via `Intl.NumberFormat`. | |
| 180 | +- Every new metric = a PR containing: registry YAML + data fixture + golden model test + source license review. | |
| 181 | +- i18n: fr + en from day one (keys in the registry and `apps/web/messages`). | |
| 182 | + | |
| 183 | +## Testing — Non-negotiable | |
| 184 | + | |
| 185 | +- **Golden tests for interpolators**: for each `modelVersion`, fixtures (input data → expected values at precise instants t, defined tolerance). Any output change without a version bump = CI failure. | |
| 186 | +- Property-based tests (fast-check): monotonicity of cumulatives, derivative bounds, continuity at piecewise junctions, `value(anchorTime) === anchorValue`. | |
| 187 | +- Client/server consistency test: `packages/counter` run in Node and in jsdom must produce bit-identical values for the same model. | |
| 188 | +- Ingestion: every source parser has fixtures of the real format (including degraded cases: truncated file, renamed columns). | |
| 189 | + | |
| 190 | +## Production Guardrails | |
| 191 | + | |
| 192 | +- A counter must **never**: go backwards if cumulative, visibly jump (> 60 s smoothing), display NaN/undefined (fallback = last frozen value + "data pending" indicator), or show precision > sigFigs. | |
| 193 | +- Ingestion failing > 2× the source's cadence ⇒ "stale" badge on the metric + alert (no silent failure). | |
| 194 | +- Public endpoints (`/m`, `/embed`, `/badge`, `/sse`): rate limiting per IP + token, read-only open CORS, revocable share tokens, no user data exposed. | |
| 195 | +- Public /methodology page: generated from the registry — every metric exposes its source, cadence, model, and last observation there. This is a product commitment, not an option. | |
| 196 | + | |
| 197 | +## What Claude Must Do in This Repo | |
| 198 | + | |
| 199 | +1. Before adding/modifying a metric: read the registry YAML and its associated golden tests. | |
| 200 | +2. Never modify `packages/models` without bumping `modelVersion` and updating the golden fixtures. | |
| 201 | +3. Prefer extending an existing `RateFunction` over creating a new one; any new kind requires implementation in `packages/counter` (client + server) AND badge rendering. | |
| 202 | +4. When creating any file: add the mandatory author header (Simon-Pierre Boucher / contact@spboucher.ai) before any code. | |
| 203 | +5. When in doubt about source data (format, license, cadence): flag it explicitly rather than assuming. | |
| 204 | +6. UI text displaying numbers goes through helpers in `packages/counter/format.ts` — never inline `toFixed`. | |
| 205 | +7. After any production deploy to `m3u96b`, run the health checks against https://www.earth-now.co before declaring the deploy done. | |
added
README.md
+139 −0
@@ -0,0 +1,139 @@ | ||
| 1 | +# 🌍 earth-now.co — La planète en direct / The planet, live | |
| 2 | + | |
| 3 | +<p align="center"> | |
| 4 | + <a href="https://www.earth-now.co"><img src="https://img.shields.io/badge/live-www.earth--now.co-2a78d6?style=flat-square" alt="Live site"></a> | |
| 5 | + <img src="https://img.shields.io/badge/metrics-76-1baf7a?style=flat-square" alt="76 metrics"> | |
| 6 | + <img src="https://img.shields.io/badge/tests-142%20passing-0ca30c?style=flat-square" alt="142 tests"> | |
| 7 | + <img src="https://img.shields.io/badge/TypeScript-strict-2a78d6?style=flat-square&logo=typescript&logoColor=white" alt="TypeScript strict"> | |
| 8 | + <img src="https://img.shields.io/badge/widget-3.2%20KB%20gzip-eda100?style=flat-square" alt="Widget size"> | |
| 9 | + <img src="https://img.shields.io/badge/monorepo-pnpm%20%2B%20turborepo-4a3aa7?style=flat-square" alt="pnpm + turborepo"> | |
| 10 | + <img src="https://img.shields.io/badge/i18n-fr%20%C2%B7%20en-e87ba4?style=flat-square" alt="fr + en"> | |
| 11 | + <img src="https://img.shields.io/badge/real--time-SSE%20%2B%20USGS-e34948?style=flat-square" alt="SSE real-time"> | |
| 12 | +</p> | |
| 13 | + | |
| 14 | +**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. It displays live world metrics (population, births, CO₂, forest loss, energy, heartbeats, GDP…) that tick continuously on screen. | |
| 15 | + | |
| 16 | +The fundamental difference from every other live-counter site: **our counters are not opaque linear extrapolations**. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals. Every number on screen is traceable: source, license, observation date, model family, confidence interval — one click away, on every counter. | |
| 17 | + | |
| 18 | +--- | |
| 19 | + | |
| 20 | +## ✨ Core principle: a counter is a model, not a stream | |
| 21 | + | |
| 22 | +**We never push one value per second over the network.** The server ships a `CounterModel`; every client animates the same pure function locally at 60 fps: | |
| 23 | + | |
| 24 | +```typescript | |
| 25 | +interface CounterModel { | |
| 26 | + metricId: string; // e.g. "co2_emissions_ytd" | |
| 27 | + anchorValue: number; // value at the anchor point | |
| 28 | + anchorTime: string; // ISO 8601 UTC | |
| 29 | + rateFn: RateFunction; // linear | piecewise | seasonal (Fourier) | spline (PCHIP) | |
| 30 | + uncertainty?: { low: number; high: number }; // 90 % CI | |
| 31 | + observedAt: string; // date of the last REAL observation | |
| 32 | + sourceId: string; // key in the source registry | |
| 33 | + modelVersion: string; // e.g. "seasonal-spline-v2" | |
| 34 | + displayHints: { decimals: number; sigFigs?: number; unit: string; scale?: number }; | |
| 35 | +} | |
| 36 | +``` | |
| 37 | + | |
| 38 | +The dashboard, the embeddable widget and the server-rendered SVG badges all evaluate the **exact same function** — bit-identical values, verified by a cross-runtime test (jsdom vs a spawned Node process). SSE only pushes *new models* (new ingestion, new forecast, correction), never per-tick values. | |
| 39 | + | |
| 40 | +## 📊 The 76 metrics | |
| 41 | + | |
| 42 | +| Domain | Count | Highlights | | |
| 43 | +|---|---|---| | |
| 44 | +| 👶 Population | 24 | World population (hero), births/deaths/net growth, **human heartbeats (~9.7 B/s)**, top-10 countries + 6 continents ranked live | | |
| 45 | +| 💰 Economy | 9 | **World GDP (~$3.7 M/s)**, military spending + its **school-meals juxtaposition**, cars, smartphones, cement, steel, e-waste, garments | | |
| 46 | +| 💧 Society | 9 | Extreme poverty, safe water, hunger, food waste, freshwater use, animals slaughtered, fish caught, **coffee cups (~26 k/s)**, food produced | | |
| 47 | +| 🏥 Health | 7 | CVD/cancer/tobacco/malaria/child/road mortality (sober editorial rule), cigarettes smoked (~165 k/s) | | |
| 48 | +| 📱 Tech | 6 | Emails (~4.35 M/s, day+week cycles), Google searches, data created (~6.3 k TB/s), data-center electricity | | |
| 49 | +| ⚡ Energy | 5 | Electricity, renewable share, coal, oil, **solar installed while you read (~19 kW/s)** | | |
| 50 | +| 🌡️ Climate | 4 | CO₂ ppm (Keeling curve fit), temperature anomaly, **1.5 °C carbon budget counting down**, years remaining | | |
| 51 | +| 🏭 Emissions | 3 | CO₂ YTD / today / per second | | |
| 52 | +| 🌳 Forest | 3 | Tree-cover loss YTD / today / football-pitch equivalent | | |
| 53 | +| 🌊 Ocean | 2 | Sea level rise, Arctic sea ice (strong seasonal cycle) | | |
| 54 | +| ⚡ Real-time | 2 | **USGS earthquakes (true event-driven, no interpolation)**, humans in space | | |
| 55 | +| 🚀 Space | 2 | Earth's orbital distance (deterministic astronomy), Earth Overshoot Day countdown | | |
| 56 | + | |
| 57 | +**Model families in production:** `seasonal-ytd-v1` ×27 (Fourier day/week/year harmonics, Jan-1-UTC reset), `seasonal-spline-v2` ×23 (monotone PCHIP obs/forecast fusion), `derived` ×13 (exact server-side compositions), `linear-ytd-v1` ×8, `keeling-fusion-v1` ×2 (least-squares trend + annual harmonics), `static-rt-v1` ×2, `linear-stock-v1` ×1. | |
| 58 | + | |
| 59 | +## 🧪 Statistical honesty — the product | |
| 60 | + | |
| 61 | +- **Level 0** linear ⟶ **Level 1** seasonal (Fourier/STL) ⟶ **Level 2** observation/forecast fusion (monotone PCHIP splines, Kalman blending, Holt-Winters nowcasting). | |
| 62 | +- **PCHIP, never natural cubic splines** — a counter can mathematically never overshoot or tick backwards (property-tested with fast-check). | |
| 63 | +- `sigFigs` caps displayed precision: world population to the unit is a lie; the ticker animates but tooltips show the honest capped value + 90 % CI. | |
| 64 | +- Wide-uncertainty metrics (food waste ±15 %, garments ±30 %, AI-era estimates) **always** display their interval and an "estimate" label. | |
| 65 | +- Guardrails enforced at fit time: cumulative monotonicity, declared rate bounds, anchor identity, and an **anti-teleportation diff** that blocks deployment if a refit would visibly jump (clients smooth over 60 s instead). | |
| 66 | +- The `/methodology` page — and a per-metric method section on every `/metric/<id>` page — is generated from the registry: source, license, cadence, model family + version, last observation, uncertainty. A product commitment, not an option. | |
| 67 | + | |
| 68 | +## 🗂️ Architecture | |
| 69 | + | |
| 70 | +``` | |
| 71 | +apps/ | |
| 72 | + web/ # Next.js 14 (App Router) — dashboard, /metric/:id, /methodology, /embed | |
| 73 | + api/ # Fastify — REST + SSE + SVG badges + RT pollers (USGS, Open Notify) | |
| 74 | + ingest/ # Ingestion workers (USGS, NOAA CO₂; BullMQ when Redis present) | |
| 75 | +packages/ | |
| 76 | + registry/ # Metric & source YAML + fixtures — the product source of truth | |
| 77 | + models/ # PCHIP, Fourier LSQ, Holt-Winters, Kalman, guardrail validation (pure) | |
| 78 | + counter/ # Shared runtime: value(model, t), formatting, windows (pure, isomorphic) | |
| 79 | + widget/ # Embeddable widget.js — IIFE, zero deps, 3.2 KB gzip (15 KB budget) | |
| 80 | +infra/ # Docker Compose, nginx, ngrok, reverse proxy, migrations, deploy scripts | |
| 81 | +scripts/ # check-headers.ts — mandatory author-header lint (runs in pnpm lint) | |
| 82 | +``` | |
| 83 | + | |
| 84 | +`packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit clock — time is always a parameter. That's what makes them testable and shareable client/server. | |
| 85 | + | |
| 86 | +**Data flow:** ingest ⟶ append-only observations ⟶ re-fit CounterModels ⟶ guardrail validation ⟶ SSE push ⟶ clients animate locally. Raw source payloads are archived with checksums so every model can be re-fitted from history. | |
| 87 | + | |
| 88 | +## 🖥️ The frontend | |
| 89 | + | |
| 90 | +- **Light-first token theme** (dark toggle), validated data-viz palette, overflow-proof ticking digits (`FitValue` measures and scales — a 17-digit heartbeat counter can't break the layout on any viewport). | |
| 91 | +- **Live-first information architecture**: a computed tier system (`secondsPerVisibleTick ≤ 2.5 s` ⟶ live grid, sorted fastest-first) decides what leads the page — not hand-picking. | |
| 92 | +- **"Since you arrived"** session strip (births, deaths, CO₂, forest, orbital km, solar installed) — the catalog's signature. | |
| 93 | +- Live-ranked **top-10 countries** and **continents** with animated magnitude bars. | |
| 94 | +- Year progress meter, pulsing live indicators (tier-driven, `prefers-reduced-motion` respected), fr/en from day one, thousands separators everywhere via `Intl.NumberFormat`. | |
| 95 | + | |
| 96 | +## 🧰 Commands | |
| 97 | + | |
| 98 | +```bash | |
| 99 | +pnpm dev # whole monorepo (turbo) | |
| 100 | +pnpm dev:web # frontend only | |
| 101 | +pnpm test # all packages — 142 tests | |
| 102 | +pnpm test:models # interpolators only (the critical ones) | |
| 103 | +pnpm lint # includes the mandatory file-header check | |
| 104 | +pnpm typecheck # strict TS across 10 packages | |
| 105 | +pnpm ingest:run <sourceId> # force an ingestion locally (usgs_fdsn, noaa_gml_mlo) | |
| 106 | +pnpm models:refit # re-fit + diff report with deployability verdicts | |
| 107 | +pnpm deploy:m3u96b # deploy to production + mandatory health checks | |
| 108 | +``` | |
| 109 | + | |
| 110 | +## ✅ Testing — non-negotiable | |
| 111 | + | |
| 112 | +- **Golden/recovery tests** for every interpolator (synthetic Keeling recovered to < 0.05 ppm; output changes without a `modelVersion` bump fail CI). | |
| 113 | +- **Property-based tests** (fast-check): cumulative monotonicity, PCHIP no-overshoot, piecewise continuity, `value(anchorTime) === anchorValue`. | |
| 114 | +- **Client/server consistency**: `counterValue` in jsdom vs a separate Node process — `Object.is`-identical results. | |
| 115 | +- **Ingestion fixtures** including degraded cases (truncated NOAA file, malformed USGS GeoJSON). | |
| 116 | +- Registry integration test: every declared metric must load, fit, and pass all guardrails — 0 blocked models is enforced. | |
| 117 | + | |
| 118 | +## 🚀 Deployment | |
| 119 | + | |
| 120 | +Production is exposed at **https://www.earth-now.co** (reverse proxy → Next.js web + Fastify API). `pnpm deploy:m3u96b` performs rsync → install/build on the node → process restart → **mandatory health checks**: `GET /api/health` must return 200 and `/sse/health` must hold an SSE stream ≥ 10 s — the deploy is not "done" until both pass. A Docker Compose stack (TimescaleDB + Redis + nginx) ships in `infra/` for the DB-backed ingestion phase. | |
| 121 | + | |
| 122 | +## 📖 Reference documents | |
| 123 | + | |
| 124 | +- [`CLAUDE.md`](./CLAUDE.md) — the full engineering contract (architecture principles, guardrails, conventions). | |
| 125 | +- [`earth-now-metrics-catalog.md`](./earth-now-metrics-catalog.md) — the complete metric catalog by domain with model levels, sources and priorities (🥇 MVP · 🥈 V1 · 🥉 V2+). | |
| 126 | + | |
| 127 | +## 👤 Author | |
| 128 | + | |
| 129 | +| | | | |
| 130 | +|---|---| | |
| 131 | +| **Author** | Simon-Pierre Boucher | | |
| 132 | +| **Contact** | [contact@spboucher.ai](mailto:contact@spboucher.ai) | | |
| 133 | +| **Site** | [www.earth-now.co](https://www.earth-now.co) | | |
| 134 | + | |
| 135 | +Every source file in this repository carries the mandatory author header (enforced by `scripts/check-headers.ts` in CI). Data sources are credited in the UI, in API responses and on the methodology page — licenses are tracked per source in the registry, with `licenseNote` flags on anything requiring legal review before commercial launch. | |
| 136 | + | |
| 137 | +--- | |
| 138 | + | |
| 139 | +<p align="center"><em>Chaque compteur est un modèle statistique documenté — jamais une extrapolation opaque.</em></p> | |
added
apps/api/package.json
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +{ | |
| 2 | + "name": "@earth-now/api", | |
| 3 | + "version": "0.1.0", | |
| 4 | + "private": true, | |
| 5 | + "type": "module", | |
| 6 | + "description": "Fastify REST + SSE API — serves CounterModels (never per-tick values), badges, and RT pollers", | |
| 7 | + "author": "Simon-Pierre Boucher <contact@spboucher.ai>", | |
| 8 | + "scripts": { | |
| 9 | + "dev": "tsx watch src/server.ts", | |
| 10 | + "build": "tsc -p tsconfig.json", | |
| 11 | + "start": "node dist/server.js", | |
| 12 | + "test": "vitest run", | |
| 13 | + "typecheck": "tsc --noEmit", | |
| 14 | + "lint": "echo 'lint: covered by root header check'", | |
| 15 | + "db:migrate": "echo 'migrations live in infra/ — applied via docker entrypoint'", | |
| 16 | + "models:refit": "tsx src/scripts/refit.ts" | |
| 17 | + }, | |
| 18 | + "dependencies": { | |
| 19 | + "@earth-now/counter": "workspace:*", | |
| 20 | + "@earth-now/models": "workspace:*", | |
| 21 | + "@earth-now/registry": "workspace:*", | |
| 22 | + "@fastify/cors": "^9.0.1", | |
| 23 | + "@fastify/rate-limit": "^9.1.0", | |
| 24 | + "fastify": "^4.28.1" | |
| 25 | + }, | |
| 26 | + "devDependencies": { | |
| 27 | + "@types/node": "^20", | |
| 28 | + "tsx": "^4.19.0", | |
| 29 | + "typescript": "^5.5.4", | |
| 30 | + "vitest": "^2.0.5" | |
| 31 | + } | |
| 32 | +} | |
added
apps/api/src/badge.ts
+61 −0
@@ -0,0 +1,61 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/badge.ts | |
| 6 | + * Purpose: Server-side SVG badge rendering using the SAME counterValue/formatValue as every client runtime | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { type CounterModel, counterValue, formatValue } from "@earth-now/counter"; | |
| 10 | +import type { MetricEntry } from "@earth-now/registry"; | |
| 11 | + | |
| 12 | +const FONT = "Verdana,Geneva,DejaVu Sans,sans-serif"; | |
| 13 | +const ATTRIBUTION = "earth-now.co"; | |
| 14 | +const HEIGHT = 28; | |
| 15 | +const PAD = 12; | |
| 16 | +const GAP = 12; | |
| 17 | + | |
| 18 | +const XML_ESCAPES: Record<string, string> = { | |
| 19 | + "&": "&", | |
| 20 | + "<": "<", | |
| 21 | + ">": ">", | |
| 22 | + '"': """, | |
| 23 | + "'": "'", | |
| 24 | +}; | |
| 25 | + | |
| 26 | +function escapeXml(text: string): string { | |
| 27 | + return text.replace(/[&<>"']/g, (c) => XML_ESCAPES[c] ?? c); | |
| 28 | +} | |
| 29 | + | |
| 30 | +/** | |
| 31 | + * Dark pill badge. The value is evaluated with the shared runtime at render | |
| 32 | + * time; badges are static contexts so the sigFigs honesty cap applies strictly. | |
| 33 | + * A metric without a deployable model renders "data pending" (never NaN). | |
| 34 | + */ | |
| 35 | +export function renderBadgeSvg( | |
| 36 | + entry: MetricEntry, | |
| 37 | + model: CounterModel | undefined, | |
| 38 | + nowMs: number, | |
| 39 | +): string { | |
| 40 | + const name = entry.name.en; | |
| 41 | + const valueText = model | |
| 42 | + ? `${formatValue(counterValue(model, nowMs), model.displayHints, { applySigFigs: true })} ${model.displayHints.unit}`.trim() | |
| 43 | + : "data pending"; | |
| 44 | + | |
| 45 | + // Approximate text advance widths (no font metrics server-side by design). | |
| 46 | + const nameW = Math.ceil(name.length * 6.4); | |
| 47 | + const valueW = Math.ceil(valueText.length * 7.4); | |
| 48 | + const attrW = Math.ceil(ATTRIBUTION.length * 4.8); | |
| 49 | + const width = PAD + nameW + GAP + valueW + GAP + attrW + PAD; | |
| 50 | + const label = `${name}: ${valueText}`; | |
| 51 | + | |
| 52 | + return [ | |
| 53 | + `<svg xmlns="http://www.w3.org/2000/svg" width="${width}" height="${HEIGHT}" role="img" aria-label="${escapeXml(label)}">`, | |
| 54 | + `<title>${escapeXml(label)} — ${ATTRIBUTION}</title>`, | |
| 55 | + `<rect width="${width}" height="${HEIGHT}" rx="${HEIGHT / 2}" fill="#0f172a"/>`, | |
| 56 | + `<text x="${PAD}" y="18" font-family="${FONT}" font-size="11" fill="#94a3b8">${escapeXml(name)}</text>`, | |
| 57 | + `<text x="${PAD + nameW + GAP}" y="18" font-family="${FONT}" font-size="12" font-weight="bold" fill="${model ? "#f1f5f9" : "#fbbf24"}">${escapeXml(valueText)}</text>`, | |
| 58 | + `<text x="${width - PAD}" y="18" text-anchor="end" font-family="${FONT}" font-size="8" fill="#64748b">${ATTRIBUTION}</text>`, | |
| 59 | + `</svg>`, | |
| 60 | + ].join(""); | |
| 61 | +} | |
added
apps/api/src/pollers/open-notify.ts
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/pollers/open-notify.ts | |
| 6 | + * Purpose: Open Notify poller (1 h) for humans_in_space — extremely failure-tolerant, fixture value stays as fallback | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { buildStaticRtModel } from "@earth-now/models"; | |
| 10 | +import { type ModelStore, displayHintsOf } from "../store.js"; | |
| 11 | +import { type PollerHandle, NOOP_POLLER, fetchJson, pollersEnabled } from "./poller.js"; | |
| 12 | + | |
| 13 | +const METRIC_ID = "humans_in_space"; | |
| 14 | +const SOURCE_ID = "open_notify"; | |
| 15 | +const ASTROS_URL = "http://api.open-notify.org/astros.json"; | |
| 16 | +// The API is known to be flaky: only flag the metric stale after repeated failures. | |
| 17 | +const MAX_CONSECUTIVE_FAILURES = 3; | |
| 18 | + | |
| 19 | +export const OPEN_NOTIFY_POLL_INTERVAL_MS = 60 * 60_000; | |
| 20 | + | |
| 21 | +async function pollOnce(store: ModelStore): Promise<void> { | |
| 22 | + const data = await fetchJson(ASTROS_URL); | |
| 23 | + const count = (data as { number?: unknown }).number; | |
| 24 | + if (typeof count !== "number" || !Number.isInteger(count) || count < 0 || count > 100) { | |
| 25 | + throw new Error("astros.json malformed or implausible 'number' field"); | |
| 26 | + } | |
| 27 | + const entry = store.registry.byId.get(METRIC_ID); | |
| 28 | + if (!entry) throw new Error(`registry has no '${METRIC_ID}' entry`); | |
| 29 | + const nowIso = new Date().toISOString(); | |
| 30 | + store.updateRtModel( | |
| 31 | + METRIC_ID, | |
| 32 | + buildStaticRtModel( | |
| 33 | + { | |
| 34 | + metricId: METRIC_ID, | |
| 35 | + sourceId: SOURCE_ID, | |
| 36 | + observedAt: nowIso, | |
| 37 | + displayHints: displayHintsOf(entry), | |
| 38 | + }, | |
| 39 | + { value: count, at: nowIso }, | |
| 40 | + ), | |
| 41 | + ); | |
| 42 | +} | |
| 43 | + | |
| 44 | +export function startOpenNotifyPoller( | |
| 45 | + store: ModelStore, | |
| 46 | + intervalMs: number = OPEN_NOTIFY_POLL_INTERVAL_MS, | |
| 47 | +): PollerHandle { | |
| 48 | + if (!pollersEnabled()) return NOOP_POLLER; | |
| 49 | + let consecutiveFailures = 0; | |
| 50 | + const tick = (): void => { | |
| 51 | + void pollOnce(store) | |
| 52 | + .then(() => { | |
| 53 | + consecutiveFailures = 0; | |
| 54 | + }) | |
| 55 | + .catch((err: unknown) => { | |
| 56 | + consecutiveFailures += 1; | |
| 57 | + if (consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) store.markStale(METRIC_ID); | |
| 58 | + console.error( | |
| 59 | + `[open-notify] poll failed ${consecutiveFailures}× (fixture/previous value keeps serving): ${String(err)}`, | |
| 60 | + ); | |
| 61 | + }); | |
| 62 | + }; | |
| 63 | + tick(); | |
| 64 | + const timer = setInterval(tick, intervalMs); | |
| 65 | + timer.unref(); | |
| 66 | + return { stop: () => clearInterval(timer) }; | |
| 67 | +} | |
added
apps/api/src/pollers/poller.ts
+31 −0
@@ -0,0 +1,31 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/pollers/poller.ts | |
| 6 | + * Purpose: Shared RT-poller plumbing — env gate, timeout-bounded JSON fetch, handle type | |
| 7 | + */ | |
| 8 | + | |
| 9 | +export interface PollerHandle { | |
| 10 | + stop(): void; | |
| 11 | +} | |
| 12 | + | |
| 13 | +export const NOOP_POLLER: PollerHandle = { stop: () => undefined }; | |
| 14 | + | |
| 15 | +/** RT pollers run unless explicitly disabled (tests, offline dev). */ | |
| 16 | +export function pollersEnabled(): boolean { | |
| 17 | + return process.env.ENABLE_RT_POLLERS !== "0"; | |
| 18 | +} | |
| 19 | + | |
| 20 | +/** GET a JSON document with a hard timeout; throws on HTTP/network/abort errors. */ | |
| 21 | +export async function fetchJson(url: string, timeoutMs = 10_000): Promise<unknown> { | |
| 22 | + const res = await fetch(url, { | |
| 23 | + signal: AbortSignal.timeout(timeoutMs), | |
| 24 | + headers: { | |
| 25 | + accept: "application/json", | |
| 26 | + "user-agent": "earth-now.co/0.1 (contact@spboucher.ai)", | |
| 27 | + }, | |
| 28 | + }); | |
| 29 | + if (!res.ok) throw new Error(`HTTP ${res.status} from ${url}`); | |
| 30 | + return (await res.json()) as unknown; | |
| 31 | +} | |
added
apps/api/src/pollers/usgs.ts
+90 −0
@@ -0,0 +1,90 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/pollers/usgs.ts | |
| 6 | + * Purpose: USGS FDSN poller (5 min) — true event-driven RT for earthquakes_24h, never interpolated, never crashes the server | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { buildStaticRtModel } from "@earth-now/models"; | |
| 10 | +import { type ModelStore, type QuakesSnapshot, displayHintsOf } from "../store.js"; | |
| 11 | +import { type PollerHandle, NOOP_POLLER, fetchJson, pollersEnabled } from "./poller.js"; | |
| 12 | + | |
| 13 | +const METRIC_ID = "earthquakes_24h"; | |
| 14 | +const SOURCE_ID = "usgs_fdsn"; | |
| 15 | +const BASE = "https://earthquake.usgs.gov/fdsnws/event/1"; | |
| 16 | +const HOUR_MS = 3_600_000; | |
| 17 | + | |
| 18 | +export const USGS_POLL_INTERVAL_MS = 5 * 60_000; | |
| 19 | + | |
| 20 | +function readCount(data: unknown): number { | |
| 21 | + const count = (data as { count?: unknown }).count; | |
| 22 | + if (typeof count !== "number" || !Number.isFinite(count) || count < 0) { | |
| 23 | + throw new Error("USGS count response malformed"); | |
| 24 | + } | |
| 25 | + return count; | |
| 26 | +} | |
| 27 | + | |
| 28 | +function readLastMajor(data: unknown): QuakesSnapshot["lastMajor"] { | |
| 29 | + const features = (data as { features?: unknown }).features; | |
| 30 | + if (!Array.isArray(features) || features.length === 0) return null; | |
| 31 | + const props = (features[0] as { properties?: { mag?: unknown; place?: unknown; time?: unknown } }) | |
| 32 | + .properties; | |
| 33 | + if (!props || typeof props.mag !== "number" || typeof props.time !== "number") return null; | |
| 34 | + return { | |
| 35 | + mag: props.mag, | |
| 36 | + place: typeof props.place === "string" ? props.place : "unknown location", | |
| 37 | + timeIso: new Date(props.time).toISOString(), | |
| 38 | + }; | |
| 39 | +} | |
| 40 | + | |
| 41 | +async function pollOnce(store: ModelStore): Promise<void> { | |
| 42 | + const nowMs = Date.now(); | |
| 43 | + const nowIso = new Date(nowMs).toISOString(); | |
| 44 | + const countUrl = `${BASE}/count?format=geojson&starttime=${new Date(nowMs - 24 * HOUR_MS).toISOString()}&minmagnitude=2.5`; | |
| 45 | + const majorUrl = `${BASE}/query?format=geojson&starttime=${new Date(nowMs - 7 * 24 * HOUR_MS).toISOString()}&minmagnitude=5&orderby=time&limit=1`; | |
| 46 | + | |
| 47 | + const [countRes, majorRes] = await Promise.allSettled([fetchJson(countUrl), fetchJson(majorUrl)]); | |
| 48 | + if (countRes.status === "rejected") { | |
| 49 | + throw new Error(`USGS count fetch failed: ${String(countRes.reason)}`); | |
| 50 | + } | |
| 51 | + const count24h = readCount(countRes.value); | |
| 52 | + // The "last major quake" leg is optional — keep the previous one on failure. | |
| 53 | + const lastMajor = | |
| 54 | + majorRes.status === "fulfilled" ? readLastMajor(majorRes.value) : store.quakes.lastMajor; | |
| 55 | + | |
| 56 | + const entry = store.registry.byId.get(METRIC_ID); | |
| 57 | + if (!entry) throw new Error(`registry has no '${METRIC_ID}' entry`); | |
| 58 | + | |
| 59 | + store.updateRtModel( | |
| 60 | + METRIC_ID, | |
| 61 | + buildStaticRtModel( | |
| 62 | + { | |
| 63 | + metricId: METRIC_ID, | |
| 64 | + sourceId: SOURCE_ID, | |
| 65 | + observedAt: nowIso, | |
| 66 | + displayHints: displayHintsOf(entry), | |
| 67 | + }, | |
| 68 | + { value: count24h, at: nowIso }, | |
| 69 | + ), | |
| 70 | + ); | |
| 71 | + store.setQuakesSnapshot({ count24h, lastMajor, fetchedAt: nowIso }); | |
| 72 | +} | |
| 73 | + | |
| 74 | +export function startUsgsPoller( | |
| 75 | + store: ModelStore, | |
| 76 | + intervalMs: number = USGS_POLL_INTERVAL_MS, | |
| 77 | +): PollerHandle { | |
| 78 | + if (!pollersEnabled()) return NOOP_POLLER; | |
| 79 | + const tick = (): void => { | |
| 80 | + void pollOnce(store).catch((err: unknown) => { | |
| 81 | + // Keep serving the previous model — a counter must never freeze silently or show NaN. | |
| 82 | + store.markStale(METRIC_ID); | |
| 83 | + console.error(`[usgs] poll failed (previous model keeps serving): ${String(err)}`); | |
| 84 | + }); | |
| 85 | + }; | |
| 86 | + tick(); | |
| 87 | + const timer = setInterval(tick, intervalMs); | |
| 88 | + timer.unref(); | |
| 89 | + return { stop: () => clearInterval(timer) }; | |
| 90 | +} | |
added
apps/api/src/routes.ts
+141 −0
@@ -0,0 +1,141 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/routes.ts | |
| 6 | + * Purpose: Public REST + SSE + badge routes — read-only, CORS-open, rate-limited; ships models, never per-tick values | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import cors from "@fastify/cors"; | |
| 10 | +import rateLimit from "@fastify/rate-limit"; | |
| 11 | +import type { CounterModel } from "@earth-now/counter"; | |
| 12 | +import type { FastifyInstance } from "fastify"; | |
| 13 | +import { renderBadgeSvg } from "./badge.js"; | |
| 14 | +import type { ModelStore } from "./store.js"; | |
| 15 | + | |
| 16 | +const SSE_HEADERS = { | |
| 17 | + "content-type": "text/event-stream; charset=utf-8", | |
| 18 | + "cache-control": "no-cache, no-transform", | |
| 19 | + connection: "keep-alive", | |
| 20 | + "x-accel-buffering": "no", | |
| 21 | +} as const; | |
| 22 | + | |
| 23 | +function modelsSnapshot(store: ModelStore): { models: Record<string, CounterModel>; generatedAt: string } { | |
| 24 | + return { | |
| 25 | + models: Object.fromEntries(store.models), | |
| 26 | + generatedAt: new Date().toISOString(), | |
| 27 | + }; | |
| 28 | +} | |
| 29 | + | |
| 30 | +export async function registerRoutes(app: FastifyInstance, store: ModelStore): Promise<void> { | |
| 31 | + // Public read-only endpoints: open CORS, GET only. | |
| 32 | + await app.register(cors, { origin: "*", methods: ["GET"] }); | |
| 33 | + // 300 req/min per IP; SSE connections are long-lived and exempt. | |
| 34 | + await app.register(rateLimit, { | |
| 35 | + max: 300, | |
| 36 | + timeWindow: "1 minute", | |
| 37 | + allowList: (req) => req.url.startsWith("/sse/"), | |
| 38 | + }); | |
| 39 | + | |
| 40 | + app.get("/api/health", () => ({ | |
| 41 | + status: "ok", | |
| 42 | + now: new Date().toISOString(), | |
| 43 | + modelCount: store.models.size, | |
| 44 | + blockedCount: store.blocked.length, | |
| 45 | + uptimeSeconds: Math.round(process.uptime()), | |
| 46 | + })); | |
| 47 | + | |
| 48 | + // Full registry (attribution included — sources+licenses are a product commitment) | |
| 49 | + // plus per-metric serving status. | |
| 50 | + app.get("/v1/metrics", () => | |
| 51 | + store.registry.metrics.map((entry) => { | |
| 52 | + const model = store.getModel(entry.id); | |
| 53 | + return { | |
| 54 | + ...entry, | |
| 55 | + stale: model === undefined || store.isStale(entry.id), | |
| 56 | + ...(model !== undefined | |
| 57 | + ? { modelVersion: model.modelVersion, observedAt: model.observedAt } | |
| 58 | + : {}), | |
| 59 | + }; | |
| 60 | + }), | |
| 61 | + ); | |
| 62 | + | |
| 63 | + app.get<{ Params: { id: string } }>("/v1/metrics/:id/model", (req, reply) => { | |
| 64 | + const { id } = req.params; | |
| 65 | + if (!store.registry.byId.has(id)) { | |
| 66 | + return reply.code(404).send({ error: "unknown_metric", metricId: id }); | |
| 67 | + } | |
| 68 | + const model = store.getModel(id); | |
| 69 | + if (!model) return reply.code(503).send({ error: "stale", metricId: id }); | |
| 70 | + return model; | |
| 71 | + }); | |
| 72 | + | |
| 73 | + app.get("/v1/models", () => modelsSnapshot(store)); | |
| 74 | + | |
| 75 | + app.get("/v1/rt/quakes", () => store.quakes); | |
| 76 | + | |
| 77 | + // /badge/:id.svg — same counterValue+formatValue as the widget and dashboard. | |
| 78 | + app.get<{ Params: { file: string } }>("/badge/:file", (req, reply) => { | |
| 79 | + const { file } = req.params; | |
| 80 | + if (!file.endsWith(".svg")) return reply.code(404).send({ error: "not_found" }); | |
| 81 | + const id = file.slice(0, -".svg".length); | |
| 82 | + const entry = store.registry.byId.get(id); | |
| 83 | + if (!entry) return reply.code(404).send({ error: "unknown_metric", metricId: id }); | |
| 84 | + return reply | |
| 85 | + .header("content-type", "image/svg+xml") | |
| 86 | + .header("cache-control", "public, max-age=60") | |
| 87 | + .send(renderBadgeSvg(entry, store.getModel(id), Date.now())); | |
| 88 | + }); | |
| 89 | + | |
| 90 | + // Model distribution stream: full snapshot on connect, then ONLY new models. | |
| 91 | + app.get("/sse/stream", (req, reply) => { | |
| 92 | + reply.hijack(); | |
| 93 | + const raw = reply.raw; | |
| 94 | + raw.writeHead(200, SSE_HEADERS); | |
| 95 | + raw.write("retry: 5000\n\n"); | |
| 96 | + raw.write(`event: models\ndata: ${JSON.stringify(modelsSnapshot(store))}\n\n`); | |
| 97 | + | |
| 98 | + const onModel = (model: CounterModel): void => { | |
| 99 | + raw.write(`event: model\ndata: ${JSON.stringify(model)}\n\n`); | |
| 100 | + }; | |
| 101 | + store.onModel(onModel); | |
| 102 | + | |
| 103 | + const heartbeat = setInterval(() => { | |
| 104 | + raw.write(": ping\n\n"); | |
| 105 | + }, 15_000); | |
| 106 | + heartbeat.unref(); | |
| 107 | + | |
| 108 | + req.raw.on("close", () => { | |
| 109 | + clearInterval(heartbeat); | |
| 110 | + store.offModel(onModel); | |
| 111 | + raw.end(); | |
| 112 | + }); | |
| 113 | + }); | |
| 114 | + | |
| 115 | + // Deploy health check holds this open ≥ 10 s; we cap the connection at 60 s. | |
| 116 | + app.get("/sse/health", (req, reply) => { | |
| 117 | + reply.hijack(); | |
| 118 | + const raw = reply.raw; | |
| 119 | + raw.writeHead(200, SSE_HEADERS); | |
| 120 | + // 2 KB padding comment: tiny comment-only chunks can sit below the flush | |
| 121 | + // threshold of tunnel/proxy edges (observed with ngrok) — pad past it, then | |
| 122 | + // send real data events rather than bare comments. | |
| 123 | + raw.write(`: ${"p".repeat(2048)}\n\n`); | |
| 124 | + raw.write(`event: hb\ndata: {"t":"${new Date().toISOString()}"}\n\n`); | |
| 125 | + | |
| 126 | + const heartbeat = setInterval(() => { | |
| 127 | + raw.write(`event: hb\ndata: {"t":"${new Date().toISOString()}"}\n\n`); | |
| 128 | + }, 2_000); | |
| 129 | + heartbeat.unref(); | |
| 130 | + const shutdown = setTimeout(() => { | |
| 131 | + clearInterval(heartbeat); | |
| 132 | + raw.end(); | |
| 133 | + }, 60_000); | |
| 134 | + shutdown.unref(); | |
| 135 | + | |
| 136 | + req.raw.on("close", () => { | |
| 137 | + clearInterval(heartbeat); | |
| 138 | + clearTimeout(shutdown); | |
| 139 | + }); | |
| 140 | + }); | |
| 141 | +} | |
added
apps/api/src/scripts/refit.ts
+60 −0
@@ -0,0 +1,60 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/scripts/refit.ts | |
| 6 | + * Purpose: CLI re-fit + diff report (pnpm models:refit) — old vs new value at now, jump, deployability gate per metric | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { counterValue, formatValue } from "@earth-now/counter"; | |
| 10 | +import { assertDeployable, diffModels } from "@earth-now/models"; | |
| 11 | +import { fitAllModels, loadFixtures, loadRegistry } from "@earth-now/registry"; | |
| 12 | +import { fitHorizon, metricConstraints } from "../store.js"; | |
| 13 | + | |
| 14 | +const nowMs = Date.now(); | |
| 15 | +const registry = loadRegistry(); | |
| 16 | +const fixtures = loadFixtures(); | |
| 17 | +const { fromMs, toMs } = fitHorizon(nowMs); | |
| 18 | +const fitOpts = { validateFromMs: fromMs, validateToMs: toMs }; | |
| 19 | +const horizon = { fromMs, toMs }; | |
| 20 | + | |
| 21 | +// The currently deployed set. TODO: read from the counter_models table once the | |
| 22 | +// DB layer lands — until then the fixture fit stands in for it, so the reported | |
| 23 | +// jump is 0 by construction and the report exercises the full gate anyway. | |
| 24 | +const previousModels = fitAllModels(registry.metrics, fixtures, fitOpts).models; | |
| 25 | +const next = fitAllModels(registry.metrics, fixtures, fitOpts); | |
| 26 | + | |
| 27 | +let failures = 0; | |
| 28 | +console.log( | |
| 29 | + `refit report @ ${new Date(nowMs).toISOString()} — ${registry.metrics.length} metrics, horizon ${new Date(fromMs).toISOString()} → ${new Date(toMs).toISOString()}\n`, | |
| 30 | +); | |
| 31 | + | |
| 32 | +for (const entry of registry.metrics) { | |
| 33 | + const previous = previousModels.get(entry.id) ?? null; | |
| 34 | + const model = next.models.get(entry.id); | |
| 35 | + | |
| 36 | + if (!model) { | |
| 37 | + failures += 1; | |
| 38 | + const issues = next.blocked.find((b) => b.metricId === entry.id)?.issues ?? []; | |
| 39 | + const detail = issues.map((i) => `${i.code}: ${i.message}`).join("; ") || "no model produced"; | |
| 40 | + console.log(`✗ ${entry.id}: BLOCKED — ${detail}`); | |
| 41 | + continue; | |
| 42 | + } | |
| 43 | + | |
| 44 | + const hints = model.displayHints; | |
| 45 | + const newValue = counterValue(model, nowMs); | |
| 46 | + const oldText = previous ? formatValue(counterValue(previous, nowMs), hints) : "(new)"; | |
| 47 | + const jump = previous ? diffModels(previous, model, nowMs) : 0; | |
| 48 | + const issues = assertDeployable(previous, model, metricConstraints(entry), nowMs, horizon); | |
| 49 | + const ok = issues.length === 0; | |
| 50 | + if (!ok) failures += 1; | |
| 51 | + | |
| 52 | + console.log( | |
| 53 | + `${ok ? "✓" : "✗"} ${entry.id}: old=${oldText} new=${formatValue(newValue, hints)} ${hints.unit} jump=${jump.toPrecision(3)}${ok ? "" : ` — NOT DEPLOYABLE: ${issues.map((i) => i.code).join(", ")}`}`, | |
| 54 | + ); | |
| 55 | +} | |
| 56 | + | |
| 57 | +console.log( | |
| 58 | + `\n${failures === 0 ? "all models deployable" : `${failures} metric(s) NOT deployable — deployment must be blocked`}`, | |
| 59 | +); | |
| 60 | +if (failures > 0) process.exit(1); | |
added
apps/api/src/server.ts
+51 −0
@@ -0,0 +1,51 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/server.ts | |
| 6 | + * Purpose: Fastify app factory (buildApp for tests) + production listener on PORT (default 4000) | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { pathToFileURL } from "node:url"; | |
| 10 | +import Fastify, { type FastifyInstance } from "fastify"; | |
| 11 | +import { startOpenNotifyPoller } from "./pollers/open-notify.js"; | |
| 12 | +import { startUsgsPoller } from "./pollers/usgs.js"; | |
| 13 | +import { registerRoutes } from "./routes.js"; | |
| 14 | +import { ModelStore } from "./store.js"; | |
| 15 | + | |
| 16 | +export interface BuildAppOptions { | |
| 17 | + logger?: boolean; | |
| 18 | + /** Start the RT pollers (additionally gated by ENABLE_RT_POLLERS !== "0"). */ | |
| 19 | + pollers?: boolean; | |
| 20 | +} | |
| 21 | + | |
| 22 | +export async function buildApp(options: BuildAppOptions = {}): Promise<FastifyInstance> { | |
| 23 | + const app = Fastify({ logger: options.logger ?? true }); | |
| 24 | + | |
| 25 | + const store = new ModelStore(); | |
| 26 | + store.boot(); | |
| 27 | + await registerRoutes(app, store); | |
| 28 | + | |
| 29 | + if (options.pollers ?? true) { | |
| 30 | + const handles = [startUsgsPoller(store), startOpenNotifyPoller(store)]; | |
| 31 | + app.addHook("onClose", async () => { | |
| 32 | + for (const handle of handles) handle.stop(); | |
| 33 | + }); | |
| 34 | + } | |
| 35 | + | |
| 36 | + return app; | |
| 37 | +} | |
| 38 | + | |
| 39 | +const isMain = | |
| 40 | + process.argv[1] !== undefined && import.meta.url === pathToFileURL(process.argv[1]).href; | |
| 41 | + | |
| 42 | +if (isMain) { | |
| 43 | + const app = await buildApp(); | |
| 44 | + const port = Number(process.env.PORT ?? 4000); | |
| 45 | + try { | |
| 46 | + await app.listen({ port, host: "0.0.0.0" }); | |
| 47 | + } catch (err) { | |
| 48 | + app.log.error(err); | |
| 49 | + process.exit(1); | |
| 50 | + } | |
| 51 | +} | |
added
apps/api/src/store.ts
+171 −0
@@ -0,0 +1,171 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/src/store.ts | |
| 6 | + * Purpose: In-memory model store — boots from registry+fixtures, tracks blocked/stale metrics, emits "model" events for SSE | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { EventEmitter } from "node:events"; | |
| 10 | +import type { CounterModel, DisplayHints } from "@earth-now/counter"; | |
| 11 | +import type { MetricConstraints, ValidationIssue } from "@earth-now/models"; | |
| 12 | +import { | |
| 13 | + type MetricEntry, | |
| 14 | + type Registry, | |
| 15 | + fitAllModels, | |
| 16 | + loadFixtures, | |
| 17 | + loadRegistry, | |
| 18 | +} from "@earth-now/registry"; | |
| 19 | + | |
| 20 | +export interface BlockedMetric { | |
| 21 | + metricId: string; | |
| 22 | + issues: ValidationIssue[]; | |
| 23 | +} | |
| 24 | + | |
| 25 | +export interface QuakesSnapshot { | |
| 26 | + count24h: number; | |
| 27 | + lastMajor: { mag: number; place: string; timeIso: string } | null; | |
| 28 | + fetchedAt: string; | |
| 29 | +} | |
| 30 | + | |
| 31 | +export type ModelListener = (model: CounterModel) => void; | |
| 32 | + | |
| 33 | +/** Registry display block → runtime DisplayHints (same mapping the fitter uses). */ | |
| 34 | +export function displayHintsOf(entry: MetricEntry, locale: "fr" | "en" = "en"): DisplayHints { | |
| 35 | + const hints: DisplayHints = { | |
| 36 | + decimals: entry.display.decimals, | |
| 37 | + unit: entry.display.unit[locale], | |
| 38 | + }; | |
| 39 | + if (entry.display.sigFigs !== undefined) hints.sigFigs = entry.display.sigFigs; | |
| 40 | + if (entry.display.scale !== undefined) hints.scale = entry.display.scale; | |
| 41 | + return hints; | |
| 42 | +} | |
| 43 | + | |
| 44 | +/** Registry constraints block → validation MetricConstraints. */ | |
| 45 | +export function metricConstraints(entry: MetricEntry): MetricConstraints { | |
| 46 | + const c: MetricConstraints = { | |
| 47 | + kind: entry.kind === "cumulative" ? "cumulative" : "stock", | |
| 48 | + }; | |
| 49 | + if (entry.constraints.maxAbsRatePerSec !== undefined) | |
| 50 | + c.maxAbsRatePerSec = entry.constraints.maxAbsRatePerSec; | |
| 51 | + if (entry.constraints.maxJumpOnRefit !== undefined) | |
| 52 | + c.maxJumpOnRefit = entry.constraints.maxJumpOnRefit; | |
| 53 | + return c; | |
| 54 | +} | |
| 55 | + | |
| 56 | +/** Fit/validation horizon: Jan 1 of the current UTC year → Dec 31 of the next year. */ | |
| 57 | +export function fitHorizon(nowMs: number): { fromMs: number; toMs: number } { | |
| 58 | + const year = new Date(nowMs).getUTCFullYear(); | |
| 59 | + return { fromMs: Date.UTC(year, 0, 1), toMs: Date.UTC(year + 2, 0, 1) }; | |
| 60 | +} | |
| 61 | + | |
| 62 | +/** | |
| 63 | + * Holds the currently deployable CounterModel of every metric. The API serves | |
| 64 | + * models, never per-tick values; SSE subscribers listen to the "model" event | |
| 65 | + * and only receive a payload when a NEW model is deployed. | |
| 66 | + */ | |
| 67 | +export class ModelStore extends EventEmitter { | |
| 68 | + private readonly modelMap = new Map<string, CounterModel>(); | |
| 69 | + private blockedList: BlockedMetric[] = []; | |
| 70 | + private readonly staleSinceMap = new Map<string, string>(); | |
| 71 | + private loadedRegistry: Registry | null = null; | |
| 72 | + private quakesSnapshot: QuakesSnapshot = { | |
| 73 | + count24h: 0, | |
| 74 | + lastMajor: null, | |
| 75 | + fetchedAt: new Date(0).toISOString(), | |
| 76 | + }; | |
| 77 | + | |
| 78 | + /** Load registry + fixtures and fit every model. Blocked metrics are served as stale. */ | |
| 79 | + boot(nowMs: number = Date.now()): void { | |
| 80 | + const registry = loadRegistry(); | |
| 81 | + const fixtures = loadFixtures(); | |
| 82 | + const { fromMs, toMs } = fitHorizon(nowMs); | |
| 83 | + const { models, blocked } = fitAllModels(registry.metrics, fixtures, { | |
| 84 | + validateFromMs: fromMs, | |
| 85 | + validateToMs: toMs, | |
| 86 | + }); | |
| 87 | + | |
| 88 | + this.loadedRegistry = registry; | |
| 89 | + this.modelMap.clear(); | |
| 90 | + for (const [id, model] of models) this.modelMap.set(id, model); | |
| 91 | + this.blockedList = blocked; | |
| 92 | + | |
| 93 | + // Loud by design: a blocked metric must never fail silently (stale badge instead). | |
| 94 | + for (const b of blocked) { | |
| 95 | + console.error(`[store] BLOCKED metric '${b.metricId}' — no model served (stale):`); | |
| 96 | + for (const issue of b.issues) console.error(`[store] ${issue.code}: ${issue.message}`); | |
| 97 | + } | |
| 98 | + | |
| 99 | + // Fixture-backed defaults for /v1/rt/quakes until the USGS poller first succeeds. | |
| 100 | + const quakesModel = models.get("earthquakes_24h"); | |
| 101 | + if (quakesModel) { | |
| 102 | + this.quakesSnapshot = { | |
| 103 | + count24h: quakesModel.anchorValue, | |
| 104 | + lastMajor: null, | |
| 105 | + fetchedAt: quakesModel.observedAt, | |
| 106 | + }; | |
| 107 | + } | |
| 108 | + } | |
| 109 | + | |
| 110 | + get registry(): Registry { | |
| 111 | + if (!this.loadedRegistry) throw new Error("ModelStore used before boot()"); | |
| 112 | + return this.loadedRegistry; | |
| 113 | + } | |
| 114 | + | |
| 115 | + get models(): ReadonlyMap<string, CounterModel> { | |
| 116 | + return this.modelMap; | |
| 117 | + } | |
| 118 | + | |
| 119 | + get blocked(): readonly BlockedMetric[] { | |
| 120 | + return this.blockedList; | |
| 121 | + } | |
| 122 | + | |
| 123 | + get quakes(): QuakesSnapshot { | |
| 124 | + return this.quakesSnapshot; | |
| 125 | + } | |
| 126 | + | |
| 127 | + getModel(metricId: string): CounterModel | undefined { | |
| 128 | + return this.modelMap.get(metricId); | |
| 129 | + } | |
| 130 | + | |
| 131 | + isStale(metricId: string): boolean { | |
| 132 | + return !this.modelMap.has(metricId) || this.staleSinceMap.has(metricId); | |
| 133 | + } | |
| 134 | + | |
| 135 | + staleSince(metricId: string): string | undefined { | |
| 136 | + return this.staleSinceMap.get(metricId); | |
| 137 | + } | |
| 138 | + | |
| 139 | + /** | |
| 140 | + * Swap the model of a true-RT (event) metric and notify SSE subscribers. | |
| 141 | + * Event-driven jumps are legitimate (no interpolation, no anti-teleportation | |
| 142 | + * diff); every other model family MUST go through fitAllModels + validation. | |
| 143 | + */ | |
| 144 | + updateRtModel(metricId: string, model: CounterModel): void { | |
| 145 | + const entry = this.registry.byId.get(metricId); | |
| 146 | + if (!entry) throw new Error(`updateRtModel: unknown metric '${metricId}'`); | |
| 147 | + if (entry.level !== "rt") { | |
| 148 | + throw new Error(`updateRtModel: metric '${metricId}' is not event-driven (level ${entry.level})`); | |
| 149 | + } | |
| 150 | + this.modelMap.set(metricId, model); | |
| 151 | + this.staleSinceMap.delete(metricId); | |
| 152 | + this.emit("model", model); | |
| 153 | + } | |
| 154 | + | |
| 155 | + /** Record when a metric's ingestion started failing (previous model keeps serving). */ | |
| 156 | + markStale(metricId: string, atIso: string = new Date().toISOString()): void { | |
| 157 | + if (!this.staleSinceMap.has(metricId)) this.staleSinceMap.set(metricId, atIso); | |
| 158 | + } | |
| 159 | + | |
| 160 | + setQuakesSnapshot(snapshot: QuakesSnapshot): void { | |
| 161 | + this.quakesSnapshot = snapshot; | |
| 162 | + } | |
| 163 | + | |
| 164 | + onModel(listener: ModelListener): void { | |
| 165 | + this.on("model", listener); | |
| 166 | + } | |
| 167 | + | |
| 168 | + offModel(listener: ModelListener): void { | |
| 169 | + this.off("model", listener); | |
| 170 | + } | |
| 171 | +} | |
added
apps/api/test/api.test.ts
+123 −0
@@ -0,0 +1,123 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/api/test/api.test.ts | |
| 6 | + * Purpose: API integration tests via app.inject() — health, registry, models, 404s, badge SVG (pollers disabled) | |
| 7 | + */ | |
| 8 | + | |
| 9 | +// Pollers read the env at buildApp() time, so setting it here (after hoisted | |
| 10 | +// imports evaluate) is safe; buildApp({ pollers: false }) is the second guard. | |
| 11 | +process.env.ENABLE_RT_POLLERS = "0"; | |
| 12 | + | |
| 13 | +import { afterAll, beforeAll, describe, expect, it } from "vitest"; | |
| 14 | +import { type CounterModel, counterValue } from "@earth-now/counter"; | |
| 15 | +import { buildApp } from "../src/server"; | |
| 16 | + | |
| 17 | +type App = Awaited<ReturnType<typeof buildApp>>; | |
| 18 | + | |
| 19 | +interface HealthBody { | |
| 20 | + status: string; | |
| 21 | + now: string; | |
| 22 | + modelCount: number; | |
| 23 | + blockedCount: number; | |
| 24 | + uptimeSeconds: number; | |
| 25 | +} | |
| 26 | + | |
| 27 | +interface MetricSummary { | |
| 28 | + id: string; | |
| 29 | + stale: boolean; | |
| 30 | + modelVersion?: string; | |
| 31 | + observedAt?: string; | |
| 32 | + sources: Array<{ id: string; license: string }>; | |
| 33 | +} | |
| 34 | + | |
| 35 | +let app: App; | |
| 36 | + | |
| 37 | +beforeAll(async () => { | |
| 38 | + app = await buildApp({ logger: false, pollers: false }); | |
| 39 | +}); | |
| 40 | + | |
| 41 | +afterAll(async () => { | |
| 42 | + await app.close(); | |
| 43 | +}); | |
| 44 | + | |
| 45 | +describe("GET /api/health", () => { | |
| 46 | + it("returns ok with model/blocked counts and uptime", async () => { | |
| 47 | + const res = await app.inject({ method: "GET", url: "/api/health" }); | |
| 48 | + expect(res.statusCode).toBe(200); | |
| 49 | + const body = JSON.parse(res.body) as HealthBody; | |
| 50 | + expect(body.status).toBe("ok"); | |
| 51 | + expect(Number.isNaN(Date.parse(body.now))).toBe(false); | |
| 52 | + expect(body.modelCount).toBeGreaterThan(0); | |
| 53 | + expect(body.blockedCount).toBeGreaterThanOrEqual(0); | |
| 54 | + expect(body.uptimeSeconds).toBeGreaterThanOrEqual(0); | |
| 55 | + }); | |
| 56 | +}); | |
| 57 | + | |
| 58 | +describe("GET /v1/metrics", () => { | |
| 59 | + it("returns the full registry with source attribution and stale flags", async () => { | |
| 60 | + const res = await app.inject({ method: "GET", url: "/v1/metrics" }); | |
| 61 | + expect(res.statusCode).toBe(200); | |
| 62 | + const metrics = JSON.parse(res.body) as MetricSummary[]; | |
| 63 | + expect(metrics.length).toBeGreaterThanOrEqual(30); | |
| 64 | + expect(metrics.map((m) => m.id)).toContain("world_population"); | |
| 65 | + for (const metric of metrics) { | |
| 66 | + expect(typeof metric.stale).toBe("boolean"); | |
| 67 | + expect(metric.sources.length).toBeGreaterThan(0); | |
| 68 | + for (const source of metric.sources) expect(source.license.length).toBeGreaterThan(0); | |
| 69 | + } | |
| 70 | + }); | |
| 71 | +}); | |
| 72 | + | |
| 73 | +describe("GET /v1/metrics/:id/model", () => { | |
| 74 | + it("serves a world_population model evaluating to a sane live value", async () => { | |
| 75 | + const res = await app.inject({ method: "GET", url: "/v1/metrics/world_population/model" }); | |
| 76 | + expect(res.statusCode).toBe(200); | |
| 77 | + const model = JSON.parse(res.body) as CounterModel; | |
| 78 | + expect(model.metricId).toBe("world_population"); | |
| 79 | + const value = counterValue(model, Date.now()); | |
| 80 | + expect(value).toBeGreaterThan(8.2e9); | |
| 81 | + expect(value).toBeLessThan(8.4e9); | |
| 82 | + }); | |
| 83 | + | |
| 84 | + it("returns 404 for an unknown metric id", async () => { | |
| 85 | + const res = await app.inject({ method: "GET", url: "/v1/metrics/nope_not_a_metric/model" }); | |
| 86 | + expect(res.statusCode).toBe(404); | |
| 87 | + }); | |
| 88 | +}); | |
| 89 | + | |
| 90 | +describe("GET /badge/:id.svg", () => { | |
| 91 | + it("renders an SVG badge for co2_ppm with CDN caching headers", async () => { | |
| 92 | + const res = await app.inject({ method: "GET", url: "/badge/co2_ppm.svg" }); | |
| 93 | + expect(res.statusCode).toBe(200); | |
| 94 | + expect(res.headers["content-type"]).toContain("image/svg+xml"); | |
| 95 | + expect(res.headers["cache-control"]).toBe("public, max-age=60"); | |
| 96 | + expect(res.body).toContain("<svg"); | |
| 97 | + expect(res.body).toContain("earth-now.co"); | |
| 98 | + }); | |
| 99 | + | |
| 100 | + it("returns 404 for an unknown metric badge", async () => { | |
| 101 | + const res = await app.inject({ method: "GET", url: "/badge/nope_not_a_metric.svg" }); | |
| 102 | + expect(res.statusCode).toBe(404); | |
| 103 | + }); | |
| 104 | +}); | |
| 105 | + | |
| 106 | +describe("GET /v1/models", () => { | |
| 107 | + it("serves every fitted model (registry count minus blocked)", async () => { | |
| 108 | + const [modelsRes, metricsRes, healthRes] = await Promise.all([ | |
| 109 | + app.inject({ method: "GET", url: "/v1/models" }), | |
| 110 | + app.inject({ method: "GET", url: "/v1/metrics" }), | |
| 111 | + app.inject({ method: "GET", url: "/api/health" }), | |
| 112 | + ]); | |
| 113 | + expect(modelsRes.statusCode).toBe(200); | |
| 114 | + const { models, generatedAt } = JSON.parse(modelsRes.body) as { | |
| 115 | + models: Record<string, CounterModel>; | |
| 116 | + generatedAt: string; | |
| 117 | + }; | |
| 118 | + const metrics = JSON.parse(metricsRes.body) as MetricSummary[]; | |
| 119 | + const health = JSON.parse(healthRes.body) as HealthBody; | |
| 120 | + expect(Number.isNaN(Date.parse(generatedAt))).toBe(false); | |
| 121 | + expect(Object.keys(models).length).toBe(metrics.length - health.blockedCount); | |
| 122 | + }); | |
| 123 | +}); | |
added
apps/api/tsconfig.json
+10 −0
@@ -0,0 +1,10 @@ | ||
| 1 | +{ | |
| 2 | + "extends": "../../tsconfig.base.json", | |
| 3 | + "compilerOptions": { | |
| 4 | + "outDir": "dist", | |
| 5 | + "rootDir": "src", | |
| 6 | + "module": "NodeNext", | |
| 7 | + "moduleResolution": "NodeNext" | |
| 8 | + }, | |
| 9 | + "include": ["src"] | |
| 10 | +} | |
added
apps/ingest/package.json
+29 −0
@@ -0,0 +1,29 @@ | ||
| 1 | +{ | |
| 2 | + "name": "@earth-now/ingest", | |
| 3 | + "version": "0.1.0", | |
| 4 | + "private": true, | |
| 5 | + "type": "module", | |
| 6 | + "description": "Ingestion workers — source fetchers/parsers, raw archiving, cron + BullMQ scheduling", | |
| 7 | + "author": "Simon-Pierre Boucher <contact@spboucher.ai>", | |
| 8 | + "scripts": { | |
| 9 | + "dev": "tsx watch src/index.ts", | |
| 10 | + "build": "tsc -p tsconfig.json", | |
| 11 | + "test": "vitest run", | |
| 12 | + "typecheck": "tsc -p tsconfig.json --noEmit", | |
| 13 | + "lint": "echo 'lint: covered by root header check'", | |
| 14 | + "run:source": "tsx src/run-source.ts" | |
| 15 | + }, | |
| 16 | + "dependencies": { | |
| 17 | + "@earth-now/counter": "workspace:*", | |
| 18 | + "@earth-now/models": "workspace:*", | |
| 19 | + "@earth-now/registry": "workspace:*", | |
| 20 | + "bullmq": "^5.12.0", | |
| 21 | + "ioredis": "^5.4.1" | |
| 22 | + }, | |
| 23 | + "devDependencies": { | |
| 24 | + "@types/node": "^20", | |
| 25 | + "tsx": "^4.19.0", | |
| 26 | + "typescript": "^5.5.4", | |
| 27 | + "vitest": "^2.0.5" | |
| 28 | + } | |
| 29 | +} | |
added
apps/ingest/src/archive.ts
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/ingest/src/archive.ts | |
| 6 | + * Purpose: Raw payload archive — every ingested byte is kept (with sha256) for future re-fitting; S3-shaped interface, local FS for now | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { createHash } from "node:crypto"; | |
| 10 | +import { mkdir, writeFile } from "node:fs/promises"; | |
| 11 | +import { join } from "node:path"; | |
| 12 | + | |
| 13 | +export interface ArchiveOptions { | |
| 14 | + /** Archive root. Defaults to $RAW_ARCHIVE_DIR, then "data/raw". */ | |
| 15 | + dir?: string; | |
| 16 | + /** Timestamp override for deterministic tests. */ | |
| 17 | + now?: Date; | |
| 18 | +} | |
| 19 | + | |
| 20 | +export interface ArchiveResult { | |
| 21 | + rawPath: string; | |
| 22 | + checksumPath: string; | |
| 23 | + sha256: string; | |
| 24 | +} | |
| 25 | + | |
| 26 | +/** | |
| 27 | + * Write `${dir}/${sourceId}/${timestamp}.raw` plus a sibling `.sha256` checksum file. | |
| 28 | + * We keep ALL raw data for re-fitting; when this moves to S3 only this module changes. | |
| 29 | + */ | |
| 30 | +export async function archiveRaw( | |
| 31 | + sourceId: string, | |
| 32 | + payload: string | Uint8Array, | |
| 33 | + options: ArchiveOptions = {}, | |
| 34 | +): Promise<ArchiveResult> { | |
| 35 | + if (!/^[a-z0-9_-]+$/i.test(sourceId)) | |
| 36 | + throw new Error(`archiveRaw: invalid sourceId '${sourceId}' (path-safe [a-z0-9_-] only)`); | |
| 37 | + const dir = options.dir ?? process.env.RAW_ARCHIVE_DIR ?? "data/raw"; | |
| 38 | + const now = options.now ?? new Date(); | |
| 39 | + // ISO timestamp made filename-safe: 2026-08-09T12:00:00.000Z → 2026-08-09T12-00-00-000Z | |
| 40 | + const stamp = now.toISOString().replace(/[:.]/g, "-"); | |
| 41 | + const targetDir = join(dir, sourceId); | |
| 42 | + await mkdir(targetDir, { recursive: true }); | |
| 43 | + | |
| 44 | + const data = typeof payload === "string" ? Buffer.from(payload, "utf8") : Buffer.from(payload); | |
| 45 | + const sha256 = createHash("sha256").update(data).digest("hex"); | |
| 46 | + const fileName = `${stamp}.raw`; | |
| 47 | + const rawPath = join(targetDir, fileName); | |
| 48 | + const checksumPath = `${rawPath}.sha256`; | |
| 49 | + | |
| 50 | + await writeFile(rawPath, data); | |
| 51 | + await writeFile(checksumPath, `${sha256} ${fileName}\n`, "utf8"); | |
| 52 | + return { rawPath, checksumPath, sha256 }; | |
| 53 | +} | |
added
apps/ingest/src/fetch-like.ts
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/ingest/src/fetch-like.ts | |
| 6 | + * Purpose: Minimal structural fetch interface so tests inject fixture-backed fakes and never hit the network | |
| 7 | + */ | |
| 8 | + | |
| 9 | +export interface FetchResponseLike { | |
| 10 | + ok: boolean; | |
| 11 | + status: number; | |
| 12 | + text(): Promise<string>; | |
| 13 | +} | |
| 14 | + | |
| 15 | +export type FetchLike = (url: string) => Promise<FetchResponseLike>; | |
| 16 | + | |
| 17 | +/** Default implementation: the global fetch (Node 20+). Response satisfies FetchResponseLike structurally. */ | |
| 18 | +export const defaultFetch: FetchLike = (url) => fetch(url); | |
added
apps/ingest/src/index.ts
+114 −0
@@ -0,0 +1,114 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/ingest/src/index.ts | |
| 6 | + * Purpose: Ingestion orchestrator — BullMQ repeatable jobs when REDIS_URL is set, plain setInterval scheduler otherwise; never crashes on a failing source | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { SOURCES, runSource } from "./runners.js"; | |
| 10 | + | |
| 11 | +const log = { | |
| 12 | + info: (msg: string): void => console.log(`${new Date().toISOString()} [ingest] ${msg}`), | |
| 13 | + warn: (msg: string): void => console.warn(`${new Date().toISOString()} [ingest] WARN ${msg}`), | |
| 14 | + error: (msg: string): void => console.error(`${new Date().toISOString()} [ingest] ERROR ${msg}`), | |
| 15 | +}; | |
| 16 | + | |
| 17 | +interface SourceState { | |
| 18 | + lastSuccessMs: number | null; | |
| 19 | + failureCount: number; | |
| 20 | + startedMs: number; | |
| 21 | +} | |
| 22 | + | |
| 23 | +const states = new Map<string, SourceState>( | |
| 24 | + SOURCES.map((s) => [s.id, { lastSuccessMs: null, failureCount: 0, startedMs: Date.now() }]), | |
| 25 | +); | |
| 26 | + | |
| 27 | +/** | |
| 28 | + * Production guardrail: a source failing for longer than 2× its cadence must raise the | |
| 29 | + * "stale" badge + alert. Stub for now — wired to the API alerting later. | |
| 30 | + */ | |
| 31 | +function checkStale(sourceId: string, cadenceMs: number): void { | |
| 32 | + const state = states.get(sourceId); | |
| 33 | + if (!state) return; | |
| 34 | + const reference = state.lastSuccessMs ?? state.startedMs; | |
| 35 | + if (Date.now() - reference > 2 * cadenceMs) { | |
| 36 | + log.warn( | |
| 37 | + `[stale] ${sourceId} has had no successful ingestion for > 2× its cadence ` + | |
| 38 | + `(${state.failureCount} consecutive failure(s)) — would trigger the stale badge + alert (stub)`, | |
| 39 | + ); | |
| 40 | + } | |
| 41 | +} | |
| 42 | + | |
| 43 | +/** Run one source, absorbing every failure: log + count, never crash the scheduler. */ | |
| 44 | +async function tick(sourceId: string, cadenceMs: number): Promise<void> { | |
| 45 | + const state = states.get(sourceId); | |
| 46 | + try { | |
| 47 | + const summary = await runSource(sourceId); | |
| 48 | + if (state) { | |
| 49 | + state.lastSuccessMs = Date.now(); | |
| 50 | + state.failureCount = 0; | |
| 51 | + } | |
| 52 | + log.info( | |
| 53 | + `${sourceId}: ok — ${summary.observationCount} observation(s), ` + | |
| 54 | + `last=${summary.last ? `${summary.last.time} → ${summary.last.value}` : "n/a"}, raw=${summary.rawPath}`, | |
| 55 | + ); | |
| 56 | + } catch (err) { | |
| 57 | + if (state) state.failureCount += 1; | |
| 58 | + log.error(`${sourceId}: run failed — ${err instanceof Error ? err.message : String(err)}`); | |
| 59 | + checkStale(sourceId, cadenceMs); | |
| 60 | + } | |
| 61 | +} | |
| 62 | + | |
| 63 | +async function startWithRedis(redisUrl: string): Promise<void> { | |
| 64 | + // bullmq/ioredis are imported ONLY when REDIS_URL is set — dev stays dependency-light. | |
| 65 | + const { Queue, Worker } = await import("bullmq"); | |
| 66 | + const { Redis } = await import("ioredis"); | |
| 67 | + const connection = new Redis(redisUrl, { maxRetriesPerRequest: null }); | |
| 68 | + | |
| 69 | + const queue = new Queue("ingest", { connection }); | |
| 70 | + for (const source of SOURCES) { | |
| 71 | + await queue.add( | |
| 72 | + source.id, | |
| 73 | + {}, | |
| 74 | + { | |
| 75 | + repeat: { every: source.cadenceMs }, | |
| 76 | + jobId: `repeat:${source.id}`, | |
| 77 | + removeOnComplete: 100, | |
| 78 | + removeOnFail: 500, | |
| 79 | + }, | |
| 80 | + ); | |
| 81 | + log.info(`scheduled '${source.id}' every ${source.cadenceMs / 1000}s (BullMQ repeatable)`); | |
| 82 | + } | |
| 83 | + | |
| 84 | + const worker = new Worker( | |
| 85 | + "ingest", | |
| 86 | + async (job) => { | |
| 87 | + const source = SOURCES.find((s) => s.id === job.name); | |
| 88 | + await tick(job.name, source?.cadenceMs ?? 0); | |
| 89 | + }, | |
| 90 | + { connection }, | |
| 91 | + ); | |
| 92 | + worker.on("error", (err) => log.error(`worker error: ${err.message}`)); | |
| 93 | + log.info(`BullMQ mode: queue "ingest" on ${redisUrl}`); | |
| 94 | +} | |
| 95 | + | |
| 96 | +function startRedisLess(): void { | |
| 97 | + log.info("REDIS_URL not set — Redis-less dev mode: plain setInterval scheduler"); | |
| 98 | + for (const source of SOURCES) { | |
| 99 | + void tick(source.id, source.cadenceMs); | |
| 100 | + setInterval(() => void tick(source.id, source.cadenceMs), source.cadenceMs); | |
| 101 | + log.info(`scheduled '${source.id}' every ${source.cadenceMs / 1000}s (setInterval)`); | |
| 102 | + } | |
| 103 | +} | |
| 104 | + | |
| 105 | +async function main(): Promise<void> { | |
| 106 | + const redisUrl = process.env.REDIS_URL; | |
| 107 | + if (redisUrl) await startWithRedis(redisUrl); | |
| 108 | + else startRedisLess(); | |
| 109 | +} | |
| 110 | + | |
| 111 | +main().catch((err: unknown) => { | |
| 112 | + log.error(`fatal startup error: ${err instanceof Error ? err.message : String(err)}`); | |
| 113 | + process.exit(1); | |
| 114 | +}); | |
added
apps/ingest/src/run-source.ts
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/ingest/src/run-source.ts | |
| 6 | + * Purpose: CLI for `pnpm ingest:run <sourceId>` — force one ingestion locally and print a summary | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { KNOWN_SOURCE_IDS, runSource } from "./runners.js"; | |
| 10 | + | |
| 11 | +const sourceId = process.argv[2]; | |
| 12 | + | |
| 13 | +if (!sourceId || !KNOWN_SOURCE_IDS.includes(sourceId)) { | |
| 14 | + console.error( | |
| 15 | + sourceId ? `Unknown source '${sourceId}'.` : "Usage: pnpm ingest:run <sourceId>", | |
| 16 | + ); | |
| 17 | + console.error("Known sources:"); | |
| 18 | + for (const id of KNOWN_SOURCE_IDS) console.error(` - ${id}`); | |
| 19 | + process.exit(1); | |
| 20 | +} | |
| 21 | + | |
| 22 | +runSource(sourceId) | |
| 23 | + .then((summary) => { | |
| 24 | + console.log(`✓ ${summary.sourceId}: ${summary.observationCount} observation(s)`); | |
| 25 | + if (summary.first) console.log(` first: ${summary.first.time} → ${summary.first.value}`); | |
| 26 | + if (summary.last) console.log(` last: ${summary.last.time} → ${summary.last.value}`); | |
| 27 | + console.log(` raw: ${summary.rawPath} (sha256 ${summary.sha256.slice(0, 12)}…)`); | |
| 28 | + }) | |
| 29 | + .catch((err: unknown) => { | |
| 30 | + console.error(`✗ ${sourceId} ingestion failed:`, err instanceof Error ? err.message : err); | |
| 31 | + process.exit(1); | |
| 32 | + }); | |
added
apps/ingest/src/runners.ts
+94 −0
@@ -0,0 +1,94 @@ | ||
| 1 | +/** | |
| 2 | + * earth-now.co | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * File: apps/ingest/src/runners.ts | |
| 6 | + * Purpose: Per-source run functions (fetch → archive raw → parse → summary) shared by the CLI and the queue worker | |
| 7 | + */ | |
| 8 | + | |
| 9 | +import { archiveRaw } from "./archive.js"; | |
| 10 | +import { type FetchLike, defaultFetch } from "./fetch-like.js"; | |
| 11 | +import { NOAA_CO2_URL, parseNoaaMonthlyCo2 } from "./sources/noaa-co2.js"; | |
| 12 | +import { parseUsgsCount, usgsCountUrl } from "./sources/usgs.js"; | |
| 13 | + | |
| 14 | +/** USGS live counter parameters: quakes M ≥ 4.5 over the trailing 24 h. */ | |
| 15 | +export const USGS_WINDOW_HOURS = 24; | |
| 16 | +// Must match the registry definition of earthquakes_24h (catalog: M ≥ 2.5). | |
| 17 | +export const USGS_MIN_MAGNITUDE = 2.5; | |
| 18 | + | |
| 19 | +export interface ObservationPoint { | |
| 20 | + time: string; | |
| 21 | + value: number; | |
| 22 | +} | |
| 23 | + | |
| 24 | +export interface RunSummary { | |
| 25 | + sourceId: string; | |
| 26 | + observationCount: number; | |
| 27 | + first: ObservationPoint | null; | |
| 28 | + last: ObservationPoint | null; | |
| 29 | + rawPath: string; | |
| 30 | + sha256: string; | |
| 31 | +} | |
| 32 | + | |
| 33 | +async function fetchText(url: string, fetchImpl: FetchLike): Promise<string> { | |
| 34 | + const res = await fetchImpl(url); | |
| 35 | + if (!res.ok) throw new Error(`fetch failed: HTTP ${res.status} for ${url}`); | |
| 36 | + return res.text(); | |
| 37 | +} | |
| 38 | + | |
| 39 | +/** USGS FDSN: one observation — the M≥4.5 quake count over the trailing 24 h window. */ | |
| 40 | +export async function runUsgsFdsn(fetchImpl: FetchLike = defaultFetch): Promise<RunSummary> { | |
| 41 | + const url = usgsCountUrl(USGS_WINDOW_HOURS, USGS_MIN_MAGNITUDE); | |
| 42 | + const text = await fetchText(url, fetchImpl); | |
| 43 | + const archived = await archiveRaw("usgs_fdsn", text); | |
| 44 | + const count = parseUsgsCount(JSON.parse(text)); | |
| 45 | + const point: ObservationPoint = { time: new Date().toISOString(), value: count }; | |
| 46 | + return { | |
| 47 | + sourceId: "usgs_fdsn", | |
| 48 | + observationCount: 1, | |
| 49 | + first: point, | |
| 50 | + last: point, | |
| 51 | + rawPath: archived.rawPath, | |
| 52 | + sha256: archived.sha256, | |
| 53 | + }; | |
| 54 | +} | |
| 55 | + | |
| 56 | +/** NOAA GML Mauna Loa: full monthly CO₂ series (Keeling curve). */ | |
| 57 | +export async function runNoaaGmlMlo(fetchImpl: FetchLike = defaultFetch): Promise<RunSummary> { | |
| 58 | + const text = await fetchText(NOAA_CO2_URL, fetchImpl); | |
| 59 | + const archived = await archiveRaw("noaa_gml_mlo", text); | |
| 60 | + const observations = parseNoaaMonthlyCo2(text); | |
| 61 | + return { | |
| 62 | + sourceId: "noaa_gml_mlo", | |
| 63 | + observationCount: observations.length, | |
| 64 | + first: observations[0] ?? null, | |
| 65 | + last: observations[observations.length - 1] ?? null, | |
| 66 | + rawPath: archived.rawPath, | |
| 67 | + sha256: archived.sha256, | |
| 68 | + }; | |
| 69 | +} | |
| 70 | + | |
| 71 | +export interface SourceSpec { | |
| 72 | + id: string; | |
| 73 | + /** Scheduling cadence in ms (usgs 5 min, noaa daily). */ | |
| 74 | + cadenceMs: number; | |
| 75 | + run: (fetchImpl?: FetchLike) => Promise<RunSummary>; | |
| 76 | +} | |
| 77 | + | |
| 78 | +/** Every source the ingest service knows how to run, with its scheduling cadence. */ | |
| 79 | +export const SOURCES: readonly SourceSpec[] = [ | |
| 80 | + { id: "usgs_fdsn", cadenceMs: 5 * 60_000, run: runUsgsFdsn }, | |
| 81 | + { id: "noaa_gml_mlo", cadenceMs: 24 * 3_600_000, run: runNoaaGmlMlo }, | |
| 82 | +]; | |
| 83 | + | |
| 84 | +export const KNOWN_SOURCE_IDS: readonly string[] = SOURCES.map((s) => s.id); | |
| 85 | + | |
| 86 | +/** Run one source by id. Throws on unknown id or on any fetch/parse failure. */ | |
| 87 | +export async function runSource( | |
| 88 | + sourceId: string, | |
| 89 | + fetchImpl: FetchLike = defaultFetch, | |
| 90 | +): Promise<RunSummary> { | |
| 91 | + const spec = SOURCES.find((s) => s.id === sourceId); | |
| 92 | + if (!spec) throw new Error(`Unknown source '${sourceId}'. Known: ${KNOWN_SOURCE_IDS.join(", ")}`); | |
| 93 | + return spec.run(fetchImpl); | |
| 94 | +} | |
added
apps/ingest/src/sources/noaa-co2.ts
+71 −0
Large diff (71 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/src/sources/usgs.ts
+89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/archive.test.ts
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/co2_mm_mlo_sample.txt
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/co2_mm_mlo_truncated.txt
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/usgs_count_malformed.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/usgs_count_sample.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/usgs_query_malformed.json
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/fixtures/usgs_query_sample.json
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/noaa-co2.test.ts
+75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/runners.test.ts
+74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/test/usgs.test.ts
+99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added
apps/ingest/tsconfig.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/embed/[id]/page.tsx
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/globals.css
+92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/layout.tsx
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/methodology/page.tsx
+136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/metric/[id]/page.tsx
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/page.tsx
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/CountryRanking.tsx
+112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/Dashboard.tsx
+418 −0
Large diff (418 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/EmbedCounter.tsx
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/FitValue.tsx
+69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/LiveCounter.tsx
+327 −0
Large diff (327 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/MetricDetail.tsx
+175 −0
Large diff (175 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/YearProgress.tsx
+50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/api.ts
+129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/derived.ts
+101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/i18n.tsx
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/messages.ts
+41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/smoothing.ts
+57 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/tiers.ts
+69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/messages/en.json
+87 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/messages/fr.json
+87 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/next-env.d.ts
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/next.config.mjs
+15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/package.json
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/postcss.config.mjs
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/tailwind.config.ts
+51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/consistency-shared.ts
+83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/consistency.jsdom.test.ts
+55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/consistency.node.test.ts
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/derived.test.ts
+202 −0
Large diff (202 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/smoothing.test.ts
+95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/test/tiers.test.ts
+97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/tsconfig.json
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/tsconfig.tsbuildinfo
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/vitest.config.ts
+23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added
earth-now-metrics-catalog.md
+234 −0
Large diff (234 lines) is not loaded by default to keep the page fast.Load diff
added
infra/deploy-m3u96b-docker.sh
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
infra/deploy-m3u96b.sh
+67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker-compose.prod.yml
+93 −0
Large diff (93 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker/api.Dockerfile
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker/ingest.Dockerfile
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker/web.Dockerfile
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
infra/migrations/001_initial.sql
+89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added
infra/nginx.conf
+68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added
infra/ngrok.yml
+27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added
infra/proxy.mjs
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
infra/tunnel-status.sh
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
package.json
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/package.json
+29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/counter-model.ts
+96 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/format.ts
+97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/index.ts
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/pchip.ts
+122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/value.ts
+129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/src/windows.ts
+49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/test/consistency.test.ts
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/test/format.test.ts
+85 −0
Large diff (85 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/test/property.test.ts
+141 −0
Large diff (141 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/test/value.test.ts
+183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added
packages/counter/tsconfig.json
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/package.json
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/builders.ts
+255 −0
Large diff (255 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/fourier.ts
+111 −0
Large diff (111 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/holt-winters.ts
+74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/index.ts
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/kalman.ts
+59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/linalg.ts
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/src/validate.ts
+127 −0
Large diff (127 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/test/builders-validate.test.ts
+183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/test/filters.test.ts
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/test/fourier.test.ts
+81 −0
Large diff (81 lines) is not loaded by default to keep the page fast.Load diff
added
packages/models/tsconfig.json
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/climate.json
+66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/continents.json
+92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/economy.json
+55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/emissions.json
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/energy.json
+100 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/forest.json
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/health.json
+50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/ocean.json
+67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/population-countries.json
+152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/population.json
+38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/realtime.json
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/society.json
+188 −0
Large diff (189 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/space.json
+16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/fixtures/tech.json
+54 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/climate.yaml
+120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/continents.yaml
+50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/economy.yaml
+231 −0
Large diff (231 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/emissions.yaml
+79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/energy.yaml
+132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/forest.yaml
+82 −0
Large diff (82 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/health.yaml
+112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/ocean.yaml
+57 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/population-countries.yaml
+66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/population.yaml
+207 −0
Large diff (207 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/realtime.yaml
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/society.yaml
+237 −0
Large diff (237 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/space.yaml
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/metrics/tech.yaml
+163 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/package.json
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/src/fit.ts
+164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/src/index.ts
+22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/src/load.ts
+79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/src/schema.ts
+136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/test/registry.test.ts
+131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added
packages/registry/tsconfig.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/package.json
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/scripts/check-size.mjs
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/src/render.ts
+103 −0
Large diff (103 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/src/widget.ts
+168 −0
Large diff (168 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/test/widget.test.ts
+138 −0
Large diff (138 lines) is not loaded by default to keep the page fast.Load diff
added
packages/widget/tsconfig.json
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
pnpm-lock.yaml
+3939 −0
Large diff (3,939 lines) is not loaded by default to keep the page fast.Load diff
added
pnpm-workspace.yaml
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/check-headers.ignore
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/check-headers.ts
+116 −0
Large diff (116 lines) is not loaded by default to keep the page fast.Load diff
added
tsconfig.base.json
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
turbo.json
+24 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff