spb/earth-now Public License

1# CLAUDE.md — earth-now.co23## Project Overview45**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. The site displays live world metrics (population, births, deaths, CO₂ emissions, temperature anomaly, forest loss, energy production, internet users, etc.) that tick continuously on screen.67The fundamental difference from Worldometer: our counters are not opaque linear extrapolations. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals.89Product pillars:101. **Live everywhere**: dashboard, shareable public pages (`/m/:token`), embeddable widgets (JS, iframe, SVG badge).112. **Statistical honesty**: every number is traceable (source, observation date, interpolation method, uncertainty).123. **Robustness**: resilient ingestion pipeline; never a frozen or absurd counter in production.1314---1516## Architecture Principle #1: A Counter Is a Model, Not a Stream1718**NEVER push one value per second over the network.** The server ships a **CounterModel**; the client animates locally.1920```typescript21// The project's central contract — types/counter-model.ts22interface CounterModel {23  metricId: string;            // e.g. "co2_emissions_ytd"24  anchorValue: number;         // value at the anchor point25  anchorTime: string;          // ISO 8601 UTC of the anchor point26  rateFn: RateFunction;        // how the value evolves from the anchor27  uncertainty?: { low: number; high: number };  // 90% CI at time T28  observedAt: string;          // date of the last REAL observation29  sourceId: string;            // key in the source registry30  modelVersion: string;        // e.g. "seasonal-spline-v2"31  displayHints: { decimals: number; sigFigs?: number; unit: string };32}3334type RateFunction =35  | { kind: "linear"; perSecond: number }36  | { kind: "piecewise"; segments: Array<{ from: string; perSecond: number }> }37  | { kind: "seasonal"; base: number; harmonics: Harmonic[] }   // Fourier38  | { kind: "spline"; knots: Array<[time: string, value: number]> };39```4041The client computes `value(t)` at 60 fps with `requestAnimationFrame` from the model. The server only pushes (via SSE) a **new model** when: new data is ingested, a new forecast arrives, or a correction is made. Result: zero per-tick traffic, perfect consistency across all clients (same model ⇒ same value at the same instant t), and the SVG badge uses the exact same function server-side.4243---4445## Statistical Methodology (the heart of the product)4647Every metric declares its model level in the registry. Escalation by level:4849**Level 0 — Linear**: `rate = (forecast_annual_value - last_obs) / remaining_seconds`. Acceptable only as a fallback or for genuinely quasi-linear metrics.5051**Level 1 — Seasonal**: trend + seasonality decomposition (Fourier harmonics or STL). Mandatory for: births/deaths (demographic seasonality), CO₂ concentration (Keeling curve: ±3 ppm annual cycle), energy consumption, fires/deforestation.5253**Level 2 — Observation/forecast fusion**: when a source publishes both delayed observations and projections (e.g. UN WPP, Global Carbon Budget, NOAA), we blend:54- Interpolation via **monotone spline (PCHIP)** between observed points — never a natural cubic spline (possible overshoot ⇒ counters absurdly going backwards).55- Observation→forecast junction via a **Kalman filter** or exponential weighting: near the last observation we track the observed data; further out we converge toward the forecast.56- **Holt-Winters** internally for nowcasting frequently-published metrics (weekly/monthly) when the source provides no forecast.5758**Uncertainty**: propagate source CIs (e.g. UN low/median/high variants) into `uncertainty`. The UI may render a band. Absolute rule: `displayHints.sigFigs` caps significant digits — we NEVER display more precision than the model justifies (world population to the unit is a lie; we animate the unit digit but the tooltip shows the CI).5960**Validity constraints** (enforced by `packages/models/validate.ts`):61- Cumulative counters (YTD births, YTD CO₂ emitted): `rateFn` strictly ≥ 0, reset on Jan 1 UTC via `piecewise`.62- Stocks (population, CO₂ ppm): may move both ways, but |derivative| bounded by a max declared in the registry.63- Every re-fitted model is diffed against the previous one: a jump > declared threshold ⇒ model deployment is blocked and an alert fires (no counter "teleportation" in prod; we catch up with a 60 s client-side smoothing transition).6465---6667## Metric & Source Registry6869Everything goes through `packages/registry/metrics/*.yaml`. A metric does not exist until it is declared there. Example:7071```yaml72id: world_population73name: { fr: "Population mondiale", en: "World population" }74unit: people75kind: stock76model: seasonal-spline-v2        # level 277sources:78  - id: un_wpp_202479    url: https://population.un.org/wpp/80    license: CC BY 3.0 IGO81    cadence: yearly82    variants: [low, median, high]83refresh: on_source_update84display: { decimals: 0, sigFigs: 7 }85constraints: { maxAbsRatePerSec: 10 }86```8788Priority sources: UN WPP (demographics), Global Carbon Project + NOAA GML (CO₂), Copernicus/ERA5 (temperature), FAO & Global Forest Watch (forests), Ember/IEA (energy), USGS (earthquakes — special case: true event-driven real time, no interpolation), OWID as a secondary aggregator. **Always check and honor each source's license; attribution is displayed in the UI and in API responses.**8990---9192## Architecture9394```95apps/96  web/          # Next.js 14 (App Router) — dashboard, /m/:token pages, /embed97  api/          # Fastify (TS) — REST + SSE98  ingest/       # Ingestion workers (cron + BullMQ queue)99packages/100  registry/     # Metric + source YAML (product source of truth)101  models/       # Interpolators, fitting, validation (pure, no I/O)102  counter/      # Shared client runtime: value(model, t) — used by web, widget AND badge103  widget/       # Embeddable widget.js (IIFE build < 15 KB gzip, zero dependencies)104infra/          # Docker, Terraform, migrations105```106107Data flow:1081. `ingest` downloads/parses sources on their cadence (exponential retry, checksums, raw archive to S3 — we keep ALL raw data for re-fitting).1092. Normalization → `observations` (append-only, PostgreSQL + TimescaleDB).1103. `models` re-fits the CounterModel of each impacted metric → validation → versioned in DB + Redis cache.1114. Redis pub/sub publish on `metric:{id}` → SSE servers relay the new model to connected clients.1125. SVG/PNG badges: rendered server-side with the same `packages/counter`, CDN cache 60 s.113114Database (key tables): `observations` (append-only), `counter_models` (versioned, never UPDATEd), `metrics_registry_sync`, `share_tokens` (revocable, per-metric visibility), `ingest_runs` (full auditability of every run).115116---117118## Stack119120- **Strict TypeScript everywhere** (`"strict": true`, no unjustified `any`). Node 20+, pnpm workspaces, Turborepo.121- Next.js 14 + Tailwind (web); Fastify (api); BullMQ + Redis (ingest); PostgreSQL 16 + TimescaleDB; Drizzle ORM + SQL migrations.122- Statistical fitting: TS implementations in `packages/models` (PCHIP, least-squares Fourier fit, Holt-Winters, 1D Kalman). If a heavy fit justifies it, an isolated Python service (scipy/statsmodels) under `apps/ingest/fitters-py` called by the orchestrator — but prefer TS whenever possible to share code with the client.123- Real time: **SSE** (EventSource) for model distribution; no WebSocket unless a future bidirectional need arises. Reconnection: `Last-Event-ID` + Redis Streams (15 min buffer).124- Cloudflare CDN in front of `/badge/*` and `widget.js`.125126## Mandatory File Header127128**Every source code file in this repository (`.ts`, `.tsx`, `.js`, `.py`, `.sql`, `.sh`, config files where comments are allowed) MUST start with the following author header:**129130```typescript131/**132 * earth-now.co133 * Author:  Simon-Pierre Boucher134 * Contact: contact@spboucher.ai135 * File:    <relative/path/to/file.ts>136 * Purpose: <one-line description of what this file does>137 */138```139140Adapt the comment syntax to the language (`#` for Python/YAML/shell, `--` for SQL). Rules:141- The `File:` line must match the file's actual path in the repo — update it if the file is moved/renamed.142- `Purpose:` is one line, kept accurate when the file's role changes.143- CI lint rule (`scripts/check-headers.ts`, run in `pnpm lint`) fails any file missing or with a malformed header. When creating a new file, Claude must add this header first, before any code.144- Generated files (build output, lockfiles, migrations auto-generated by drizzle-kit) are exempt and listed in `scripts/check-headers.ignore`.145146## Deployment147148- Production runs on **node `m3u96b`**, exposed publicly through **ngrok** at **www.earth-now.co**.149- Stack on `m3u96b`: Docker Compose (`infra/docker-compose.prod.yml`) running web, api, ingest, PostgreSQL/Timescale, and Redis; ngrok tunnel (reserved domain `www.earth-now.co`) fronting the web/api reverse proxy on port 8080.150- Deploy flow:151  ```bash152  pnpm build                      # turbo build of all apps153  pnpm deploy:m3u96b              # rsync + docker compose up -d on node m3u96b154  pnpm tunnel:status              # verify the ngrok tunnel is up and serving www.earth-now.co155  ```156- ngrok config lives in `infra/ngrok.yml` (reserved domain + edge). The ngrok authtoken is provided via environment/secret manager — **never commit it**.157- Health check after every deploy: `GET https://www.earth-now.co/api/health` must return 200 and `GET /sse/health` must hold an SSE connection open ≥ 10 s.158- Note: ngrok is the current exposure layer; keep the reverse-proxy config portable so we can swap to Cloudflare Tunnel or a plain LB later without app changes.159160## Commands161162```bash163pnpm dev              # whole monorepo (turbo)164pnpm dev:web          # frontend only165pnpm test             # vitest, all packages166pnpm test:models      # interpolator tests (the most critical ones)167pnpm lint && pnpm typecheck   # lint includes the file-header check168pnpm db:migrate       # drizzle-kit169pnpm ingest:run <sourceId>    # force an ingestion locally170pnpm models:refit <metricId>  # manual re-fit + diff report171pnpm deploy:m3u96b    # deploy to production node m3u96b (ngrok → www.earth-now.co)172```173174## Code Conventions175176- Time: **UTC everywhere internally**, ISO 8601; convert to local time only at display. No hand-rolled date arithmetic — use `date-fns` or `Temporal`.177- Units: SI in the data layer; display conversions (t → Gt, etc.) live in `displayHints`, never in the pipeline.178- `packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit system clock (time is always a parameter). This is what makes them testable and shareable client/server.179- Numbers: never floats for ingested cumulative values (use integers in the base unit when possible); formatting via `Intl.NumberFormat`.180- Every new metric = a PR containing: registry YAML + data fixture + golden model test + source license review.181- i18n: fr + en from day one (keys in the registry and `apps/web/messages`).182183## Testing — Non-negotiable184185- **Golden tests for interpolators**: for each `modelVersion`, fixtures (input data → expected values at precise instants t, defined tolerance). Any output change without a version bump = CI failure.186- Property-based tests (fast-check): monotonicity of cumulatives, derivative bounds, continuity at piecewise junctions, `value(anchorTime) === anchorValue`.187- Client/server consistency test: `packages/counter` run in Node and in jsdom must produce bit-identical values for the same model.188- Ingestion: every source parser has fixtures of the real format (including degraded cases: truncated file, renamed columns).189190## Production Guardrails191192- A counter must **never**: go backwards if cumulative, visibly jump (> 60 s smoothing), display NaN/undefined (fallback = last frozen value + "data pending" indicator), or show precision > sigFigs.193- Ingestion failing > 2× the source's cadence ⇒ "stale" badge on the metric + alert (no silent failure).194- Public endpoints (`/m`, `/embed`, `/badge`, `/sse`): rate limiting per IP + token, read-only open CORS, revocable share tokens, no user data exposed.195- Public /methodology page: generated from the registry — every metric exposes its source, cadence, model, and last observation there. This is a product commitment, not an option.196197## What Claude Must Do in This Repo1981991. Before adding/modifying a metric: read the registry YAML and its associated golden tests.2002. Never modify `packages/models` without bumping `modelVersion` and updating the golden fixtures.2013. Prefer extending an existing `RateFunction` over creating a new one; any new kind requires implementation in `packages/counter` (client + server) AND badge rendering.2024. When creating any file: add the mandatory author header (Simon-Pierre Boucher / contact@spboucher.ai) before any code.2035. When in doubt about source data (format, license, cadence): flag it explicitly rather than assuming.2046. UI text displaying numbers goes through helpers in `packages/counter/format.ts` — never inline `toFixed`.2057. After any production deploy to `m3u96b`, run the health checks against https://www.earth-now.co before declaring the deploy done.206