SPB Git

spb/earth-now Public License

earth-now.co — real-time planetary dashboard: live world metrics modeled, not streamed.

🔗 https://earth-now.co
webappdashboarddatarealtime
TypeScript 93% Shell 2.3% SQL 1.4% JavaScript 1.3% Dockerfile 1.2% CSS 0.8%
13.1 KB
View source Raw Blame History

# CLAUDE.md — earth-now.co

# Project Overview

earth-now.co is a real-time planetary dashboard — a robust successor to Worldometer. The site displays live world metrics (population, births, deaths, CO₂ emissions, temperature anomaly, forest loss, energy production, internet users, etc.) that tick continuously on screen.

The fundamental difference from Worldometer: our counters are not opaque linear extrapolations. Every metric is driven by a documented statistical model that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals.

Product pillars:

  1. Live everywhere: dashboard, shareable public pages (/m/:token), embeddable widgets (JS, iframe, SVG badge).
  2. Statistical honesty: every number is traceable (source, observation date, interpolation method, uncertainty).
  3. Robustness: resilient ingestion pipeline; never a frozen or absurd counter in production.

# Architecture Principle #1: A Counter Is a Model, Not a Stream

NEVER push one value per second over the network. The server ships a CounterModel; the client animates locally.

typescript
// The project's central contract — types/counter-model.ts
interface CounterModel {
  metricId: string;            // e.g. "co2_emissions_ytd"
  anchorValue: number;         // value at the anchor point
  anchorTime: string;          // ISO 8601 UTC of the anchor point
  rateFn: RateFunction;        // how the value evolves from the anchor
  uncertainty?: { low: number; high: number };  // 90% CI at time T
  observedAt: string;          // date of the last REAL observation
  sourceId: string;            // key in the source registry
  modelVersion: string;        // e.g. "seasonal-spline-v2"
  displayHints: { decimals: number; sigFigs?: number; unit: string };
}

type RateFunction =
  | { kind: "linear"; perSecond: number }
  | { kind: "piecewise"; segments: Array<{ from: string; perSecond: number }> }
  | { kind: "seasonal"; base: number; harmonics: Harmonic[] }   // Fourier
  | { kind: "spline"; knots: Array<[time: string, value: number]> };

The client computes value(t) at 60 fps with requestAnimationFrame from the model. The server only pushes (via SSE) a new model when: new data is ingested, a new forecast arrives, or a correction is made. Result: zero per-tick traffic, perfect consistency across all clients (same model ⇒ same value at the same instant t), and the SVG badge uses the exact same function server-side.

# Statistical Methodology (the heart of the product)

Every metric declares its model level in the registry. Escalation by level:

Level 0 — Linear: rate = (forecast_annual_value - last_obs) / remaining_seconds. Acceptable only as a fallback or for genuinely quasi-linear metrics.

Level 1 — Seasonal: trend + seasonality decomposition (Fourier harmonics or STL). Mandatory for: births/deaths (demographic seasonality), CO₂ concentration (Keeling curve: ±3 ppm annual cycle), energy consumption, fires/deforestation.

Level 2 — Observation/forecast fusion: when a source publishes both delayed observations and projections (e.g. UN WPP, Global Carbon Budget, NOAA), we blend:

  • Interpolation via monotone spline (PCHIP) between observed points — never a natural cubic spline (possible overshoot ⇒ counters absurdly going backwards).
  • Observation→forecast junction via a Kalman filter or exponential weighting: near the last observation we track the observed data; further out we converge toward the forecast.
  • Holt-Winters internally for nowcasting frequently-published metrics (weekly/monthly) when the source provides no forecast.

Uncertainty: propagate source CIs (e.g. UN low/median/high variants) into uncertainty. The UI may render a band. Absolute rule: displayHints.sigFigs caps significant digits — we NEVER display more precision than the model justifies (world population to the unit is a lie; we animate the unit digit but the tooltip shows the CI).

Validity constraints (enforced by packages/models/validate.ts):

  • Cumulative counters (YTD births, YTD CO₂ emitted): rateFn strictly ≥ 0, reset on Jan 1 UTC via piecewise.
  • Stocks (population, CO₂ ppm): may move both ways, but |derivative| bounded by a max declared in the registry.
  • Every re-fitted model is diffed against the previous one: a jump > declared threshold ⇒ model deployment is blocked and an alert fires (no counter "teleportation" in prod; we catch up with a 60 s client-side smoothing transition).

# Metric & Source Registry

Everything goes through packages/registry/metrics/*.yaml. A metric does not exist until it is declared there. Example:

yaml
id: world_population
name: { fr: "Population mondiale", en: "World population" }
unit: people
kind: stock
model: seasonal-spline-v2        # level 2
sources:
  - id: un_wpp_2024
    url: https://population.un.org/wpp/
    license: CC BY 3.0 IGO
    cadence: yearly
    variants: [low, median, high]
refresh: on_source_update
display: { decimals: 0, sigFigs: 7 }
constraints: { maxAbsRatePerSec: 10 }

Priority sources: UN WPP (demographics), Global Carbon Project + NOAA GML (CO₂), Copernicus/ERA5 (temperature), FAO & Global Forest Watch (forests), Ember/IEA (energy), USGS (earthquakes — special case: true event-driven real time, no interpolation), OWID as a secondary aggregator. Always check and honor each source's license; attribution is displayed in the UI and in API responses.

# Architecture

text
apps/
  web/          # Next.js 14 (App Router) — dashboard, /m/:token pages, /embed
  api/          # Fastify (TS) — REST + SSE
  ingest/       # Ingestion workers (cron + BullMQ queue)
packages/
  registry/     # Metric + source YAML (product source of truth)
  models/       # Interpolators, fitting, validation (pure, no I/O)
  counter/      # Shared client runtime: value(model, t) — used by web, widget AND badge
  widget/       # Embeddable widget.js (IIFE build < 15 KB gzip, zero dependencies)
infra/          # Docker, Terraform, migrations

Data flow:

  1. ingest downloads/parses sources on their cadence (exponential retry, checksums, raw archive to S3 — we keep ALL raw data for re-fitting).
  2. Normalization → observations (append-only, PostgreSQL + TimescaleDB).
  3. models re-fits the CounterModel of each impacted metric → validation → versioned in DB + Redis cache.
  4. Redis pub/sub publish on metric:{id} → SSE servers relay the new model to connected clients.
  5. SVG/PNG badges: rendered server-side with the same packages/counter, CDN cache 60 s.

Database (key tables): observations (append-only), counter_models (versioned, never UPDATEd), metrics_registry_sync, share_tokens (revocable, per-metric visibility), ingest_runs (full auditability of every run).

# Stack

  • Strict TypeScript everywhere ("strict": true, no unjustified any). Node 20+, pnpm workspaces, Turborepo.
  • Next.js 14 + Tailwind (web); Fastify (api); BullMQ + Redis (ingest); PostgreSQL 16 + TimescaleDB; Drizzle ORM + SQL migrations.
  • Statistical fitting: TS implementations in packages/models (PCHIP, least-squares Fourier fit, Holt-Winters, 1D Kalman). If a heavy fit justifies it, an isolated Python service (scipy/statsmodels) under apps/ingest/fitters-py called by the orchestrator — but prefer TS whenever possible to share code with the client.
  • Real time: SSE (EventSource) for model distribution; no WebSocket unless a future bidirectional need arises. Reconnection: Last-Event-ID + Redis Streams (15 min buffer).
  • Cloudflare CDN in front of /badge/* and widget.js.

# Mandatory File Header

Every source code file in this repository (.ts, .tsx, .js, .py, .sql, .sh, config files where comments are allowed) MUST start with the following author header:

typescript
/**
 * earth-now.co
 * Author:  Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * File:    <relative/path/to/file.ts>
 * Purpose: <one-line description of what this file does>
 */

Adapt the comment syntax to the language (# for Python/YAML/shell, -- for SQL). Rules:

  • The File: line must match the file's actual path in the repo — update it if the file is moved/renamed.
  • Purpose: is one line, kept accurate when the file's role changes.
  • CI lint rule (scripts/check-headers.ts, run in pnpm lint) fails any file missing or with a malformed header. When creating a new file, Claude must add this header first, before any code.
  • Generated files (build output, lockfiles, migrations auto-generated by drizzle-kit) are exempt and listed in scripts/check-headers.ignore.

# Deployment

  • Production runs on node m3u96b, exposed publicly through ngrok at www.earth-now.co.
  • Stack on m3u96b: Docker Compose (infra/docker-compose.prod.yml) running web, api, ingest, PostgreSQL/Timescale, and Redis; ngrok tunnel (reserved domain www.earth-now.co) fronting the web/api reverse proxy on port 8080.
  • Deploy flow:
    bash
    pnpm build                      # turbo build of all apps
pnpm deploy:m3u96b              # rsync + docker compose up -d on node m3u96b
pnpm tunnel:status              # verify the ngrok tunnel is up and serving www.earth-now.co
  • ngrok config lives in infra/ngrok.yml (reserved domain + edge). The ngrok authtoken is provided via environment/secret manager — never commit it.
  • Health check after every deploy: GET https://www.earth-now.co/api/health must return 200 and GET /sse/health must hold an SSE connection open ≥ 10 s.
  • Note: ngrok is the current exposure layer; keep the reverse-proxy config portable so we can swap to Cloudflare Tunnel or a plain LB later without app changes.

# Commands

bash
pnpm dev              # whole monorepo (turbo)
pnpm dev:web          # frontend only
pnpm test             # vitest, all packages
pnpm test:models      # interpolator tests (the most critical ones)
pnpm lint && pnpm typecheck   # lint includes the file-header check
pnpm db:migrate       # drizzle-kit
pnpm ingest:run <sourceId>    # force an ingestion locally
pnpm models:refit <metricId>  # manual re-fit + diff report
pnpm deploy:m3u96b    # deploy to production node m3u96b (ngrok → www.earth-now.co)

# Code Conventions

  • Time: UTC everywhere internally, ISO 8601; convert to local time only at display. No hand-rolled date arithmetic — use date-fns or Temporal.
  • Units: SI in the data layer; display conversions (t → Gt, etc.) live in displayHints, never in the pipeline.
  • packages/models and packages/counter are pure and deterministic: no I/O, no implicit system clock (time is always a parameter). This is what makes them testable and shareable client/server.
  • Numbers: never floats for ingested cumulative values (use integers in the base unit when possible); formatting via Intl.NumberFormat.
  • Every new metric = a PR containing: registry YAML + data fixture + golden model test + source license review.
  • i18n: fr + en from day one (keys in the registry and apps/web/messages).

# Testing — Non-negotiable

  • Golden tests for interpolators: for each modelVersion, fixtures (input data → expected values at precise instants t, defined tolerance). Any output change without a version bump = CI failure.
  • Property-based tests (fast-check): monotonicity of cumulatives, derivative bounds, continuity at piecewise junctions, value(anchorTime) === anchorValue.
  • Client/server consistency test: packages/counter run in Node and in jsdom must produce bit-identical values for the same model.
  • Ingestion: every source parser has fixtures of the real format (including degraded cases: truncated file, renamed columns).

# Production Guardrails

  • A counter must never: go backwards if cumulative, visibly jump (> 60 s smoothing), display NaN/undefined (fallback = last frozen value + "data pending" indicator), or show precision > sigFigs.
  • Ingestion failing > 2× the source's cadence ⇒ "stale" badge on the metric + alert (no silent failure).
  • Public endpoints (/m, /embed, /badge, /sse): rate limiting per IP + token, read-only open CORS, revocable share tokens, no user data exposed.
  • Public /methodology page: generated from the registry — every metric exposes its source, cadence, model, and last observation there. This is a product commitment, not an option.

# What Claude Must Do in This Repo

  1. Before adding/modifying a metric: read the registry YAML and its associated golden tests.
  2. Never modify packages/models without bumping modelVersion and updating the golden fixtures.
  3. Prefer extending an existing RateFunction over creating a new one; any new kind requires implementation in packages/counter (client + server) AND badge rendering.
  4. When creating any file: add the mandatory author header (Simon-Pierre Boucher / contact@spboucher.ai) before any code.
  5. When in doubt about source data (format, license, cadence): flag it explicitly rather than assuming.
  6. UI text displaying numbers goes through helpers in packages/counter/format.ts — never inline toFixed.
  7. After any production deploy to m3u96b, run the health checks against https://www.earth-now.co before declaring the deploy done.