# CLAUDE.md — earth-now.co

## Project Overview

**earth-now.co** is a real-time planetary dashboard — a robust successor to Worldometer. The site displays live world metrics (population, births, deaths, CO₂ emissions, temperature anomaly, forest loss, energy production, internet users, etc.) that tick continuously on screen.

The fundamental difference from Worldometer: our counters are not opaque linear extrapolations. Every metric is driven by a **documented statistical model** that interpolates between the latest real observations and forecasts from authoritative sources, with seasonality and uncertainty intervals.

Product pillars:
1. **Live everywhere**: dashboard, shareable public pages (`/m/:token`), embeddable widgets (JS, iframe, SVG badge).
2. **Statistical honesty**: every number is traceable (source, observation date, interpolation method, uncertainty).
3. **Robustness**: resilient ingestion pipeline; never a frozen or absurd counter in production.

---

## Architecture Principle #1: A Counter Is a Model, Not a Stream

**NEVER push one value per second over the network.** The server ships a **CounterModel**; the client animates locally.

```typescript
// The project's central contract — types/counter-model.ts
interface CounterModel {
  metricId: string;            // e.g. "co2_emissions_ytd"
  anchorValue: number;         // value at the anchor point
  anchorTime: string;          // ISO 8601 UTC of the anchor point
  rateFn: RateFunction;        // how the value evolves from the anchor
  uncertainty?: { low: number; high: number };  // 90% CI at time T
  observedAt: string;          // date of the last REAL observation
  sourceId: string;            // key in the source registry
  modelVersion: string;        // e.g. "seasonal-spline-v2"
  displayHints: { decimals: number; sigFigs?: number; unit: string };
}

type RateFunction =
  | { kind: "linear"; perSecond: number }
  | { kind: "piecewise"; segments: Array<{ from: string; perSecond: number }> }
  | { kind: "seasonal"; base: number; harmonics: Harmonic[] }   // Fourier
  | { kind: "spline"; knots: Array<[time: string, value: number]> };
```

The client computes `value(t)` at 60 fps with `requestAnimationFrame` from the model. The server only pushes (via SSE) a **new model** when: new data is ingested, a new forecast arrives, or a correction is made. Result: zero per-tick traffic, perfect consistency across all clients (same model ⇒ same value at the same instant t), and the SVG badge uses the exact same function server-side.

---

## Statistical Methodology (the heart of the product)

Every metric declares its model level in the registry. Escalation by level:

**Level 0 — Linear**: `rate = (forecast_annual_value - last_obs) / remaining_seconds`. Acceptable only as a fallback or for genuinely quasi-linear metrics.

**Level 1 — Seasonal**: trend + seasonality decomposition (Fourier harmonics or STL). Mandatory for: births/deaths (demographic seasonality), CO₂ concentration (Keeling curve: ±3 ppm annual cycle), energy consumption, fires/deforestation.

**Level 2 — Observation/forecast fusion**: when a source publishes both delayed observations and projections (e.g. UN WPP, Global Carbon Budget, NOAA), we blend:
- Interpolation via **monotone spline (PCHIP)** between observed points — never a natural cubic spline (possible overshoot ⇒ counters absurdly going backwards).
- Observation→forecast junction via a **Kalman filter** or exponential weighting: near the last observation we track the observed data; further out we converge toward the forecast.
- **Holt-Winters** internally for nowcasting frequently-published metrics (weekly/monthly) when the source provides no forecast.

**Uncertainty**: propagate source CIs (e.g. UN low/median/high variants) into `uncertainty`. The UI may render a band. Absolute rule: `displayHints.sigFigs` caps significant digits — we NEVER display more precision than the model justifies (world population to the unit is a lie; we animate the unit digit but the tooltip shows the CI).

**Validity constraints** (enforced by `packages/models/validate.ts`):
- Cumulative counters (YTD births, YTD CO₂ emitted): `rateFn` strictly ≥ 0, reset on Jan 1 UTC via `piecewise`.
- Stocks (population, CO₂ ppm): may move both ways, but |derivative| bounded by a max declared in the registry.
- Every re-fitted model is diffed against the previous one: a jump > declared threshold ⇒ model deployment is blocked and an alert fires (no counter "teleportation" in prod; we catch up with a 60 s client-side smoothing transition).

---

## Metric & Source Registry

Everything goes through `packages/registry/metrics/*.yaml`. A metric does not exist until it is declared there. Example:

```yaml
id: world_population
name: { fr: "Population mondiale", en: "World population" }
unit: people
kind: stock
model: seasonal-spline-v2        # level 2
sources:
  - id: un_wpp_2024
    url: https://population.un.org/wpp/
    license: CC BY 3.0 IGO
    cadence: yearly
    variants: [low, median, high]
refresh: on_source_update
display: { decimals: 0, sigFigs: 7 }
constraints: { maxAbsRatePerSec: 10 }
```

Priority sources: UN WPP (demographics), Global Carbon Project + NOAA GML (CO₂), Copernicus/ERA5 (temperature), FAO & Global Forest Watch (forests), Ember/IEA (energy), USGS (earthquakes — special case: true event-driven real time, no interpolation), OWID as a secondary aggregator. **Always check and honor each source's license; attribution is displayed in the UI and in API responses.**

---

## Architecture

```
apps/
  web/          # Next.js 14 (App Router) — dashboard, /m/:token pages, /embed
  api/          # Fastify (TS) — REST + SSE
  ingest/       # Ingestion workers (cron + BullMQ queue)
packages/
  registry/     # Metric + source YAML (product source of truth)
  models/       # Interpolators, fitting, validation (pure, no I/O)
  counter/      # Shared client runtime: value(model, t) — used by web, widget AND badge
  widget/       # Embeddable widget.js (IIFE build < 15 KB gzip, zero dependencies)
infra/          # Docker, Terraform, migrations
```

Data flow:
1. `ingest` downloads/parses sources on their cadence (exponential retry, checksums, raw archive to S3 — we keep ALL raw data for re-fitting).
2. Normalization → `observations` (append-only, PostgreSQL + TimescaleDB).
3. `models` re-fits the CounterModel of each impacted metric → validation → versioned in DB + Redis cache.
4. Redis pub/sub publish on `metric:{id}` → SSE servers relay the new model to connected clients.
5. SVG/PNG badges: rendered server-side with the same `packages/counter`, CDN cache 60 s.

Database (key tables): `observations` (append-only), `counter_models` (versioned, never UPDATEd), `metrics_registry_sync`, `share_tokens` (revocable, per-metric visibility), `ingest_runs` (full auditability of every run).

---

## Stack

- **Strict TypeScript everywhere** (`"strict": true`, no unjustified `any`). Node 20+, pnpm workspaces, Turborepo.
- Next.js 14 + Tailwind (web); Fastify (api); BullMQ + Redis (ingest); PostgreSQL 16 + TimescaleDB; Drizzle ORM + SQL migrations.
- Statistical fitting: TS implementations in `packages/models` (PCHIP, least-squares Fourier fit, Holt-Winters, 1D Kalman). If a heavy fit justifies it, an isolated Python service (scipy/statsmodels) under `apps/ingest/fitters-py` called by the orchestrator — but prefer TS whenever possible to share code with the client.
- Real time: **SSE** (EventSource) for model distribution; no WebSocket unless a future bidirectional need arises. Reconnection: `Last-Event-ID` + Redis Streams (15 min buffer).
- Cloudflare CDN in front of `/badge/*` and `widget.js`.

## Mandatory File Header

**Every source code file in this repository (`.ts`, `.tsx`, `.js`, `.py`, `.sql`, `.sh`, config files where comments are allowed) MUST start with the following author header:**

```typescript
/**
 * earth-now.co
 * Author:  Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * File:    <relative/path/to/file.ts>
 * Purpose: <one-line description of what this file does>
 */
```

Adapt the comment syntax to the language (`#` for Python/YAML/shell, `--` for SQL). Rules:
- The `File:` line must match the file's actual path in the repo — update it if the file is moved/renamed.
- `Purpose:` is one line, kept accurate when the file's role changes.
- CI lint rule (`scripts/check-headers.ts`, run in `pnpm lint`) fails any file missing or with a malformed header. When creating a new file, Claude must add this header first, before any code.
- Generated files (build output, lockfiles, migrations auto-generated by drizzle-kit) are exempt and listed in `scripts/check-headers.ignore`.

## Deployment

- Production runs on **node `m3u96b`**, exposed publicly through **ngrok** at **www.earth-now.co**.
- Stack on `m3u96b`: Docker Compose (`infra/docker-compose.prod.yml`) running web, api, ingest, PostgreSQL/Timescale, and Redis; ngrok tunnel (reserved domain `www.earth-now.co`) fronting the web/api reverse proxy on port 8080.
- Deploy flow:
  ```bash
  pnpm build                      # turbo build of all apps
  pnpm deploy:m3u96b              # rsync + docker compose up -d on node m3u96b
  pnpm tunnel:status              # verify the ngrok tunnel is up and serving www.earth-now.co
  ```
- ngrok config lives in `infra/ngrok.yml` (reserved domain + edge). The ngrok authtoken is provided via environment/secret manager — **never commit it**.
- Health check after every deploy: `GET https://www.earth-now.co/api/health` must return 200 and `GET /sse/health` must hold an SSE connection open ≥ 10 s.
- Note: ngrok is the current exposure layer; keep the reverse-proxy config portable so we can swap to Cloudflare Tunnel or a plain LB later without app changes.

## Commands

```bash
pnpm dev              # whole monorepo (turbo)
pnpm dev:web          # frontend only
pnpm test             # vitest, all packages
pnpm test:models      # interpolator tests (the most critical ones)
pnpm lint && pnpm typecheck   # lint includes the file-header check
pnpm db:migrate       # drizzle-kit
pnpm ingest:run <sourceId>    # force an ingestion locally
pnpm models:refit <metricId>  # manual re-fit + diff report
pnpm deploy:m3u96b    # deploy to production node m3u96b (ngrok → www.earth-now.co)
```

## Code Conventions

- Time: **UTC everywhere internally**, ISO 8601; convert to local time only at display. No hand-rolled date arithmetic — use `date-fns` or `Temporal`.
- Units: SI in the data layer; display conversions (t → Gt, etc.) live in `displayHints`, never in the pipeline.
- `packages/models` and `packages/counter` are **pure and deterministic**: no I/O, no implicit system clock (time is always a parameter). This is what makes them testable and shareable client/server.
- Numbers: never floats for ingested cumulative values (use integers in the base unit when possible); formatting via `Intl.NumberFormat`.
- Every new metric = a PR containing: registry YAML + data fixture + golden model test + source license review.
- i18n: fr + en from day one (keys in the registry and `apps/web/messages`).

## Testing — Non-negotiable

- **Golden tests for interpolators**: for each `modelVersion`, fixtures (input data → expected values at precise instants t, defined tolerance). Any output change without a version bump = CI failure.
- Property-based tests (fast-check): monotonicity of cumulatives, derivative bounds, continuity at piecewise junctions, `value(anchorTime) === anchorValue`.
- Client/server consistency test: `packages/counter` run in Node and in jsdom must produce bit-identical values for the same model.
- Ingestion: every source parser has fixtures of the real format (including degraded cases: truncated file, renamed columns).

## Production Guardrails

- A counter must **never**: go backwards if cumulative, visibly jump (> 60 s smoothing), display NaN/undefined (fallback = last frozen value + "data pending" indicator), or show precision > sigFigs.
- Ingestion failing > 2× the source's cadence ⇒ "stale" badge on the metric + alert (no silent failure).
- Public endpoints (`/m`, `/embed`, `/badge`, `/sse`): rate limiting per IP + token, read-only open CORS, revocable share tokens, no user data exposed.
- Public /methodology page: generated from the registry — every metric exposes its source, cadence, model, and last observation there. This is a product commitment, not an option.

## What Claude Must Do in This Repo

1. Before adding/modifying a metric: read the registry YAML and its associated golden tests.
2. Never modify `packages/models` without bumping `modelVersion` and updating the golden fixtures.
3. Prefer extending an existing `RateFunction` over creating a new one; any new kind requires implementation in `packages/counter` (client + server) AND badge rendering.
4. When creating any file: add the mandatory author header (Simon-Pierre Boucher / contact@spboucher.ai) before any code.
5. When in doubt about source data (format, license, cadence): flag it explicitly rather than assuming.
6. UI text displaying numbers goes through helpers in `packages/counter/format.ts` — never inline `toFixed`.
7. After any production deploy to `m3u96b`, run the health checks against https://www.earth-now.co before declaring the deploy done.