SPB Git
spboucher.ai

spb/countryatlas

Public

CountryAtlas: architecture contract, registries (218 countries, 259 indicators, 32 groups, 19 topics), shared models, connector base, DuckDB schema, mld manifest

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 11, 2026) Browse files

22 changed files +12,138 −0

22 changed files
Load all 22 files Unified Split
added .gitignore +20 −0
@@ -0,0 +1,20 @@
1 +node_modules/
2 +.next/
3 +.env
4 +.env.*
5 +!.env.example
6 +*.tsbuildinfo
7 +.venv/
8 +__pycache__/
9 +*.pyc
10 +.pytest_cache/
11 +.ruff_cache/
12 +data/
13 +*.duckdb
14 +*.duckdb.wal
15 +logs/
16 +tmp/
17 +.DS_Store
18 +.claude/
19 +qa/screens/
20 +apps/web/next-env.d.ts
added README.md +54 −0
@@ -0,0 +1,54 @@
1 +# CountryAtlas
2 +
3 +**www.countryatlas.co — Understand the world, one country at a time.**
4 +
5 +The definitive, sourced, historical data profile for every country: economy, government, population, labour, income,
6 +housing, health, education, trade, energy, climate, environment, infrastructure, digital, innovation, agriculture,
7 +tourism, security and quality of life — with rankings, comparisons, similarity, change detection and full provenance.
8 +
9 +## Layout
10 +
11 +```
12 +registry/ canonical registries (YAML, versioned): countries, groups, indicators, topics, similarity, insights
13 +src/countryatlas/ Python platform
14 + registry/ loaders + country-code lookups
15 + connectors/ one module per provider (worldbank, owid, imf, oecd, eurostat, who, fred, bis, ilo)
16 + pipeline/ fetch → normalize → validate → build (DuckDB snapshot) → derived tables; scheduler; exports
17 + storage/schema.sql DuckDB schema (single source of truth for pipeline and API)
18 + api/ FastAPI public API (/api/v1, OpenAPI at /api/v1/docs)
19 + cli.py `ca` command line
20 +apps/web/ Next.js 16 app (App Router, Tailwind v4, own SVG chart kit, d3-geo maps)
21 +deploy/ mld manifest for the MacLustr cluster
22 +docs/ ARCHITECTURE.md (contract), PIPELINE.md, API.md, sources-research.md
23 +tests/ pytest
24 +```
25 +
26 +## Run locally
27 +
28 +```bash
29 +uv venv --python 3.12 .venv && uv pip install -e ".[dev]"
30 +.venv/bin/ca registry validate
31 +CA_DATA_DIR=~/countryatlas-data .venv/bin/ca refresh # fetch all connectors, build ~/countryatlas-data/atlas.duckdb
32 +CA_DATA_DIR=~/countryatlas-data .venv/bin/ca status
33 +CA_DATA_DIR=~/countryatlas-data .venv/bin/python -m uvicorn countryatlas.api.main:app --port 8291
34 +pnpm install && cd apps/web && API_URL=http://127.0.0.1:8291 pnpm dev # http://localhost:8290
35 +```
36 +
37 +## Deploy (MacLustr)
38 +
39 +Manifest `deploy/countryatlas.mld.json` → `M1M32:~/dispatch/apps/countryatlas.json`, then from the laptop:
40 +
41 +```bash
42 +~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/countryatlas countryatlas
43 +~/Desktop/cluster-skill/mld deploy countryatlas --node M2M32
44 +```
45 +
46 +Public route `https://www.countryatlas.co → M2M32:8290` is created on the BHS64 gateway by `mld deploy`
47 +(DNS `A www → 51.161.112.61`). Processes (PM2): `countryatlas-web` (:8290), `countryatlas-api` (127.0.0.1:8291),
48 +`countryatlas-scheduler` (daily refresh 03:15 America/Toronto).
49 +
50 +## Data sources & licences
51 +
52 +World Bank WDI (CC BY 4.0), IMF WEO (IMF terms), OECD (CC BY 4.0), Eurostat (CC BY 4.0), WHO GHO (CC BY-NC-SA 3.0 IGO),
53 +FRED (St. Louis Fed terms; series-level licences), Our World in Data (CC BY 4.0), BIS (BIS terms). Every value on the
54 +site and in the API carries its source, dataset, series code, retrieval and source-update dates.
added deploy/.admin-token +1 −0
@@ -0,0 +1 @@
1 +c9ae6c67cdeb415c9df7db09d255c86d3537fb7670cf3c5a
added deploy/countryatlas.mld.json +105 −0
@@ -0,0 +1,105 @@
1 +{
2 + "app": "countryatlas",
3 + "label": "CountryAtlas — Understand the world, one country at a time",
4 + "domain": "www.countryatlas.co",
5 + "port": 8290,
6 + "health_path": "/api/v1/health",
7 + "dir": "~/apps/countryatlas",
8 + "extra_paths": [],
9 + "sync_excludes": [
10 + ".venv/", "__pycache__/", ".pytest_cache/", ".ruff_cache/", "*.egg-info/", ".git/", ".env", ".env.*", "!.env.example",
11 + "node_modules/", "apps/web/.next/", "apps/web/next-env.d.ts", "*.tsbuildinfo", "apps/web/qa/screens/",
12 + "data/", "*.duckdb", "*.duckdb.wal", "logs/", "tmp/", ".DS_Store", ".claude/", "deploy/.admin-token", "docs/sources-research.md"
13 + ],
14 + "requires": {
15 + "runtimes": ["pm2", "node", "pnpm", "uv", "uv-python@3.12"],
16 + "ram_gb": 6,
17 + "ports": [8290, 8291]
18 + },
19 + "ram_mb_observed": 2500,
20 + "size_mb": 60,
21 + "placement": {
22 + "pin": "M2M32",
23 + "prefer": null,
24 + "avoid": ["M3U96b", "M1M32"],
25 + "reason": "Mac Studio M2 Max 12 c / 32 Go dédié (aucune autre app), uv + pnpm + node présents, raccordé au tunnel wg1 ; pipeline quotidien + DuckDB en lecture"
26 + },
27 + "processes": [
28 + {
29 + "name": "countryatlas-api",
30 + "manager": "pm2",
31 + "script": "{{HOME}}/apps/countryatlas/.venv/bin/python",
32 + "args": ["-m", "uvicorn", "countryatlas.api.main:app", "--host", "127.0.0.1", "--port", "8291", "--no-access-log", "--proxy-headers", "--timeout-keep-alive", "75"],
33 + "interpreter": null,
34 + "cwd": "{{HOME}}/apps/countryatlas",
35 + "env": {
36 + "CA_DATA_DIR": "{{HOME}}/countryatlas-data",
37 + "CA_API_HOST": "127.0.0.1",
38 + "CA_API_PORT": "8291",
39 + "CA_ADMIN_TOKEN": "c9ae6c67cdeb415c9df7db09d255c86d3537fb7670cf3c5a",
40 + "CA_SITE_URL": "https://www.countryatlas.co",
41 + "FRED_API_KEY": "5d6f76382d7d188e9166bb3b96c8f934",
42 + "PYTHONUNBUFFERED": "1"
43 + },
44 + "cron_restart": null,
45 + "autorestart": true,
46 + "max_memory_restart": "3G"
47 + },
48 + {
49 + "name": "countryatlas-scheduler",
50 + "manager": "pm2",
51 + "script": "{{HOME}}/apps/countryatlas/.venv/bin/ca",
52 + "args": ["schedule"],
53 + "interpreter": null,
54 + "cwd": "{{HOME}}/apps/countryatlas",
55 + "env": {
56 + "CA_DATA_DIR": "{{HOME}}/countryatlas-data",
57 + "CA_ADMIN_TOKEN": "c9ae6c67cdeb415c9df7db09d255c86d3537fb7670cf3c5a",
58 + "FRED_API_KEY": "5d6f76382d7d188e9166bb3b96c8f934",
59 + "CA_TZ": "America/Toronto",
60 + "CA_REFRESH_HOUR": "3",
61 + "CA_REFRESH_MINUTE": "15",
62 + "PYTHONUNBUFFERED": "1"
63 + },
64 + "cron_restart": null,
65 + "autorestart": true,
66 + "max_memory_restart": "6G"
67 + },
68 + {
69 + "name": "countryatlas-web",
70 + "manager": "pm2",
71 + "script": "/opt/homebrew/bin/node",
72 + "args": ["node_modules/next/dist/bin/next", "start", "-p", "8290", "-H", "0.0.0.0"],
73 + "interpreter": null,
74 + "cwd": "{{HOME}}/apps/countryatlas/apps/web",
75 + "env": {
76 + "NODE_ENV": "production",
77 + "API_URL": "http://127.0.0.1:8291",
78 + "NEXT_PUBLIC_SITE_URL": "https://www.countryatlas.co",
79 + "CA_ADMIN_TOKEN": "c9ae6c67cdeb415c9df7db09d255c86d3537fb7670cf3c5a",
80 + "NEXT_TELEMETRY_DISABLED": "1"
81 + },
82 + "cron_restart": null,
83 + "autorestart": true,
84 + "max_memory_restart": "2G"
85 + }
86 + ],
87 + "ngrok": null,
88 + "launchd": [],
89 + "env_overrides": {},
90 + "hooks": {
91 + "post_sync": [
92 + "export PATH=\"$HOME/.local/bin:/opt/homebrew/bin:$PATH\"; (test -x .venv/bin/python || uv venv --python 3.12 .venv) && uv pip install -q --python .venv/bin/python -e . && echo ' python deps ok'",
93 + "export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' web deps ok'",
94 + "export PATH=\"/opt/homebrew/bin:$PATH\"; cd apps/web && API_URL=http://127.0.0.1:8291 NEXT_PUBLIC_SITE_URL=https://www.countryatlas.co NEXT_TELEMETRY_DISABLED=1 pnpm build 2>&1 | tail -3 && echo ' web build ok'",
95 + "mkdir -p $HOME/countryatlas-data/{raw,staging,build,snapshots,exports,logs} && echo ' data dirs ok'"
96 + ],
97 + "post_start": []
98 + },
99 + "notes": "v0.1.0 (2026-09-11) : Next 16 :8290 (rewrites /api/v1/* → FastAPI 127.0.0.1:8291), pipeline Python (ca refresh quotidien 03:15 via countryatlas-scheduler), DuckDB snapshot ~/countryatlas-data/atlas.duckdb (swap atomique). Sources : World Bank, IMF WEO, OECD, Eurostat, WHO GHO, FRED, OWID, BIS. Secrets (CA_ADMIN_TOKEN, FRED_API_KEY) uniquement ici.",
100 + "tunnel": {
101 + "domain": "www.countryatlas.co",
102 + "gateway": "BHS64",
103 + "note": "DNS A www → 51.161.112.61 posé par l'utilisateur le 2026-09-11 ; apex countryatlas.co sans A record (redirect Caddy à poser quand l'A @ existera)"
104 + }
105 +}
added docs/ARCHITECTURE.md +257 −0
@@ -0,0 +1,257 @@
1 +# CountryAtlas — Architecture (contract for all contributors)
2 +
3 +> **www.countryatlas.co — Understand the world, one country at a time.**
4 +> The country is the primary unit of navigation. Every visible number is traceable to its source.
5 +
6 +This document is the binding contract between the data platform (Python), the API (FastAPI) and the
7 +web app (Next.js). Change it first, then the code.
8 +
9 +## 1. Deployment topology (MacLustr cluster)
10 +
11 +```
12 +Internet ─ DNS A www.countryatlas.co → 51.161.112.61 (BHS64, OVH Beauharnois)
13 + └─ Caddy (TLS auto) ─ WireGuard wg1 ─→ M2M32 (Mac Studio M2, 12c/32GB) 10.67.0.x
14 + ├─ countryatlas-web Next.js 16 :8290 (public upstream)
15 + ├─ countryatlas-api FastAPI :8291 (loopback only; reached via Next rewrites /api/v1/*)
16 + └─ countryatlas-scheduler Python loop (daily refresh, writes snapshots)
17 + ~/countryatlas-data/ (raw/, staging/, build/, atlas.duckdb, exports/, logs/)
18 +```
19 +
20 +* Orchestrated by `mld` from the gateway M1M32 (manifest `deploy/countryatlas.mld.json`, PM2 processes).
21 +* No database port is ever public. The API listens on 127.0.0.1:8291 only.
22 +* The web app listens on 0.0.0.0:8290 (reached through the WireGuard tunnel only; the Mac has no public IP).
23 +* Apex `countryatlas.co` → 301 to `www` (Caddy redirect on BHS64; requires an A record for the apex).
24 +
25 +## 2. Storage: DuckDB snapshots (no Postgres/ClickHouse/Redis for the MVP)
26 +
27 +Volumes: ~220 countries × ~250 indicators × ≤65 years ≈ 3–4 M observations. DuckDB handles this in
28 +milliseconds, exports Parquet/CSV natively and needs zero operations. The design keeps the door open to
29 +Postgres/ClickHouse later (same logical schema).
30 +
31 +**Snapshot-swap rule (critical):**
32 +
33 +1. The pipeline builds a brand-new file `~/countryatlas-data/build/atlas-<run_id>.duckdb`.
34 +2. Derived tables are computed inside it (latest, rankings, changes, events, similarity, insights, coverage).
35 +3. Integrity checks pass → `os.replace()` it onto `~/countryatlas-data/atlas.duckdb` (atomic rename) and copy
36 + to `snapshots/atlas-<run_id>.duckdb` (keep last 7).
37 +4. The API opens the DB **read-only**; on every request it compares the file's `st_ino`/`st_mtime` with the open
38 + one and reopens when they differ. Readers never block the writer and vice-versa.
39 +
40 +Raw source payloads are kept forever (gzip JSON/CSV) under `raw/<connector>/<dataset>/<YYYY-MM-DD>/…`.
41 +Nothing is ever fetched without also being stored raw.
42 +
43 +### 2.1 Logical schema (DuckDB)
44 +
45 +```sql
46 +countries(id TEXT PK /* = iso3 */, iso2, iso3, iso_numeric, slug, short_name, official_name, capital,
47 + continent, region_wb /* WB region id */, region_wb_name, subregion, income_group /* HIC/UMC/LMC/LIC */,
48 + currency_code, currency_name, area_km2 DOUBLE, latitude, longitude, flag_emoji,
49 + un_member BOOLEAN, status TEXT /* country|territory|historical */, independent BOOLEAN,
50 + landlocked BOOLEAN, borders TEXT[] , languages TEXT[], demonym, kind TEXT /* country|aggregate */)
51 +groups(id TEXT PK /* world|oecd|g7|g20|eu|brics|nac|ecs|hic|... */, slug, name, kind /* world|region|income|org|custom */,
52 + description, wb_code /* WLD, OED, EUU, HIC ... when the source has an aggregate */)
53 +group_members(group_id, country_id)
54 +sources(id TEXT PK /* worldbank|imf|oecd|eurostat|who|fred|owid|bis|ilo */, name, organization, url, licence,
55 + attribution, api_base, last_success_at, notes)
56 +indicators(id TEXT PK /* = slug */, slug, name, short_name, description, topic, subtopic, unit, unit_short,
57 + frequency /* A|Q|M */, precision INT, aggregation /* sum|mean|weighted_mean|none */,
58 + higher_is_better BOOLEAN NULL, ranking_eligible BOOLEAN, per_capita_of TEXT NULL,
59 + format TEXT /* number|percent|currency|index|years|per_1000|... */, scale TEXT /* raw|thousands|millions|billions */,
60 + source_priority TEXT[] , methodology TEXT, tags TEXT[], featured BOOLEAN)
61 +indicator_sources(indicator_id, source_id, dataset, series_code, params JSON, priority INT, transform TEXT NULL, notes)
62 +observations(country_id, indicator_id, period DATE /* first day of period */, year INT, frequency,
63 + value DOUBLE, unit, source_id, source_dataset, source_series_code,
64 + is_estimate BOOL, is_forecast BOOL, revision INT, retrieved_at TIMESTAMP, source_updated_at TIMESTAMP,
65 + status TEXT /* verified|imported|warning|stale|quarantined */, metadata JSON)
66 + -- PK (country_id, indicator_id, period, frequency). Exactly ONE source per (indicator,country,period) is kept in
67 + -- observations: the highest-priority source that has a value. Alternatives are kept in observations_alt.
68 +observations_alt(same columns) -- lower-priority sources, for provenance/inspection and fallbacks
69 +observation_revisions(country_id, indicator_id, period, frequency, old_value, new_value, old_source_id, new_source_id,
70 + changed_at TIMESTAMP, run_id) -- never silently overwrite: carry forward from the previous snapshot
71 +latest(country_id, indicator_id, period, year, value, prev_period, prev_value, change_abs, change_pct,
72 + rank_world INT, n_world INT, rank_region INT, n_region INT, rank_income INT, n_income INT,
73 + source_id, is_forecast, is_estimate, status) -- latest NON-forecast observation per country×indicator
74 +rankings(indicator_id, year, country_id, value, rank INT, n INT, pct_rank DOUBLE) -- countries only (kind='country'), non-forecast
75 +changes(id, country_id, indicator_id, kind /* yoy_drop|yoy_jump|record_high|record_low|n_year_high|n_year_low|sign_flip|accelerating|decelerating */,
76 + period, year, value, ref_value, delta, delta_pct, window_years INT, severity DOUBLE /* 0-1 */, headline TEXT, detail JSON, detected_at)
77 + -- "What changed" = recent (latest period) ; events = whole history
78 +events(id, country_id, indicator_id, kind, period, year, value, ref_value, delta, delta_pct, severity, headline, detail JSON)
79 +similarity(country_id, mode /* overall|economic|demographic|energy|social */, peer_id, score DOUBLE /* 0-100 */, rank INT,
80 + contributions JSON /* {indicator: {z_a, z_b, weight, contribution}} */)
81 +insights(id, country_id, template_id, text, values JSON, indicators TEXT[], computed_at)
82 +country_dna(country_id, dims JSON /* {income:0-100, demographics:..., urbanization, trade, energy, emissions, innovation, education, public_spending} */, year_ref)
83 +coverage(country_id, n_indicators INT, n_observations INT, latest_year INT, coverage_pct DOUBLE, updated_at)
84 +import_runs(run_id, connector, dataset, started_at, finished_at, status /* ok|failed|partial */, rows_raw, rows_norm, rows_valid,
85 + warnings INT, errors INT, message, raw_path)
86 +validation_issues(run_id, connector, indicator_id, country_id, period, severity /* info|warning|error */, code, message)
87 +meta(key, value) -- build_run_id, built_at, schema_version, indicator_count, observation_count, ...
88 +```
89 +
90 +## 3. Registries (YAML, versioned in git, loaded by both pipeline and API)
91 +
92 +* `registry/countries.yaml` — canonical registry generated by `scripts/build_country_registry.py` from the World Bank
93 + country list (region, income group, capital, lat/long) merged with the mledoze/countries dataset (ISO numeric, official
94 + name, subregion, currency, area, UN membership, flag, borders, languages). Hand-edited fields survive regeneration
95 + (`overrides:` block). Aggregates (WLD, OED, EUU, HIC …) are **not** countries; they live in `groups.yaml`.
96 +* `registry/groups.yaml` — World, WB regions (7), income groups (4), organisations (OECD, EU, G7, G20, BRICS, ASEAN,
97 + African Union, Eurozone, Nordic, NAFTA/USMCA, Commonwealth…) with explicit member lists (ISO3).
98 +* `registry/indicators.yaml` — the canonical indicator registry (see §4). Every indicator maps to one or more
99 + external series (connector + code + params + priority).
100 +* `registry/topics.yaml` — topics (economy, government, population, …) with order, blurb, icon, and the ordered list of
101 + indicator slugs to show on the country topic page, plus `headline` indicators for the country overview.
102 +
103 +**Rule:** the code never identifies a country by its name. Always `iso3` (id) or `slug`.
104 +URLs use slugs (`/countries/canada`); the API accepts both iso3 and slug.
105 +
106 +## 4. Indicator conventions
107 +
108 +* `slug` is kebab-case, stable, and is the URL: `/indicators/gdp-per-capita`.
109 +* `unit` is human (`current US$`, `% of GDP`, `years`, `people`, `% of population`), `unit_short` for axes (`US$`, `%`, `yrs`).
110 +* `format` drives display: `currency` (scale automatically: 1.2T / 45.3B), `percent` (1 decimal), `number` (thousand separators,
111 + scale for large), `years`, `index`, `per_1000`, `per_100k`, `ratio`, `celsius`, `tonnes`.
112 +* `higher_is_better` is only set when it is unambiguous (life expectancy = true, infant mortality = false). Otherwise null
113 + (inflation, population, exchange rate) — rankings then sort descending by value and say "highest".
114 +* `ranking_eligible` false for series that make no sense to rank (exchange rate, policy rate in local terms, indexes).
115 +* Period → `period` is the first day: annual `YYYY-01-01`, quarterly `YYYY-{01,04,07,10}-01`, monthly `YYYY-MM-01`.
116 +* Forecast observations (IMF WEO projections) are stored with `is_forecast=true` and never enter `latest`, `rankings`,
117 + `changes` or `events`; they are shown dashed on charts.
118 +* A "topic" belongs to the fixed set: `economy, government, population, labor, income, housing, health, education, trade,
119 + energy, climate, environment, infrastructure, digital, innovation, agriculture, tourism, security, quality-of-life`.
120 +
121 +## 5. Pipeline (`ca` CLI, package `countryatlas`)
122 +
123 +```
124 +ca registry validate # YAML sanity (unique slugs, known connectors, topics)
125 +ca fetch [--connector X] [--indicator Y] # raw payloads → raw/… (retries, backoff, rate limit, ETag/If-Modified-Since when available)
126 +ca normalize [--connector X] # raw → staging/<connector>/<dataset>.parquet in the NormalizedObservation schema
127 +ca validate # rules (§6) → statuses + validation_issues
128 +ca build # merge staging by source priority → new DuckDB → derived tables → integrity → atomic swap
129 +ca refresh [--connector X] # fetch + normalize + validate + build (idempotent; safe to run any time)
130 +ca schedule # long-running loop: refresh daily at 03:15 America/Toronto; also refresh on SIGUSR1
131 +ca status # last runs, row counts, freshness, stale sources
132 +ca export indicator <slug> | country <iso3> # CSV/JSON/Parquet into exports/ (also served by the API on demand)
133 +```
134 +
135 +Connector interface (`countryatlas/connectors/base.py`):
136 +
137 +```python
138 +class Connector(Protocol):
139 + id: str # "worldbank"
140 + def discover(self) -> list[DatasetDescriptor]
141 + def fetch(self, spec: IndicatorSourceSpec, ctx: FetchContext) -> RawPayload # one HTTP "unit" per spec; stored raw
142 + def normalize(self, raw: RawPayload, spec: IndicatorSourceSpec) -> list[NormalizedObservation]
143 + def validate(self, rows: list[NormalizedObservation]) -> ValidationReport # source-specific checks
144 +```
145 +
146 +Error isolation: each (connector, dataset) runs in its own try/except and its own `import_runs` row. A failed fetch
147 +**never** removes previously good data: `build` merges the newest successful staging file per dataset. A connector
148 +that fails validation with `error` severity is quarantined (its staging file is ignored and the previous one kept).
149 +
150 +## 6. Validation rules (deterministic)
151 +
152 +* duplicates (same key twice in one dataset) → error (dataset quarantined)
153 +* impossible values: negative for `nonnegative` indicators, percentages outside [-5, 105] for `percent_share`
154 + indicators, life expectancy outside [20, 100], etc. (bounds in registry `bounds: [min, max]`) → row `quarantined`
155 +* unit change vs registry unit → dataset quarantined
156 +* extreme jump: |Δ| > `jump_threshold` × robust std of the country series (MAD) → row `warning` (kept, flagged)
157 +* partial download: rows < 30 % of the previous run for the same dataset → dataset quarantined (previous kept)
158 +* stale: `source_updated_at` / latest period older than `stale_after_days` (annual: 800 d, quarterly: 200 d, monthly: 75 d) → `stale`
159 +* mapping errors: unknown country code → logged, row dropped (aggregates are dropped unless mapped to a group)
160 +
161 +Unusual values are never deleted, only flagged.
162 +
163 +## 7. Derived computations
164 +
165 +* **latest**: last non-forecast observation per country×indicator; previous = previous period of the same frequency; ranks
166 + among `kind='country'` for the same *year* (if a country's latest year is older than the global max year by > 2,
167 + its rank is still computed within that year but flagged `rank_year != max_year` in the API response).
168 +* **rankings**: for every `ranking_eligible` indicator and every year with ≥ 20 countries.
169 +* **changes / events**: per country×indicator series (annual or higher), deterministic detectors:
170 + YoY change beyond ±(2 × MAD of yearly diffs) and beyond an indicator-specific absolute floor (e.g. inflation ±2 pts,
171 + unemployment ±1 pt, GDP growth ±3 pts, population growth ±0.5 pt), record high/low over the whole series, N-year high/low
172 + (N ∈ {10, 20, 30}), sign flip (growth → contraction), acceleration (3 consecutive increases of the diff).
173 + `severity` = min(1, |z| / 4) blended with the floor ratio. Headlines are template strings with computed numbers —
174 + **no LLM**.
175 +* **similarity**: features per mode (see `registry/similarity.yaml`): log-transform heavy-tailed features (GDP pc, population,
176 + area), z-score across countries (latest values, only countries with ≥ 70 % of the mode's features), weighted Euclidean
177 + distance → score = 100 × exp(−d / d₀); store the top 12 peers with per-feature contributions (explainable).
178 +* **insights**: templates in `registry/insights.yaml` (e.g. "{country}'s population grew {pct}% since {y0}."),
179 + each fully computed from data; numbers are never generated by a model.
180 +* **country_dna**: 9 dimensions in [0, 100] = percentile rank of the country for a representative indicator (or the mean of
181 + 2-3 indicators) among all countries: income (GDP pc PPP), demographics (median age ↔ fertility), urbanization,
182 + trade (trade % GDP), energy (energy use pc), emissions (CO₂ pc), innovation (R&D % GDP, patents pc), education
183 + (tertiary enrolment, expected years), public spending (gov. expenditure % GDP). Descriptive, not a score.
184 +
185 +## 8. API (FastAPI, `/api/v1`, OpenAPI at `/api/v1/openapi.json`, docs at `/api/v1/docs`)
186 +
187 +Every response carries `meta: {built_at, run_id, generated_at}`; every value object carries provenance:
188 +
189 +```json
190 +{"value": 53372.1, "period": "2024-01-01", "year": 2024, "unit": "current US$", "is_estimate": false, "is_forecast": false,
191 + "status": "verified",
192 + "provenance": {"source": "worldbank", "source_name": "World Bank", "dataset": "WDI", "series_code": "NY.GDP.PCAP.CD",
193 + "retrieved_at": "2026-09-11T03:20:11Z", "source_updated_at": "2026-07-01", "url": "https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?locations=CA",
194 + "transform": null, "licence": "CC BY 4.0"}}
195 +```
196 +
197 +Endpoints (all GET, JSON, cached in-process with the snapshot run_id as cache key):
198 +
199 +```
200 +/health {status, run_id, built_at, observations}
201 +/countries?region=&income=&q= list (id, slug, name, flag, region, income, population_latest, gdp_latest, coverage_pct)
202 +/countries/{id} header + headline metrics (topics.yaml `headline`) with latest+rank+change
203 +/countries/{id}/topics/{topic} all indicators of the topic: latest + sparkline (last 30 pts) + rank
204 +/countries/{id}/series/{indicator}?from=&to=&freq= full history (values[], with per-observation provenance summary)
205 +/countries/{id}/changes recent changes (sorted by severity)
206 +/countries/{id}/events?limit= timeline
207 +/countries/{id}/similar?mode=overall peers with contributions
208 +/countries/{id}/insights
209 +/countries/{id}/dna
210 +/countries/{id}/download.{csv|json} full country dataset
211 +/indicators?topic=&q= list
212 +/indicators/{slug} definition + sources + coverage + world/regions latest + freshness
213 +/indicators/{slug}/map?year= {year, values: {ISO3: value}, legend: {min,max,breaks[]}, n}
214 +/indicators/{slug}/trend?group=world median/aggregate trend (WB aggregate when available)
215 +/indicators/{slug}/download.{csv|json}
216 +/series?country=CAN,FRA&indicator=gdp-per-capita&from=1990&to=2026&freq=A
217 +/rankings list of rankable indicators (featured first)
218 +/rankings/{indicator}?year=&group=&sort=asc|desc&limit=&offset= rows: rank, country, value, change_1y, change_10y, sparkline
219 +/rankings/{indicator}/history?countries=CAN,USA rank by year
220 +/compare?countries=CAN,USA,FRA&indicators=…&from=&to=&mode=absolute|per-capita|index100|pct series bundle
221 +/compare/snapshot?countries=…&topic= table of latest values for the topic's indicators
222 +/regions | /regions/{slug} group page data (members, aggregate metrics, member rankings)
223 +/search?q=&limit= [{type: country|indicator|topic|region|source, id, slug, name, hint, score}]
224 +/home global snapshot + curated lists (largest economies, fastest growth, …) + recent changes + recently updated
225 +/changes?limit=&kind= global recent changes feed
226 +/sources | /sources/{id} source metadata + datasets + import runs + freshness
227 +/methodology static from registry (units, priorities, validation rules) — for the page
228 +/admin/* (header X-Admin-Token) connectors health, import_runs, validation_issues, coverage matrix, trigger refresh (SIGUSR1)
229 +```
230 +
231 +Errors: RFC 7807 problem+json. Unknown country/indicator → 404. Rate limit 120 req/min/IP (in-process token bucket).
232 +
233 +## 9. Web app (Next.js 16, React 19, TypeScript, Tailwind v4, App Router)
234 +
235 +* Server components fetch the API at `process.env.API_URL` (`http://127.0.0.1:8291`). Browser-side fetches go to the
236 + same origin `/api/v1/*` (Next `rewrites`).
237 +* Routes: `/`, `/countries`, `/countries/[slug]`, `/countries/[slug]/[topic]`, `/compare`, `/compare/[...slugs]`, `/rankings`,
238 + `/rankings/[indicator]`, `/indicators`, `/indicators/[slug]`, `/regions`, `/regions/[slug]`, `/explore`, `/data`, `/sources`,
239 + `/sources/[id]`, `/methodology`, `/api` (docs landing), `/admin` (token), `/sitemap.xml`, `/robots.txt`, `/opengraph-image`.
240 +* Charts: our own SVG chart kit in `apps/web/src/components/charts/` (server-renderable, touch, accessible summaries):
241 + LineChart, AreaChart, StackedArea, RankedBars, Scatter/Bubble, Sparkline, SlopeChart, IndexedLine, SmallMultiples,
242 + PopulationPyramid, Choropleth (d3-geo + world-atlas 110m, SVG), DNA radial.
243 +* i18n: all UI strings through `apps/web/src/i18n/en.ts` (`t('key')`), no hard-coded English in components.
244 +* Design tokens in `apps/web/src/app/globals.css` (@theme). Editorial, calm, data-dense. No card soup.
245 +* Mobile: bottom navigation (Home · Countries · Compare · Rankings · Search), bottom sheets for filters, sticky compare
246 + controls, no horizontal overflow at 320–430 px.
247 +* Provenance drawer: any value component (`<Metric>`, chart point) opens `<ProvenanceSheet>` with the provenance object.
248 +
249 +## 10. Ports & env
250 +
251 +| Process | Port | Env |
252 +|---|---|---|
253 +| countryatlas-web | 8290 | `API_URL=http://127.0.0.1:8291`, `NEXT_PUBLIC_SITE_URL=https://www.countryatlas.co` |
254 +| countryatlas-api | 8291 (127.0.0.1) | `CA_DATA_DIR=~/countryatlas-data`, `CA_ADMIN_TOKEN`, `FRED_API_KEY` |
255 +| countryatlas-scheduler | — | same as API |
256 +
257 +Secrets (FRED key, admin token) live only in the mld manifest on M1M32 and in local `.env` (git-ignored).
added pyproject.toml +41 −0
@@ -0,0 +1,41 @@
1 +[project]
2 +name = "countryatlas"
3 +version = "0.1.0"
4 +description = "CountryAtlas — global country intelligence platform: data pipeline, connectors and API"
5 +requires-python = ">=3.12"
6 +dependencies = [
7 + "duckdb>=1.3",
8 + "pyarrow>=17",
9 + "polars>=1.10",
10 + "httpx>=0.27",
11 + "pyyaml>=6",
12 + "pydantic>=2.8",
13 + "fastapi>=0.115",
14 + "uvicorn[standard]>=0.30",
15 + "orjson>=3.10",
16 + "typer>=0.12",
17 + "rich>=13",
18 + "python-dateutil>=2.9",
19 + "tenacity>=9",
20 + "numpy>=2",
21 +]
22 +
23 +[project.optional-dependencies]
24 +dev = ["pytest>=8", "pytest-asyncio>=0.24", "ruff>=0.6", "respx>=0.21"]
25 +
26 +[project.scripts]
27 +ca = "countryatlas.cli:app"
28 +
29 +[build-system]
30 +requires = ["hatchling"]
31 +build-backend = "hatchling.build"
32 +
33 +[tool.hatch.build.targets.wheel]
34 +packages = ["src/countryatlas"]
35 +
36 +[tool.ruff]
37 +line-length = 120
38 +target-version = "py312"
39 +
40 +[tool.pytest.ini_options]
41 +testpaths = ["tests"]
added registry/countries.overrides.yaml +23 −0
@@ -0,0 +1,23 @@
1 +# Hand overrides applied on top of the generated registry (keyed by iso3). Regenerate with
2 +# .venv/bin/python scripts/build_country_registry.py
3 +XKX:
4 + status: country # partially recognised state, treated as a country (World Bank member)
5 + official_name: Republic of Kosovo
6 + iso_numeric: null
7 +VIR:
8 + slug: us-virgin-islands
9 +CHI:
10 + short_name: Channel Islands
11 + official_name: Channel Islands (Jersey and Guernsey)
12 + status: territory
13 + continent: Europe
14 + subregion: Northern Europe
15 + currency_code: GBP
16 + currency_name: British pound
17 + area_km2: 194
18 + flag_emoji: "🇯🇪"
19 +PSE:
20 + short_name: Palestine
21 + status: territory
22 +TWN:
23 + status: territory
added registry/countries.yaml +7117 −0
Large diff (7,117 lines) is not loaded by default to keep the page fast.Load diff
added registry/groups.yaml +212 −0
Large diff (212 lines) is not loaded by default to keep the page fast.Load diff
added registry/indicators.yaml +3086 −0
Large diff (3,086 lines) is not loaded by default to keep the page fast.Load diff
added registry/topics.yaml +176 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added scripts/build_country_registry.py +191 −0
Large diff (191 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/api/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/config.py +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/connectors/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/connectors/base.py +203 −0
Large diff (203 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/models.py +121 −0
Large diff (121 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/pipeline/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/registry/__init__.py +368 −0
Large diff (368 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/storage/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added src/countryatlas/storage/schema.sql +104 −0
Large diff (105 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere