SPB Git forge
spboucher.ai ↗

spb/countryatlas

Public
20commits 1branches 0releases
268.3 MBsize
maindefault branch
12 days agolast push
TypeScript 57% Python 38.6% JavaScript 3.6% CSS 0.6%

Analytics API 1.1 + pipeline reliability and change detection 2.0

API: /pulse, /movers, /extremes, /scatter, /trajectory, /finder, /peers, /indicators/{slug}/{related,distribution,frames,quality}, /rankings/{id}/race, /regions/compare, /countries/{id}/{story,quality}, dna?reference=, /updates, search intents (type action); shared per-snapshot cross-section cache; 13 API tests.
Pipeline: structural_break / trend_reversal / volatility_spike detectors, similarity contributions carry value_a/value_b, deterministic retry/backoff honouring Retry-After, validation rules schema_change / impossible_year / null_spike / vintage_shift, ≥90 % observation-count integrity guard, meta.source_health + previous_observation_count, build logs; group descriptions in the registry; docs updated. 138 tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent d588bee

25 changed files +2,856 −52

modified docs/API.md +20 −0
@@ -326,3 +326,23 @@ url, score: 1.0}`: `compare <c1> <c2> [<c3>…]` → `/compare/<slugs>`; `rank[i
326 326 `structural_break` (single mean shift with gain ≥ 0.5 and shift ≥ 1.5 × series sd, segments ≥ 5 years, break within the last 10 years for
327 327 `changes`), `trend_reversal` (three consecutive yearly moves of one sign after three of the opposite sign), `volatility_spike` (sd of the
328 328 last 5 yearly differences ≥ 3 × the sd of the previous 15). Same row shape; `detail` documents the parameters.
329 +
330 +### Implementation notes (as built, 2026-09-12 — deviations from the contract above)
331 +
332 +* `/pulse` takes `min_population` (default 1 000 000, `0` disables) that filters the **top movers only** (`top_up`/`top_down`); counts and
333 + shares use every reporting country. The response echoes `min_population` and `filter_note`. Pools need ≥ 30 reporting countries (not 20)
334 + so a thin indicator (Gini, 25 countries) never headlines; `year_reference` is the latest year of the annual pillars population / GDP /
335 + life expectancy (monthly/quarterly series already carry the current year). Convergence compares max-year vs max-year − 10.
336 +* `/movers` windows 5/10 report kinds `change_5y` / `change_10y` (direction in `direction`), keep moves with robust |z| ≥ 1.5 and rank by
337 + severity = min(1, |z|/4) × importance; `kind=record|reversal|acceleration|structural` are only meaningful for `window=1` and return an
338 + empty list otherwise. Window 1 reads the `changes` table (any kind, including the pipeline's structural_break / trend_reversal /
339 + volatility_spike).
340 +* `/extremes` facets each carry their own `from_year` / `to_year` (the indicator's latest common year and its window start); the top-level
341 + pair is the min/max across facets. Start values use the nearest observation within ±2 years of the window start.
342 +* `/indicators/{slug}/related`: `min_n` accepts ≥ 5 (default 40). Pearson is computed on the association scale (log10 for level series,
343 + flagged `log_x`/`log_y`), Spearman on raw values.
344 +* `/indicators/{slug}/distribution`: `highlight` also returns `year`; group medians need ≥ 2 members with data.
345 +* `/regions/compare` `history[slug]` also carries `kind`; `/updates` sources carry `runs {n, n_failed}` and `snapshot.reference_year`.
346 +* `/countries/{id}/story` text template is `"{Country}: {Indicator} rose|fell from {first} in {y0} to {last} in {y1} ({signed change}[, ±x % a year])."`
347 + (colon form keeps acronyms such as GDP / PPP intact); `/countries/{id}/dna` adds `note`.
348 +* `/search` intents: countries are resolved by ISO2/ISO3 code first, then exact single word, then (near-)exact 2–3-word names, then fuzzy.
modified docs/ARCHITECTURE.md +7 −0
@@ -264,6 +264,13 @@ No table or column of `schema.sql` was changed. The following precisions/deviati
264 264 * **`validation_issues`** are capped at 2 000 rows per rule per spec (5 000 per spec file) to stay small; counts are exact in
265 265 `import_runs.warnings/errors`.
266 266 * `ca sql "<query>"` was added to the CLI (read-only DuckDB); `ca normalize` re-reads the newest raw files per spec.
267 +* **Change detection 2.0 (2026-09-12)**: three more `kind` values in `changes`/`events` — `structural_break`, `trend_reversal`,
268 + `volatility_spike` — computed with `countryatlas.stats` (docs/PIPELINE.md §Build step 6). Same row shape; `detail` records the
269 + parameters (`gain`, `shift_ratio`, `run`, `ratio`…). Similarity `contributions[feature]` gain `value_a` / `value_b` (raw values).
270 +* **Reliability (2026-09-12)**: validation adds `impossible_year` (row quarantined), `schema_change` (dataset quarantined),
271 + `null_spike` and `vintage_shift` (warnings vs the previous staging file); the build refuses to publish a snapshot with < 90 % of
272 + the previous snapshot's observations, records `meta.previous_observation_count` and `meta.source_health`, and writes
273 + `logs/build-<run_id>.json`. HTTP retries are deterministic (2/4/8/16 s, 5 attempts, `Retry-After` honoured).
267 274
268 275 ## 8. API (FastAPI, `/api/v1`, OpenAPI at `/api/v1/openapi.json`, docs at `/api/v1/docs`)
269 276
modified docs/PIPELINE.md +29 −7
@@ -74,9 +74,21 @@ latest period ended more than `stale_after_days` ago; only that latest row) · `
74 74 or non-finite). Nothing is deleted. Quarantined rows stay in `observations` for inspection but are **excluded from every
75 75 derived table** (`latest`, `rankings`, `changes`, `events`, `similarity`, `insights`, `country_dna`).
76 76
77 −Dataset-level quarantine (`quarantine_dataset=True`): duplicates, unit ≠ registry unit, or fewer than 30 % of the rows
78 −of the previous staging file for the same spec. The new file is **not written**, the previous one is kept, and the
79 −`run.json` says `quarantined` with the reason. `ca status` lists these.
77 +Row statuses also cover `impossible_year` (year < 1750 or > current year + 10 → `quarantined`).
78 +
79 +Dataset-level quarantine (`quarantine_dataset=True`): `schema_change` (a required staging column is missing or has an
80 +incompatible dtype), duplicates, unit ≠ registry unit, or fewer than 30 % of the rows of the previous staging file for the
81 +same spec (`partial_download`). The new file is **not written**, the previous one is kept, and the `run.json` says
82 +`quarantined` with the reason. `ca status` lists these.
83 +
84 +Dataset-level **warnings** compared with the previous staging file of the same spec (`PreviousStats`, read by
85 +`fetch.py` before validation): `null_spike` (share of null values > 3 × the previous share and > 5 %) and `vintage_shift`
86 +(median |relative change| over ≥ 20 overlapping keys > 25 % — a source revision or a changed definition). Both are logged,
87 +stored in `validation_issues`, and the file is still written: suspicious values are flagged, never deleted.
88 +
89 +HTTP reliability (`connectors/base.py::Connector.get`): every request goes through one method with 5 attempts and a
90 +deterministic exponential backoff (2, 4, 8, 16 s, capped at 60 s, no jitter) on 408/425/429/5xx and transport errors
91 +(timeouts, resets); a `Retry-After` header (seconds or HTTP-date) is honoured before the backoff, capped at 120 s.
80 92
81 93 Extreme jumps: positive level series (`format` in currency/number/tonnes/kwh/per_*/km/ha with a lower bound ≥ 0) are
82 94 compared on log-differences, others on absolute differences; threshold = `jump_threshold` × 1.4826 × MAD of the
@@ -110,12 +122,22 @@ country's differences, with a floor of 10 % (relative) or 2 % of the series rang
110 122 monotone series and `cumulative`-tagged indicators are silent; severity = detector score × importance weight
111 123 (headline 1.0 / featured 0.9 / other 0.7). Q/M series are annualised first (one detection per year at most); index /
112 124 non-rankable series need a ≥ 10 % move; `events` keep severity ≥ 0.35 and ≤ 3 rows per (country, indicator, year).
125 + **Change detection 2.0** (`countryatlas.stats`): `structural_break` — single mean shift (binary segmentation) with
126 + gain ≥ 0.5 of the total sum of squares, |shift| ≥ 1.5 × sd, ≥ 5 years on each side, non-monotone series; a `change` when
127 + the break is within the last 10 years, always an `event` (one per series); `trend_reversal` — three yearly moves of one
128 + sign after three of the opposite sign, cumulative move ≥ floor (severity 0.55); `volatility_spike` — sd of the last 5
129 + yearly differences ≥ 3 × sd of the previous 15 and ≥ floor (severity 0.45–0.7). Real snapshot (2026-09-12): 3 355 breaks,
130 + 1 799 volatility spikes (mostly life expectancy 2019–2024), 546 reversals in `changes`; 19 593 break events.
113 131 7. `similarity` (5 modes, `registry/similarity.yaml`) and `country_dna` (9 percentile dimensions, `dna:` section of the
114 − same file). 8. `insights` (`registry/insights.yaml`, 16 templates, all numbers computed).
115 −9. `meta` (schema_version, build_run_id, built_at, counts, connectors, duration) → `CHECKPOINT`.
132 + same file). Each contribution carries `z_a`, `z_b`, `weight`, `contribution` **and the raw latest values `value_a` /
133 + `value_b`** (after an optional `per` ratio, before the log transform) so the UI can quote real numbers. 8. `insights` (`registry/insights.yaml`, 16 templates, all numbers computed).
134 +9. `meta` (schema_version, build_run_id, built_at, counts, `previous_observation_count`, `source_health` JSON = per
135 + connector ok / partial / failed / quarantined spec counts from `import_runs`, connectors, duration) → `CHECKPOINT`.
116 136 10. Integrity: ≥ 100 000 observations when World Bank staging exists; every headline indicator that has staging data
117 − has ≥ 100 countries in `latest`. Failure (or any exception) deletes the build file and leaves the live DB untouched.
118 −11. `os.replace` → `atlas.duckdb`, copy to `snapshots/`, prune to `CA_KEEP_SNAPSHOTS`.
137 + has ≥ 100 countries in `latest`; **the new snapshot has ≥ 90 % of the previous snapshot's observations** (a shrunken
138 + build is never published). Failure (or any exception) deletes the build file and leaves the live DB untouched.
139 +11. `os.replace` → `atlas.duckdb`, copy to `snapshots/`, prune to `CA_KEEP_SNAPSHOTS`. Every build (published or failed)
140 + writes `logs/build-<run_id>.json` (counts, warnings, error, staging files used per connector).
119 141
120 142 ## Adding a connector
121 143
modified registry/groups.yaml +12 −0
@@ -71,21 +71,25 @@ groups:
71 71 slug: asia
72 72 name: Asia
73 73 kind: continent
74 + description: Countries and territories on the Asian continent (registry `continent` field; transcontinental states follow their registry assignment).
74 75 members: CONTINENT:Asia
75 76 - id: africa
76 77 slug: africa
77 78 name: Africa
78 79 kind: continent
80 + description: Countries and territories on the African continent.
79 81 members: CONTINENT:Africa
80 82 - id: americas
81 83 slug: americas
82 84 name: Americas
83 85 kind: continent
86 + description: North, Central and South America and the Caribbean.
84 87 members: CONTINENT:Americas
85 88 - id: oceania
86 89 slug: oceania
87 90 name: Oceania
88 91 kind: continent
92 + description: Australia, New Zealand and the Pacific island countries and territories.
89 93 members: CONTINENT:Oceania
90 94
91 95 # ---- Income groups (World Bank FY2026 classification) -------------------------------------------
@@ -101,18 +105,21 @@ groups:
101 105 name: Upper middle income
102 106 kind: income
103 107 wb_code: UMC
108 + description: World Bank income classification (GNI per capita, Atlas method).
104 109 members: WB_INCOME:UMC
105 110 - id: lmc
106 111 slug: lower-middle-income
107 112 name: Lower middle income
108 113 kind: income
109 114 wb_code: LMC
115 + description: World Bank income classification (GNI per capita, Atlas method).
110 116 members: WB_INCOME:LMC
111 117 - id: lic
112 118 slug: low-income
113 119 name: Low income
114 120 kind: income
115 121 wb_code: LIC
122 + description: World Bank income classification (GNI per capita, Atlas method).
116 123 members: WB_INCOME:LIC
117 124
118 125 # ---- Organisations ------------------------------------------------------------------------------
@@ -179,6 +186,7 @@ groups:
179 186 slug: african-union
180 187 name: African Union
181 188 kind: org
189 + description: African Union — the 54 African states in the registry (all UN member states of the continent; the Sahrawi Republic is an AU member but is not a registry country).
182 190 members: CONTINENT:Africa
183 191 - id: opec
184 192 slug: opec
@@ -190,6 +198,7 @@ groups:
190 198 slug: commonwealth
191 199 name: Commonwealth of Nations
192 200 kind: org
201 + description: Commonwealth of Nations — 56 member states (Gabon and Togo joined in 2022).
193 202 members: [ATG, AUS, BHS, BGD, BRB, BLZ, BWA, BRN, CMR, CAN, CYP, DMA, SWZ, FJI, GAB, GMB, GHA, GRD, GUY, IND, JAM, KEN, KIR, LSO,
194 203 MWI, MYS, MDV, MLT, MUS, MOZ, NAM, NRU, NZL, NGA, PAK, PNG, RWA, KNA, LCA, VCT, WSM, SYC, SLE, SGP, SLB, ZAF, LKA,
195 204 TZA, TGO, TON, TTO, TUV, UGA, GBR, VUT, ZMB]
@@ -197,16 +206,19 @@ groups:
197 206 slug: mercosur
198 207 name: Mercosur
199 208 kind: org
209 + description: Southern Common Market — full members Argentina, Brazil, Paraguay, Uruguay and Bolivia (full member since 2024); Venezuela is suspended since 2016 and excluded.
200 210 members: [ARG, BRA, PRY, URY, BOL]
201 211 - id: gcc
202 212 slug: gulf-cooperation-council
203 213 name: Gulf Cooperation Council
204 214 kind: org
215 + description: Gulf Cooperation Council — Bahrain, Kuwait, Oman, Qatar, Saudi Arabia and the United Arab Emirates.
205 216 members: [BHR, KWT, OMN, QAT, SAU, ARE]
206 217 - id: small-island-developing-states
207 218 slug: small-island-developing-states
208 219 name: Small island developing states
209 220 kind: org
210 221 wb_code: SST
222 + description: UN list of Small Island Developing States present in the registry (World Bank aggregate code SST).
211 223 members: [ATG, BHS, BRB, BLZ, CPV, COM, CUB, DMA, DOM, FJI, GRD, GNB, GUY, HTI, JAM, KIR, MDV, MHL, MUS, FSM, NRU, PLW, PNG,
212 224 WSM, STP, SYC, SGP, SLB, KNA, LCA, VCT, SUR, TLS, TON, TTO, TUV, VUT]
modified src/countryatlas/api/main.py +8 −3
@@ -26,6 +26,7 @@ from countryatlas.api.errors import Problem, problem_body
26 26 from countryatlas.api.ratelimit import TokenBucketLimiter, client_ip
27 27 from countryatlas.api.routers import (
28 28 admin,
29 + analytics,
29 30 changes,
30 31 compare,
31 32 countries,
@@ -34,6 +35,7 @@ from countryatlas.api.routers import (
34 35 home,
35 36 indicators,
36 37 methodology,
38 + quality,
37 39 rankings,
38 40 regions,
39 41 search,
@@ -79,7 +81,7 @@ def create_app(db_path: str | Path | None = None, *, rate_limit_per_minute: int
79 81 app = FastAPI(
80 82 lifespan=lifespan,
81 83 title="CountryAtlas API",
82 − version="1.0.0",
84 + version="1.1.0",
83 85 summary="Country statistics with provenance on every value — www.countryatlas.co",
84 86 description=(
85 87 "Public data API of CountryAtlas. Countries (ISO3 or slug), indicators (slug), rankings, comparisons, groups, "
@@ -176,8 +178,11 @@ def create_app(db_path: str | Path | None = None, *, rate_limit_per_minute: int
176 178 return _problem(request, 500, "Internal server error", f"{type(exc).__name__}: {exc}")
177 179
178 180 # ------------------------------------------------------------------ routers
179 − for r in (health.router, countries.router, indicators.router, series.router, rankings.router, compare.router, regions.router,
180 − search.router, home.router, changes.router, sources.router, methodology.router, download.router, admin.router):
181 + # quality before countries/indicators: `/indicators/{slug}/quality` and `/countries/{id}/quality` are fixed-suffix routes that must
182 + # win over the `/{id}/{topic}`-style paths of the generic routers.
183 + for r in (health.router, quality.router, countries.router, indicators.router, series.router, rankings.router, compare.router,
184 + regions.router, search.router, home.router, changes.router, sources.router, methodology.router, download.router,
185 + analytics.router, admin.router):
181 186 app.include_router(r, prefix=API_PREFIX)
182 187
183 188 @app.get("/", include_in_schema=False)
added src/countryatlas/api/routers/analytics.py +887 −0
@@ -0,0 +1,887 @@
1 +"""Analytics endpoints (API 1.1): /pulse, /movers, /extremes, /scatter, /trajectory, /finder, /peers.
2 +
3 +All deterministic, country-only (kind='country'), canonical annual frequency, non-forecast, non-quarantined. Statistics come
4 +from countryatlas.stats; per-snapshot precomputations (the latest cross-section matrix) live in `snap._cached`. Wording is
5 +neutral: increase / decrease — "improvement" / "deterioration" only when the indicator declares `higher_is_better`.
6 +"""
7 +from __future__ import annotations
8 +
9 +import math
10 +from typing import Any
11 +
12 +import numpy as np
13 +from fastapi import APIRouter, Depends, Query
14 +
15 +from countryatlas import stats
16 +from countryatlas.api import schemas
17 +from countryatlas.api.common import (
18 + clean_float,
19 + country_card,
20 + indicator_card,
21 + merged_indicator,
22 + meta_block,
23 + parse_csv,
24 + resolve_country,
25 + resolve_group,
26 + resolve_indicator,
27 +)
28 +from countryatlas.api.db import Snapshot, get_snapshot
29 +from countryatlas.api.errors import bad_request
30 +from countryatlas.api.formatting import format_change, format_value
31 +from countryatlas.api.provenance import build_provenance
32 +from countryatlas.registry import topics as registry_topics
33 +
34 +router = APIRouter(tags=["analytics"])
35 +
36 +NEAREST_YEARS = 3
37 +LEVEL_FORMATS = {"currency", "number", "tonnes", "kwh", "per_1000", "per_100k", "per_million", "km", "ha"}
38 +CATEGORY_TOPICS: dict[str, list[str]] = {
39 + "economic": ["economy", "government", "trade", "income"],
40 + "demographic": ["population"],
41 + "health": ["health"],
42 + "energy": ["energy"],
43 + "climate": ["climate", "environment"],
44 + "digital": ["digital", "innovation"],
45 + "housing": ["housing"],
46 + "labor": ["labor"],
47 +}
48 +CATEGORIES = ["all", *CATEGORY_TOPICS]
49 +KIND_FILTERS = ["all", "improvement", "deterioration", "increase", "decrease", "record", "reversal", "acceleration", "structural"]
50 +UP_KINDS = {"yoy_jump", "record_high", "n_year_high", "accelerating", "change_5y_up", "change_10y_up"}
51 +DOWN_KINDS = {"yoy_drop", "record_low", "n_year_low", "decelerating", "change_5y_down", "change_10y_down"}
52 +KIND_GROUPS: dict[str, set[str]] = {
53 + "increase": {"yoy_jump", "record_high", "n_year_high", "accelerating"},
54 + "decrease": {"yoy_drop", "record_low", "n_year_low", "decelerating"},
55 + "record": {"record_high", "record_low"},
56 + "reversal": {"sign_flip", "trend_reversal"},
57 + "acceleration": {"accelerating", "decelerating"},
58 + "structural": {"structural_break", "volatility_spike"},
59 +}
60 +NOTE_CORRELATION = "Correlation does not imply causation. These are descriptive cross-sectional statistics."
61 +PEER_PAIRS = [
62 + {"x": "gdp-per-capita-ppp", "y": "life-expectancy", "label": "Life expectancy given income"},
63 + {"x": "gdp-per-capita-ppp", "y": "expected-years-of-schooling", "label": "Expected schooling given income"},
64 + {"x": "gdp-per-capita-ppp", "y": "co2-per-capita", "label": "CO₂ per capita given income"},
65 + {"x": "health-expenditure-per-capita", "y": "life-expectancy", "label": "Life expectancy given health spending"},
66 + {"x": "gdp-per-capita-ppp", "y": "internet-users", "label": "Internet adoption given income"},
67 + {"x": "gdp-per-capita-ppp", "y": "infant-mortality-rate", "label": "Infant mortality given income"},
68 +]
69 +
70 +
71 +# ------------------------------------------------------------------------------------------------ shared helpers
72 +def country_ids(snap: Snapshot) -> list[str]:
73 + """ISO3 of kind='country' rows, stable order (by id)."""
74 + return snap._cached("country_ids", lambda: sorted(c["id"] for c in snap.countries().values() if (c.get("kind") or "country") == "country"))
75 +
76 +
77 +def point_country(c: dict[str, Any]) -> dict[str, Any]:
78 + return {"id": c["id"], "slug": c.get("slug"), "name": c.get("short_name"), "flag": c.get("flag_emoji"), "region": c.get("region_wb"),
79 + "income": c.get("income_group")}
80 +
81 +
82 +def importance(ind: dict[str, Any], headline: set[str]) -> float:
83 + if ind["id"] in headline:
84 + return 1.0
85 + return 0.9 if ind.get("featured") else 0.7
86 +
87 +
88 +def headline_ids(snap: Snapshot) -> set[str]:
89 + return set(registry_topics()["headline"])
90 +
91 +
92 +def featured_ids(snap: Snapshot) -> list[str]:
93 + """Headline first, then featured (registry order), deduplicated, present in the snapshot."""
94 + out: list[str] = []
95 + for s in registry_topics()["headline"]:
96 + if s in snap.indicators() and s not in out:
97 + out.append(s)
98 + for i in snap.indicators().values():
99 + m = merged_indicator(i)
100 + if m.get("featured") and m["id"] not in out:
101 + out.append(m["id"])
102 + return out
103 +
104 +
105 +def is_level(ind: dict[str, Any]) -> bool:
106 + """Positive level series (GDP, population, emissions…): analysed on relative changes / log scale."""
107 + fmt = (ind.get("format") or "number").lower()
108 + lo = ind.get("bounds_min")
109 + return fmt in LEVEL_FORMATS and lo is not None and float(lo) >= 0
110 +
111 +
112 +def delta_mode(ind: dict[str, Any]) -> str:
113 + """'points' for percent-like, 'pct' for positive level series, 'abs' otherwise."""
114 + fmt = (ind.get("format") or "").lower()
115 + if fmt == "percent" or (ind.get("unit") or "").startswith("%"):
116 + return "points"
117 + if is_level(ind):
118 + return "pct"
119 + return "abs"
120 +
121 +
122 +def auto_log(ind: dict[str, Any], values: np.ndarray | None = None) -> bool:
123 + if not is_level(ind):
124 + return False
125 + if values is None:
126 + return True
127 + v = values[np.isfinite(values)]
128 + v = v[v > 0]
129 + return bool(len(v) >= 3 and v.max() / v.min() > 50)
130 +
131 +
132 +def interpretation(ind: dict[str, Any], direction: str) -> str | None:
133 + hib = ind.get("higher_is_better")
134 + if hib is None:
135 + return None
136 + return "improvement" if (direction == "up") == bool(hib) else "deterioration"
137 +
138 +
139 +class CrossSection:
140 + """Latest cross-section: countries × indicators value/year matrices (from `latest`, countries only)."""
141 +
142 + def __init__(self, snap: Snapshot) -> None:
143 + self.countries = country_ids(snap)
144 + self.idx = {c: i for i, c in enumerate(self.countries)}
145 + self.indicators = sorted(snap.indicators())
146 + self.iidx = {i: k for k, i in enumerate(self.indicators)}
147 + n, m = len(self.countries), len(self.indicators)
148 + self.values = np.full((n, m), np.nan)
149 + self.years = np.full((n, m), np.nan)
150 + rows = snap.query_rows(
151 + "SELECT l.country_id, l.indicator_id, l.value, l.year FROM latest l JOIN countries c ON c.id = l.country_id "
152 + "WHERE coalesce(c.kind, 'country') = 'country' AND l.value IS NOT NULL"
153 + )
154 + for cid, iid, v, y in rows:
155 + i, j = self.idx.get(cid), self.iidx.get(iid)
156 + if i is None or j is None:
157 + continue
158 + self.values[i, j] = float(v)
159 + self.years[i, j] = int(y)
160 + with np.errstate(invalid="ignore"):
161 + self.max_year = np.nanmax(self.years, axis=0)
162 +
163 + def col(self, indicator_id: str, within: int | None = NEAREST_YEARS) -> tuple[np.ndarray, np.ndarray]:
164 + """(values, years) for one indicator; values older than `within` years of the indicator's max year → nan."""
165 + j = self.iidx.get(indicator_id)
166 + if j is None:
167 + return np.full(len(self.countries), np.nan), np.full(len(self.countries), np.nan)
168 + v, y = self.values[:, j].copy(), self.years[:, j].copy()
169 + if within is not None and np.isfinite(self.max_year[j]):
170 + stale = y < self.max_year[j] - within
171 + v[stale] = np.nan
172 + y[stale] = np.nan
173 + return v, y
174 +
175 +
176 +def cross_section(snap: Snapshot) -> CrossSection:
177 + return snap._cached("cross_section", lambda: CrossSection(snap))
178 +
179 +
180 +def indicator_provenances(snap: Snapshot) -> dict[str, dict[str, Any]]:
181 + """indicator_id → provenance of its dominant source (most rows), cached per snapshot."""
182 +
183 + def load() -> dict[str, dict[str, Any]]:
184 + rows = snap.query(
185 + """SELECT indicator_id, source_id, source_dataset, source_series_code, max(retrieved_at) AS ra, max(source_updated_at) AS su,
186 + count(*) AS n
187 + FROM observations WHERE NOT is_forecast GROUP BY 1, 2, 3, 4 QUALIFY row_number() OVER (PARTITION BY indicator_id ORDER BY n DESC) = 1"""
188 + )
189 + return {r["indicator_id"]: build_provenance(snap, r["indicator_id"], r["source_id"], r["source_dataset"], r["source_series_code"],
190 + r["ra"], r["su"]) for r in rows}
191 +
192 + return snap._cached("indicator_provenances", load)
193 +
194 +
195 +def population_latest(snap: Snapshot) -> dict[str, float]:
196 + return snap._cached("population_latest", lambda: {r[0]: float(r[1]) for r in snap.query_rows(
197 + "SELECT country_id, value FROM latest WHERE indicator_id = 'population' AND value IS NOT NULL")})
198 +
199 +
200 +def filter_note(min_pop: int | None) -> str | None:
201 + if not min_pop:
202 + return None
203 + return f"Countries above {min_pop / 1_000_000:g}M inhabitants"
204 +
205 +
206 +def latest_common_year(snap: Snapshot, indicator_id: str, min_countries: int = 50) -> int | None:
207 + from countryatlas.api.routers.indicators import latest_common_year as _lcy
208 +
209 + y, _ = _lcy(snap, indicator_id, min_countries)
210 + return y
211 +
212 +
213 +def group_members_or_all(snap: Snapshot, group: str) -> tuple[dict[str, Any], list[str] | None]:
214 + g = resolve_group(snap, group)
215 + if g["id"] == "world":
216 + return g, None
217 + return g, snap.group_members(g["id"])
218 +
219 +
220 +def _mover_lite(snap: Snapshot, cid: str, ind: dict[str, Any], value: float | None, ref: float | None, year: int | None) -> dict[str, Any]:
221 + delta = None if value is None or ref is None else value - ref
222 + return {"country": country_card(snap.countries()[cid]), "value": value, "ref_value": ref, "delta": delta,
223 + "delta_pct": stats.yoy(value, ref), "formatted": format_value(value, ind), "year": year}
224 +
225 +
226 +def _pct_txt(v: float) -> str:
227 + return f"{v:.0f} %"
228 +
229 +
230 +# ------------------------------------------------------------------------------------------------ /pulse
231 +PULSE_MIN_COUNTRIES = 30
232 +
233 +
234 +@router.get("/pulse", response_model=schemas.PulseResponse, summary="World Pulse — what is changing globally (latest year vs previous)")
235 +def pulse(min_population: int | None = Query(1_000_000, ge=0, description="Population floor for the top movers only (counts use every country)"),
236 + snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
237 + ids = featured_ids(snap)
238 + pop = population_latest(snap)
239 + ph = ",".join("?" * len(ids))
240 + rows = snap.query(
241 + f"""SELECT l.indicator_id, l.country_id, l.value, l.prev_value, l.change_abs, l.change_pct, l.year
242 + FROM latest l JOIN countries c ON c.id = l.country_id
243 + WHERE coalesce(c.kind, 'country') = 'country' AND l.indicator_id IN ({ph}) AND l.value IS NOT NULL""",
244 + ids,
245 + )
246 + max_years = {r[0]: int(r[1]) for r in snap.query_rows(f"SELECT indicator_id, max(year) FROM latest WHERE indicator_id IN ({ph}) GROUP BY 1", ids)}
247 + # reference = latest year of the annual headline pillars (monthly/quarterly series may already carry the current year)
248 + year_reference = max([max_years.get(k, 0) for k in ("population", "gdp", "life-expectancy")] or [0]) or (max(max_years.values()) if max_years else 0)
249 + records = {(r[0], r[1]): int(r[2]) for r in snap.query_rows(
250 + f"""SELECT ch.indicator_id, ch.kind, count(*) FROM changes ch JOIN countries c ON c.id = ch.country_id
251 + WHERE coalesce(c.kind, 'country') = 'country' AND ch.kind IN ('record_high', 'record_low') AND ch.indicator_id IN ({ph})
252 + GROUP BY 1, 2""", ids)}
253 + n_changes = int(snap.scalar("SELECT count(*) FROM changes ch JOIN countries c ON c.id = ch.country_id WHERE coalesce(c.kind,'country') = 'country'") or 0)
254 + by_ind: dict[str, list[dict[str, Any]]] = {}
255 + for r in rows:
256 + by_ind.setdefault(r["indicator_id"], []).append(r)
257 + # convergence inputs: values 10 years before the indicator max year and at the max year
258 + conv_rows = snap.query(
259 + f"""SELECT o.indicator_id, o.country_id, o.year, o.value FROM observations o JOIN countries c ON c.id = o.country_id
260 + JOIN indicators i ON i.id = o.indicator_id
261 + WHERE o.indicator_id IN ({ph}) AND o.frequency = i.frequency AND NOT o.is_forecast AND o.value IS NOT NULL
262 + AND coalesce(o.status, 'imported') <> 'quarantined' AND coalesce(c.kind, 'country') = 'country'
263 + AND o.year IN (SELECT unnest([y, y - 10]) FROM (SELECT indicator_id AS ii, max(year) AS y FROM latest GROUP BY 1) WHERE ii = o.indicator_id)""",
264 + ids,
265 + )
266 + conv: dict[str, dict[str, dict[int, float]]] = {}
267 + for r in conv_rows:
268 + conv.setdefault(r["indicator_id"], {}).setdefault(r["country_id"], {})[int(r["year"])] = float(r["value"])
269 + provs = indicator_provenances(snap)
270 + items = []
271 + reporting: set[str] = set()
272 + for iid in ids:
273 + ind = merged_indicator(snap.indicators()[iid])
274 + my = max_years.get(iid)
275 + if my is None or my < year_reference - 1:
276 + continue
277 + recent = [r for r in by_ind.get(iid, []) if r["year"] is not None and int(r["year"]) >= my - 1 and r.get("change_abs") is not None]
278 + if len(recent) < PULSE_MIN_COUNTRIES:
279 + continue
280 + for r in recent:
281 + reporting.add(r["country_id"])
282 + big = np.array([not min_population or pop.get(r["country_id"], 0) >= min_population for r in recent])
283 + if big.sum() < 20:
284 + big = np.ones(len(recent), dtype=bool)
285 + mode = delta_mode(ind)
286 + key = "change_pct" if mode == "pct" else "change_abs"
287 + deltas = np.array([clean_float(r[key]) if clean_float(r[key]) is not None else np.nan for r in recent])
288 + absd = np.array([clean_float(r["change_abs"]) or 0.0 for r in recent])
289 + n = len(recent)
290 + n_up = int(np.sum(absd > 0))
291 + n_down = int(np.sum(absd < 0))
292 + n_flat = n - n_up - n_down
293 + share_up = round(100.0 * n_up / n, 1)
294 + share_down = round(100.0 * n_down / n, 1)
295 + hib = ind.get("higher_is_better")
296 + sem = "neutral" if hib is None else ("higher_is_better" if hib else "lower_is_better")
297 + dd = np.where(big, deltas, np.nan)
298 + order = np.argsort(np.nan_to_num(dd, nan=-np.inf))
299 + top_up = recent[int(order[-1])] if np.isfinite(dd[order[-1]]) and dd[order[-1]] > 0 else None
300 + order_lo = np.argsort(np.nan_to_num(dd, nan=np.inf))
301 + top_down = recent[int(order_lo[0])] if np.isfinite(dd[order_lo[0]]) and dd[order_lo[0]] < 0 else None
302 + rh, rl = records.get((iid, "record_high"), 0), records.get((iid, "record_low"), 0)
303 + name = ind.get("short_name") or ind.get("name")
304 + # headline template
305 + if rh >= 20 and rh >= n * 0.2 and share_up >= 50:
306 + headline = f"{name} hit a record high in {rh} countries"
307 + elif rl >= 20 and rl >= n * 0.2 and share_down >= 50:
308 + headline = f"{name} fell to a record low in {rl} countries"
309 + elif share_down >= 60:
310 + headline = f"{name} fell in {_pct_txt(share_down)} of {n} reporting countries"
311 + elif share_up >= 60:
312 + headline = f"{name} rose in {_pct_txt(share_up)} of {n} reporting countries"
313 + elif iid == "population" and n_down:
314 + headline = f"Population is shrinking in {n_down} countries"
315 + else:
316 + headline = f"{name} rose in {n_up} and fell in {n_down} of {n} reporting countries"
317 + # convergence over 10 years
318 + cv = None
319 + cd = conv.get(iid, {})
320 + s0 = [cd[c].get(my - 10) for c in cd]
321 + s1 = [cd[c].get(my) for c in cd]
322 + pairs = [(a, b) for a, b in zip(s0, s1, strict=True) if a is not None and b is not None]
323 + if len(pairs) >= 10:
324 + c = stats.convergence([p[0] for p in pairs], [p[1] for p in pairs])
325 + if c:
326 + cv = {"direction": c["direction"], "cv_start": c["cv_start"], "cv_end": c["cv_end"], "n": c["n"], "from_year": my - 10}
327 + items.append({
328 + "indicator": indicator_card(ind), "year": my, "n": n, "n_up": n_up, "n_down": n_down, "n_flat": n_flat,
329 + "share_up": share_up, "share_down": share_down,
330 + "median_change_abs": clean_float(np.nanmedian(absd)) if n else None,
331 + "median_change_pct": clean_float(np.nanmedian([clean_float(r["change_pct"]) if clean_float(r["change_pct"]) is not None else np.nan for r in recent])),
332 + "direction_semantics": sem, "record_highs": rh, "record_lows": rl, "headline": headline,
333 + "top_up": _mover_lite(snap, top_up["country_id"], ind, clean_float(top_up["value"]), clean_float(top_up["prev_value"]), top_up["year"]) if top_up else None,
334 + "top_down": _mover_lite(snap, top_down["country_id"], ind, clean_float(top_down["value"]), clean_float(top_down["prev_value"]), top_down["year"]) if top_down else None,
335 + "convergence": cv, "provenance": provs.get(iid),
336 + })
337 + return {
338 + "meta": meta_block(snap), "year_reference": year_reference, "min_population": min_population, "filter_note": filter_note(min_population),
339 + "summary": {"n_indicators": len(items), "n_countries_reporting": len(reporting),
340 + "n_record_highs": sum(i["record_highs"] for i in items), "n_record_lows": sum(i["record_lows"] for i in items),
341 + "n_changes": n_changes},
342 + "items": items,
343 + }
344 +
345 +
346 +# ------------------------------------------------------------------------------------------------ /movers
347 +def _mover_item(snap: Snapshot, r: dict[str, Any], ind: dict[str, Any], kind: str, headline: str | None, severity: float | None,
348 + year: int | None, ref_year: int | None, prov: dict[str, Any] | None) -> dict[str, Any]:
349 + value, ref = clean_float(r.get("value")), clean_float(r.get("ref_value"))
350 + delta = None if value is None or ref is None else value - ref
351 + direction = "up" if (delta or 0) >= 0 else "down"
352 + if kind in UP_KINDS:
353 + direction = "up"
354 + elif kind in DOWN_KINDS:
355 + direction = "down"
356 + return {
357 + "country": country_card(snap.countries()[r["country_id"]]), "indicator": indicator_card(ind), "kind": kind, "year": year,
358 + "ref_year": ref_year, "value": value, "ref_value": ref, "delta": delta, "delta_pct": stats.yoy(value, ref),
359 + "formatted": format_value(value, ind), "formatted_ref": format_value(ref, ind), "severity": clean_float(severity),
360 + "direction": direction, "interpretation": interpretation(ind, direction), "headline": headline, "provenance": prov,
361 + }
362 +
363 +
364 +def _kind_matches(kind: str, ind: dict[str, Any], flt: str, direction: str) -> bool:
365 + if flt == "all":
366 + return True
367 + if flt in ("improvement", "deterioration"):
368 + return interpretation(ind, direction) == flt
369 + if flt == "increase":
370 + return direction == "up" and kind not in KIND_GROUPS["structural"] | KIND_GROUPS["reversal"]
371 + if flt == "decrease":
372 + return direction == "down" and kind not in KIND_GROUPS["structural"] | KIND_GROUPS["reversal"]
373 + return kind in KIND_GROUPS.get(flt, set())
374 +
375 +
376 +@router.get("/movers", response_model=schemas.MoversResponse, summary="Biggest movers by window / category / kind")
377 +def movers(
378 + window: int = Query(1, description="1 (latest detections), 5 or 10 years"),
379 + category: str = Query("all"),
380 + kind: str = Query("all"),
381 + limit: int = Query(40, ge=1, le=300),
382 + min_population: int | None = Query(None, ge=0),
383 + snap: Snapshot = Depends(get_snapshot),
384 +) -> dict[str, Any]:
385 + if window not in (1, 5, 10):
386 + raise bad_request("window must be 1, 5 or 10.")
387 + if category not in CATEGORIES:
388 + raise bad_request(f"category must be one of {', '.join(CATEGORIES)}.")
389 + if kind not in KIND_FILTERS:
390 + raise bad_request(f"kind must be one of {', '.join(KIND_FILTERS)}.")
391 + topics = CATEGORY_TOPICS.get(category)
392 + pop = population_latest(snap)
393 + provs = indicator_provenances(snap)
394 + hl = headline_ids(snap)
395 + items: list[dict[str, Any]] = []
396 + if window == 1:
397 + where = ["coalesce(c.kind, 'country') = 'country'"]
398 + params: list[Any] = []
399 + if topics:
400 + where.append(f"i.topic IN ({','.join('?' * len(topics))})")
401 + params += topics
402 + rows = snap.query(
403 + f"""SELECT ch.country_id, ch.indicator_id, ch.kind, ch.year, ch.value, ch.ref_value, ch.severity, ch.headline, ch.window_years
404 + FROM changes ch JOIN countries c ON c.id = ch.country_id JOIN indicators i ON i.id = ch.indicator_id
405 + WHERE {' AND '.join(where)} ORDER BY ch.severity DESC NULLS LAST, ch.year DESC, ch.country_id LIMIT 4000""",
406 + params,
407 + )
408 + for r in rows:
409 + if min_population and pop.get(r["country_id"], 0) < min_population:
410 + continue
411 + ind = merged_indicator(snap.indicators()[r["indicator_id"]])
412 + direction = "up" if r["kind"] in UP_KINDS else "down" if r["kind"] in DOWN_KINDS else ("up" if (clean_float(r["value"]) or 0) >= (clean_float(r["ref_value"]) or 0) else "down")
413 + if not _kind_matches(r["kind"], ind, kind, direction):
414 + continue
415 + ref_year = (int(r["year"]) - 1) if r["kind"] in ("yoy_jump", "yoy_drop", "sign_flip") else None
416 + items.append(_mover_item(snap, r, ind, r["kind"], r["headline"], r["severity"], int(r["year"]) if r["year"] is not None else None,
417 + ref_year, provs.get(r["indicator_id"])))
418 + if len(items) >= limit:
419 + break
420 + else:
421 + if kind in ("record", "reversal", "acceleration", "structural"):
422 + return {"meta": meta_block(snap), "window": window, "category": category, "kind": kind, "min_population": min_population,
423 + "filter_note": filter_note(min_population), "categories": CATEGORIES, "kinds": KIND_FILTERS, "n": 0, "items": []}
424 + where = ["coalesce(c.kind, 'country') = 'country'", "l.value IS NOT NULL", "o.value IS NOT NULL", "l.year >= (SELECT max(year) - 2 FROM latest)"]
425 + params = [window]
426 + if topics:
427 + where.append(f"i.topic IN ({','.join('?' * len(topics))})")
428 + params += topics
429 + rows = snap.query(
430 + f"""SELECT l.indicator_id, l.country_id, l.year, l.value, o.value AS ref_value, o.year AS ref_year
431 + FROM latest l JOIN countries c ON c.id = l.country_id JOIN indicators i ON i.id = l.indicator_id
432 + JOIN observations o ON o.country_id = l.country_id AND o.indicator_id = l.indicator_id AND o.frequency = l.frequency
433 + AND o.year = l.year - ? AND NOT o.is_forecast AND coalesce(o.status, 'imported') <> 'quarantined'
434 + WHERE {' AND '.join(where)}""",
435 + params,
436 + )
437 + by_ind: dict[str, list[dict[str, Any]]] = {}
438 + for r in rows:
439 + by_ind.setdefault(r["indicator_id"], []).append(r)
440 + for iid, rs in by_ind.items():
441 + ind = merged_indicator(snap.indicators()[iid])
442 + if not ind.get("ranking_eligible", True) or len(rs) < 20:
443 + continue
444 + mode = delta_mode(ind)
445 + v = np.array([float(r["value"]) for r in rs])
446 + ref = np.array([float(r["ref_value"]) for r in rs])
447 + with np.errstate(divide="ignore", invalid="ignore"):
448 + d = np.where((v > 0) & (ref > 0), np.log(v) - np.log(ref), np.nan) if mode == "pct" else v - ref
449 + z = stats.robust_zscore(d.tolist())
450 + w = importance(ind, hl)
451 + for r, zi, di in zip(rs, z, d, strict=True):
452 + if zi is None or not np.isfinite(di) or abs(zi) < 1.5:
453 + continue
454 + if min_population and pop.get(r["country_id"], 0) < min_population:
455 + continue
456 + direction = "up" if di > 0 else "down"
457 + k = f"change_{window}y"
458 + if not _kind_matches(k, ind, kind, direction):
459 + continue
460 + verb = "rose" if direction == "up" else "fell"
461 + ch = format_change(float(r["value"]) - float(r["ref_value"]), stats.yoy(float(r["value"]), float(r["ref_value"])), ind)
462 + headline = (f"{ind.get('short_name') or ind.get('name')} {verb} {ch.replace('+', '').replace('−', '') if ch else ''} in "
463 + f"{snap.countries()[r['country_id']]['short_name']} between {int(r['ref_year'])} and {int(r['year'])} "
464 + f"(from {format_value(float(r['ref_value']), ind)} to {format_value(float(r['value']), ind)}).")
465 + items.append(_mover_item(snap, r, ind, k, headline, min(1.0, abs(zi) / 4.0) * w, int(r["year"]), int(r["ref_year"]), provs.get(iid)))
466 + items.sort(key=lambda x: -(x["severity"] or 0))
467 + items = items[:limit]
468 + return {"meta": meta_block(snap), "window": window, "category": category, "kind": kind, "min_population": min_population,
469 + "filter_note": filter_note(min_population), "categories": CATEGORIES, "kinds": KIND_FILTERS, "n": len(items), "items": items}
470 +
471 +
472 +# ------------------------------------------------------------------------------------------------ /extremes
473 +FACETS: list[tuple[str, str, str, str, str]] = [ # (id, title, indicator, direction, metric)
474 + ("aging", "Most rapidly ageing", "median-age", "up", "abs"),
475 + ("urbanizing", "Fastest urbanising", "urban-population-share", "up", "points"),
476 + ("fertility-decline", "Largest fertility declines", "fertility-rate", "down", "abs"),
477 + ("life-expectancy-gains", "Biggest gains in life expectancy", "life-expectancy", "up", "abs"),
478 + ("gdp-transformations", "Largest GDP per capita transformations", "gdp-per-capita-ppp", "up", "pct"),
479 + ("digital-adoption", "Fastest digital adoption", "internet-users", "up", "points"),
480 + ("renewable-transitions", "Largest renewable transitions", "renewable-electricity-share", "up", "points"),
481 + ("co2-reductions", "Largest CO₂ per capita reductions", "co2-per-capita", "down", "pct"),
482 + ("population-decline", "Fastest shrinking populations", "population", "down", "pct"),
483 + ("population-boom", "Fastest growing populations", "population", "up", "pct"),
484 + ("inflation-surges", "Largest inflation surges", "inflation", "up", "points"),
485 + ("debt-buildup", "Largest public debt build-ups", "general-government-gross-debt-pct-gdp", "up", "points"),
486 + ("unemployment-falls", "Largest falls in unemployment", "unemployment-rate", "down", "points"),
487 +]
488 +EXTREME_WINDOWS = {"1": 1, "5": 5, "10": 10, "25": 25, "since1990": None}
489 +
490 +
491 +@router.get("/extremes", response_model=schemas.ExtremesResponse, summary="Curated extremes facets over a window")
492 +def extremes(
493 + window: str = Query("10", description="1 | 5 | 10 | 25 | since1990"),
494 + topic: str | None = Query(None),
495 + min_population: int | None = Query(1_000_000, ge=0),
496 + rows: int = Query(10, ge=3, le=50),
497 + snap: Snapshot = Depends(get_snapshot),
498 +) -> dict[str, Any]:
499 + if window not in EXTREME_WINDOWS:
500 + raise bad_request("window must be 1, 5, 10, 25 or since1990.")
501 + n_years = EXTREME_WINDOWS[window]
502 + pop = population_latest(snap)
503 + provs = indicator_provenances(snap)
504 + facets = []
505 + from_years: list[int] = []
506 + to_years: list[int] = []
507 + for fid, title, iid, direction, metric in FACETS:
508 + ind_row = snap.indicators().get(iid)
509 + if ind_row is None:
510 + continue
511 + ind = merged_indicator(ind_row)
512 + if topic and ind.get("topic") != topic:
513 + continue
514 + to_year = latest_common_year(snap, iid)
515 + if to_year is None:
516 + continue
517 + from_year = 1990 if n_years is None else to_year - n_years
518 + if from_year >= to_year:
519 + continue
520 + rs = snap.query(
521 + """WITH o AS (
522 + SELECT o.country_id, o.year, o.value FROM observations o JOIN countries c ON c.id = o.country_id JOIN indicators i ON i.id = o.indicator_id
523 + WHERE o.indicator_id = ? AND o.frequency = i.frequency AND NOT o.is_forecast AND o.value IS NOT NULL
524 + AND coalesce(o.status, 'imported') <> 'quarantined' AND coalesce(c.kind, 'country') = 'country'),
525 + s AS (SELECT country_id, arg_min(value, abs(year - ?)) AS v0, arg_min(year, abs(year - ?)) AS y0 FROM o WHERE abs(year - ?) <= 2 GROUP BY 1),
526 + e AS (SELECT country_id, arg_max(value, year) AS v1, max(year) AS y1 FROM o WHERE year BETWEEN ? AND ? GROUP BY 1)
527 + SELECT s.country_id, s.v0, s.y0, e.v1, e.y1 FROM s JOIN e ON e.country_id = s.country_id WHERE e.y1 > s.y0""",
528 + [iid, from_year, from_year, from_year, to_year - 2, to_year],
529 + )
530 + cand = []
531 + for r in rs:
532 + if min_population and pop.get(r["country_id"], 0) < min_population:
533 + continue
534 + v0, v1 = float(r["v0"]), float(r["v1"])
535 + delta = v1 - v0
536 + pct = stats.yoy(v1, v0)
537 + score = pct if metric == "pct" else delta
538 + if score is None or not math.isfinite(score):
539 + continue
540 + if (direction == "up" and score <= 0) or (direction == "down" and score >= 0):
541 + continue
542 + cand.append((abs(score), {
543 + "country": country_card(snap.countries()[r["country_id"]]), "value_start": v0, "value_end": v1,
544 + "year_start": int(r["y0"]), "year_end": int(r["y1"]), "delta": delta, "delta_pct": pct,
545 + "formatted_start": format_value(v0, ind), "formatted_end": format_value(v1, ind)}))
546 + if len(rs) < 30:
547 + continue
548 + cand.sort(key=lambda x: -x[0])
549 + from_years.append(from_year)
550 + to_years.append(to_year)
551 + facets.append({"id": fid, "title": title, "indicator": indicator_card(ind), "direction": direction, "metric": metric,
552 + "rows": [c[1] for c in cand[:rows]], "n": len(cand), "from_year": from_year, "to_year": to_year, "provenance": provs.get(iid)})
553 + return {"meta": meta_block(snap), "window": window, "from_year": min(from_years) if from_years else None,
554 + "to_year": max(to_years) if to_years else None, "min_population": min_population, "filter_note": filter_note(min_population),
555 + "facets": facets}
556 +
557 +
558 +# ------------------------------------------------------------------------------------------------ /scatter
559 +def _nearest_values(snap: Snapshot, indicator_id: str, year: int, members: list[str] | None, within: int = NEAREST_YEARS) -> dict[str, tuple[float, int]]:
560 + """country → (value, year) using the value nearest to `year` within ±within years (later year wins ties)."""
561 + where = ["o.indicator_id = ?", "o.frequency = i.frequency", "NOT o.is_forecast", "o.value IS NOT NULL",
562 + "coalesce(o.status, 'imported') <> 'quarantined'", "coalesce(c.kind, 'country') = 'country'", "abs(o.year - ?) <= ?"]
563 + params: list[Any] = [indicator_id, year, within]
564 + if members is not None:
565 + if not members:
566 + return {}
567 + where.append(f"o.country_id IN ({','.join('?' * len(members))})")
568 + params += members
569 + rows = snap.query_rows(
570 + f"""SELECT o.country_id, arg_min(o.value, abs(o.year - ?) * 2 - (o.year > ?)::INT) AS v, arg_min(o.year, abs(o.year - ?) * 2 - (o.year > ?)::INT) AS y
571 + FROM observations o JOIN countries c ON c.id = o.country_id JOIN indicators i ON i.id = o.indicator_id
572 + WHERE {' AND '.join(where)} GROUP BY 1""",
573 + [year, year, year, year, *params],
574 + )
575 + return {r[0]: (float(r[1]), int(r[2])) for r in rows}
576 +
577 +
578 +def _parse_log(v: str | None, ind: dict[str, Any], values: np.ndarray | None) -> bool:
579 + if v is None or v.lower() == "auto":
580 + return auto_log(ind, values)
581 + return v.lower() in ("1", "true", "yes")
582 +
583 +
584 +def _size_indicator(snap: Snapshot, size: str | None) -> dict[str, Any] | None:
585 + if size is None:
586 + size = "population"
587 + if size.lower() in ("none", "", "0"):
588 + return None
589 + return resolve_indicator(snap, size)
590 +
591 +
592 +@router.get("/scatter", response_model=schemas.ScatterResponse, summary="Cross-section scatter with Pearson / Spearman / OLS")
593 +def scatter(
594 + x: str = Query(...), y: str = Query(...), size: str | None = Query("population"), year: int | None = Query(None, ge=1800, le=2100),
595 + group: str = Query("world"), log_x: str | None = Query("auto"), log_y: str | None = Query("auto"),
596 + snap: Snapshot = Depends(get_snapshot),
597 +) -> dict[str, Any]:
598 + xi, yi = resolve_indicator(snap, x), resolve_indicator(snap, y)
599 + if xi["id"] == yi["id"]:
600 + raise bad_request("x and y must differ.")
601 + si = _size_indicator(snap, size)
602 + g, members = group_members_or_all(snap, group)
603 + if year is None:
604 + yx, yy = latest_common_year(snap, xi["id"]), latest_common_year(snap, yi["id"])
605 + year_used = min([v for v in (yx, yy) if v is not None], default=None)
606 + else:
607 + year_used = year
608 + points: list[dict[str, Any]] = []
609 + xs: list[float] = []
610 + ys: list[float] = []
611 + if year_used is not None:
612 + vx = _nearest_values(snap, xi["id"], year_used, members)
613 + vy = _nearest_values(snap, yi["id"], year_used, members)
614 + vs = _nearest_values(snap, si["id"], year_used, members) if si else {}
615 + for cid in sorted(set(vx) & set(vy)):
616 + c = snap.countries().get(cid)
617 + if c is None:
618 + continue
619 + p = point_country(c)
620 + p.update({"x": vx[cid][0], "y": vy[cid][0], "size": vs.get(cid, (None, None))[0], "year_x": vx[cid][1], "year_y": vy[cid][1]})
621 + points.append(p)
622 + xs.append(vx[cid][0])
623 + ys.append(vy[cid][0])
624 + xa, ya = np.array(xs, dtype=float), np.array(ys, dtype=float)
625 + lx, ly = _parse_log(log_x, xi, xa), _parse_log(log_y, yi, ya)
626 + with np.errstate(divide="ignore", invalid="ignore"):
627 + tx = np.where(xa > 0, np.log10(xa), np.nan) if lx else xa
628 + ty = np.where(ya > 0, np.log10(ya), np.nan) if ly else ya
629 + r, n = stats.pearson(tx.tolist(), ty.tolist())
630 + rho, _ = stats.spearman(xa.tolist(), ya.tolist())
631 + fit = stats.ols(tx.tolist(), ty.tolist())
632 + provs = indicator_provenances(snap)
633 + return {
634 + "meta": meta_block(snap), "x": indicator_card(xi), "y": indicator_card(yi), "size": indicator_card(si) if si else None,
635 + "year": year, "year_used": year_used, "nearest_years": NEAREST_YEARS,
636 + "group": {"id": g["id"], "slug": g.get("slug"), "name": g.get("name"), "kind": g.get("kind"), "wb_code": g.get("wb_code"), "n_members": g.get("n_members")},
637 + "n": len(points), "points": points,
638 + "stats": {"n": n, "pearson": r, "spearman": rho, "ols": {"slope": fit["slope"], "intercept": fit["intercept"], "r2": fit["r2"]} if fit else None,
639 + "log_x": lx, "log_y": ly},
640 + "note": NOTE_CORRELATION,
641 + "provenance": [p for p in (provs.get(xi["id"]), provs.get(yi["id"]), provs.get(si["id"]) if si else None) if p],
642 + }
643 +
644 +
645 +# ------------------------------------------------------------------------------------------------ /trajectory
646 +@router.get("/trajectory", response_model=schemas.TrajectoryResponse, summary="Gapminder-style frames for two indicators")
647 +def trajectory(
648 + x: str = Query("gdp-per-capita-ppp"), y: str = Query("life-expectancy"), size: str | None = Query("population"),
649 + from_: int | None = Query(None, alias="from", ge=1800, le=2100), to: int | None = Query(None, ge=1800, le=2100),
650 + group: str = Query("world"), snap: Snapshot = Depends(get_snapshot),
651 +) -> dict[str, Any]:
652 + xi, yi = resolve_indicator(snap, x), resolve_indicator(snap, y)
653 + if xi["id"] == yi["id"]:
654 + raise bad_request("x and y must differ.")
655 + si = _size_indicator(snap, size)
656 + g, members = group_members_or_all(snap, group)
657 + ids = [xi["id"], yi["id"]] + ([si["id"]] if si else [])
658 + where = [f"o.indicator_id IN ({','.join('?' * len(ids))})", "o.frequency = 'A'", "NOT o.is_forecast", "o.value IS NOT NULL",
659 + "coalesce(o.status, 'imported') <> 'quarantined'", "coalesce(c.kind, 'country') = 'country'"]
660 + params: list[Any] = list(ids)
661 + if members is not None:
662 + if not members:
663 + where.append("FALSE")
664 + else:
665 + where.append(f"o.country_id IN ({','.join('?' * len(members))})")
666 + params += members
667 + rows = snap.query_rows(f"SELECT o.indicator_id, o.country_id, o.year, o.value FROM observations o JOIN countries c ON c.id = o.country_id WHERE {' AND '.join(where)}", params)
668 + data: dict[str, dict[str, dict[int, float]]] = {i: {} for i in ids}
669 + for iid, cid, yr, v in rows:
670 + data[iid].setdefault(cid, {})[int(yr)] = float(v)
671 + # years with ≥ 50 countries on both axes
672 + count_by_year: dict[int, int] = {}
673 + for cid in set(data[xi["id"]]) & set(data[yi["id"]]):
674 + for yr in set(data[xi["id"]][cid]) & set(data[yi["id"]][cid]):
675 + count_by_year[yr] = count_by_year.get(yr, 0) + 1
676 + good = sorted(y_ for y_, n in count_by_year.items() if n >= 50)
677 + if not good:
678 + good = sorted(count_by_year)
679 + y0 = from_ if from_ is not None else (good[0] if good else None)
680 + y1 = to if to is not None else (good[-1] if good else None)
681 + years = list(range(y0, y1 + 1)) if y0 is not None and y1 is not None and y0 <= y1 else []
682 + countries_out: list[dict[str, Any]] = []
683 + series: dict[str, Any] = {}
684 + dom = {k: [math.inf, -math.inf] for k in ("x", "y", "size")}
685 + for cid in sorted(set(data[xi["id"]]) | set(data[yi["id"]])):
686 + c = snap.countries().get(cid)
687 + if c is None or not years:
688 + continue
689 + sx = [data[xi["id"]].get(cid, {}).get(yr) for yr in years]
690 + sy = [data[yi["id"]].get(cid, {}).get(yr) for yr in years]
691 + both = sum(1 for a, b in zip(sx, sy, strict=True) if a is not None and b is not None)
692 + if both < 0.3 * len(years) or both == 0:
693 + continue
694 + ss = [data[si["id"]].get(cid, {}).get(yr) for yr in years] if si else [None] * len(years)
695 + for k, arr in (("x", sx), ("y", sy), ("size", ss)):
696 + for v in arr:
697 + if v is not None:
698 + dom[k][0] = min(dom[k][0], v)
699 + dom[k][1] = max(dom[k][1], v)
700 + countries_out.append(point_country(c))
701 + series[cid] = {"x": sx, "y": sy, "size": ss}
702 + xs_all = np.array([v for s in series.values() for v in s["x"] if v is not None] or [np.nan])
703 + ys_all = np.array([v for s in series.values() for v in s["y"] if v is not None] or [np.nan])
704 + provs = indicator_provenances(snap)
705 +
706 + def dom_or_none(k: str) -> list[float] | None:
707 + return None if dom[k][0] == math.inf else dom[k]
708 +
709 + return {
710 + "meta": meta_block(snap), "x": indicator_card(xi), "y": indicator_card(yi), "size": indicator_card(si) if si else None,
711 + "group": {"id": g["id"], "slug": g.get("slug"), "name": g.get("name"), "kind": g.get("kind"), "wb_code": g.get("wb_code"), "n_members": g.get("n_members")},
712 + "years": years, "countries": countries_out, "series": series,
713 + "domains": {"x": dom_or_none("x") or [0, 1], "y": dom_or_none("y") or [0, 1], "size": dom_or_none("size")},
714 + "log_x": auto_log(xi, xs_all), "log_y": auto_log(yi, ys_all),
715 + "provenance": [p for p in (provs.get(i) for i in ids) if p],
716 + }
717 +
718 +
719 +# ------------------------------------------------------------------------------------------------ /finder
720 +FINDER_OPS = ("gt", "gte", "lt", "lte", "eq", "between")
721 +
722 +
723 +def _parse_filter(snap: Snapshot, raw: str) -> dict[str, Any]:
724 + parts = raw.split(":")
725 + if len(parts) != 3:
726 + raise bad_request(f"Filter '{raw}' must look like slug:op:value (ops: {', '.join(FINDER_OPS)}).")
727 + slug, op, val = parts[0].strip().lower(), parts[1].strip().lower(), parts[2].strip()
728 + if op not in FINDER_OPS:
729 + raise bad_request(f"Unknown operator '{op}' in '{raw}'.")
730 + ind = resolve_indicator(snap, slug)
731 + try:
732 + if op == "between":
733 + a, b = val.split("..")
734 + v1, v2 = float(a), float(b)
735 + if v1 > v2:
736 + v1, v2 = v2, v1
737 + else:
738 + v1, v2 = float(val), None
739 + except ValueError as e:
740 + raise bad_request(f"Bad value in filter '{raw}'.") from e
741 + return {"indicator": ind, "op": op, "value": v1, "value2": v2}
742 +
743 +
744 +def _apply(op: str, v: np.ndarray, a: float, b: float | None) -> np.ndarray:
745 + with np.errstate(invalid="ignore"):
746 + if op == "gt":
747 + return v > a
748 + if op == "gte":
749 + return v >= a
750 + if op == "lt":
751 + return v < a
752 + if op == "lte":
753 + return v <= a
754 + if op == "eq":
755 + return np.isclose(v, a)
756 + return (v >= a) & (v <= (b if b is not None else a))
757 +
758 +
759 +@router.get("/finder", response_model=schemas.FinderResponse, summary="Structured country finder over latest values")
760 +def finder(
761 + f: list[str] | None = Query(None, description="Repeatable filter slug:op:value (ops gt gte lt lte eq between a..b)"),
762 + mode: str = Query("and", pattern="^(and|or)$"),
763 + region: str | None = Query(None), income: str | None = Query(None), sort: str | None = Query(None),
764 + limit: int = Query(218, ge=1, le=300), snap: Snapshot = Depends(get_snapshot),
765 +) -> dict[str, Any]:
766 + filters = [_parse_filter(snap, raw) for raw in (f or []) if raw]
767 + if not filters:
768 + raise bad_request("Provide at least one filter, e.g. f=gdp-per-capita:gt:40000.")
769 + cs = cross_section(snap)
770 + n = len(cs.countries)
771 + pool = np.ones(n, dtype=bool)
772 + for gslug in (region, income):
773 + if gslug:
774 + g = resolve_group(snap, gslug)
775 + if g["id"] != "world":
776 + mem = set(snap.group_members(g["id"]))
777 + pool &= np.array([c in mem for c in cs.countries])
778 + matched_matrix = np.zeros((n, len(filters)), dtype=bool)
779 + evaluated = np.zeros(n, dtype=bool)
780 + for k, flt in enumerate(filters):
781 + v, _y = cs.col(flt["indicator"]["id"], within=None)
782 + evaluated |= np.isfinite(v)
783 + matched_matrix[:, k] = _apply(flt["op"], v, flt["value"], flt["value2"]) & np.isfinite(v)
784 + ok = matched_matrix.all(axis=1) if mode == "and" else matched_matrix.any(axis=1)
785 + ok &= pool
786 + # sort
787 + sort_slug, sort_dir = filters[0]["indicator"]["id"], "desc"
788 + if sort:
789 + parts = sort.split(":")
790 + sort_slug = resolve_indicator(snap, parts[0])["id"]
791 + if len(parts) > 1 and parts[1].lower() in ("asc", "desc"):
792 + sort_dir = parts[1].lower()
793 + sv, _ = cs.col(sort_slug, within=None)
794 + idx = [i for i in range(n) if ok[i]]
795 + idx.sort(key=lambda i: ((0, -sv[i] if sort_dir == "desc" else sv[i]) if np.isfinite(sv[i]) else (1, 0)))
796 + slugs = list(dict.fromkeys([flt["indicator"]["id"] for flt in filters] + [sort_slug]))
797 + provs = indicator_provenances(snap)
798 + items = []
799 + for i in idx[:limit]:
800 + cid = cs.countries[i]
801 + vals: dict[str, Any] = {}
802 + for s in slugs:
803 + v, y = cs.col(s, within=None)
804 + ind = merged_indicator(snap.indicators()[s])
805 + vals[s] = {"value": clean_float(v[i]), "year": int(y[i]) if np.isfinite(y[i]) else None,
806 + "formatted": format_value(clean_float(v[i]), ind), "provenance": provs.get(s)}
807 + items.append({"country": country_card(snap.countries()[cid]), "matched": [flt["indicator"]["id"] for k, flt in enumerate(filters) if matched_matrix[i, k]],
808 + "values": vals})
809 + return {
810 + "meta": meta_block(snap), "mode": mode,
811 + "filters": [{"indicator": indicator_card(flt["indicator"]), "op": flt["op"], "value": flt["value"], "value2": flt["value2"],
812 + "year_used": int(cs.max_year[cs.iidx[flt["indicator"]["id"]]]) if flt["indicator"]["id"] in cs.iidx and np.isfinite(cs.max_year[cs.iidx[flt["indicator"]["id"]]]) else None}
813 + for flt in filters],
814 + "n_matching": int(ok.sum()), "n_evaluated": int((evaluated & pool).sum()), "items": items,
815 + }
816 +
817 +
818 +# ------------------------------------------------------------------------------------------------ /peers
819 +PEERS_METHODOLOGY = (
820 + "A robust line (Theil–Sen: median of pairwise slopes; or ordinary least squares) is fitted across countries for the chosen year, "
821 + "y on x (x on a log10 scale when the indicator spans orders of magnitude). Each country's residual is its value minus the fitted "
822 + "value at its x; residual_z divides the residual by 1.4826 × MAD of all residuals. 'Above' and 'below' describe position relative "
823 + "to the fitted line for that year only — a descriptive cross-sectional deviation, not a causal or normative judgement."
824 +)
825 +
826 +
827 +@router.get("/peers", response_model=schemas.PeersResponse, summary="Above / below expected given another indicator (robust fit)")
828 +def peers(
829 + y: str = Query("life-expectancy"), x: str = Query("gdp-per-capita-ppp"), year: int | None = Query(None, ge=1800, le=2100),
830 + method: str = Query("theil-sen", pattern="^(theil-sen|ols)$"), log_x: str | None = Query("auto"), top: int = Query(12, ge=1, le=50),
831 + snap: Snapshot = Depends(get_snapshot),
832 +) -> dict[str, Any]:
833 + xi, yi = resolve_indicator(snap, x), resolve_indicator(snap, y)
834 + if xi["id"] == yi["id"]:
835 + raise bad_request("x and y must differ.")
836 + if year is None:
837 + yx, yy = latest_common_year(snap, xi["id"]), latest_common_year(snap, yi["id"])
838 + year_used = min([v for v in (yx, yy) if v is not None], default=None)
839 + else:
840 + year_used = year
841 + pts: list[dict[str, Any]] = []
842 + if year_used is not None:
843 + vx = _nearest_values(snap, xi["id"], year_used, None)
844 + vy = _nearest_values(snap, yi["id"], year_used, None)
845 + for cid in sorted(set(vx) & set(vy)):
846 + c = snap.countries().get(cid)
847 + if c is None:
848 + continue
849 + p = point_country(c)
850 + p.update({"x": vx[cid][0], "y": vy[cid][0]})
851 + pts.append(p)
852 + xa = np.array([p["x"] for p in pts], dtype=float)
853 + ya = np.array([p["y"] for p in pts], dtype=float)
854 + lx = _parse_log(log_x, xi, xa)
855 + with np.errstate(divide="ignore", invalid="ignore"):
856 + tx = np.where(xa > 0, np.log10(xa), np.nan) if lx else xa
857 + fit = stats.theil_sen(tx.tolist(), ya.tolist()) if method == "theil-sen" else stats.ols(tx.tolist(), ya.tolist())
858 + if fit:
859 + expected = fit["intercept"] + fit["slope"] * tx
860 + resid = ya - expected
861 + scale = fit.get("residual_mad") if method == "theil-sen" else None
862 + if not scale:
863 + fr = resid[np.isfinite(resid)]
864 + scale = float(np.median(np.abs(fr - np.median(fr))) * stats.MAD_SCALE) if len(fr) else None
865 + for p, e, r in zip(pts, expected, resid, strict=True):
866 + ok = bool(np.isfinite(e))
867 + p["expected"] = float(e) if ok else None
868 + p["residual"] = float(r) if ok else None
869 + p["residual_z"] = float(r / scale) if ok and scale else None
870 + else:
871 + scale = None
872 + for p in pts:
873 + p.update({"expected": None, "residual": None, "residual_z": None})
874 + ranked = sorted([p for p in pts if p["residual_z"] is not None], key=lambda p: p["residual_z"])
875 + return {
876 + "meta": meta_block(snap), "x": indicator_card(xi), "y": indicator_card(yi), "year_used": year_used, "n": len(pts), "method": method,
877 + "fit": {"slope": fit["slope"], "intercept": fit["intercept"], "r2": fit["r2"], "log_x": lx, "residual_scale": scale} if fit else None,
878 + "points": pts, "above": list(reversed(ranked[-top:])) if ranked else [], "below": ranked[:top],
879 + "pairs": [p for p in PEER_PAIRS if p["x"] in snap.indicators() and p["y"] in snap.indicators()],
880 + "note": "Countries above the fitted line have a higher value than the cross-country relationship would suggest for their x; "
881 + "below, a lower one. " + NOTE_CORRELATION,
882 + "methodology": PEERS_METHODOLOGY,
883 + }
884 +
885 +
886 +__all__ = ["CrossSection", "auto_log", "country_ids", "cross_section", "delta_mode", "indicator_provenances", "is_level", "latest_common_year",
887 + "parse_csv", "point_country", "population_latest", "resolve_country", "router"]
modified src/countryatlas/api/routers/countries.py +133 −6
@@ -416,19 +416,146 @@ DNA_LABELS = {
416 416 }
417 417
418 418
419 −@router.get("/{id}/dna", response_model=schemas.DNAResponse, summary="Country DNA (9 percentile dimensions)")
420 −def get_country_dna(id: str, snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
419 +def _dna_dims(snap: Snapshot, country_id: str) -> tuple[dict[str, float | None], int | None]:
421 420 import json
422 421
423 − c = resolve_country(snap, id)
424 − row = snap.one("SELECT * FROM country_dna WHERE country_id = ?", [c["id"]])
422 + row = snap.one("SELECT * FROM country_dna WHERE country_id = ?", [country_id])
425 423 dims: dict[str, float | None] = {}
426 424 if row and row.get("dims") is not None:
427 425 raw = row["dims"]
428 426 if isinstance(raw, str):
429 427 raw = json.loads(raw)
430 428 dims = {k: clean_float(v) for k, v in dict(raw).items()}
429 + return dims, (row.get("year_ref") if row else None)
430 +
431 +
432 +def _dna_reference(snap: Snapshot, c: dict[str, Any], reference: str | None, dims: dict[str, float | None]) -> dict[str, Any] | None:
433 + """world → 50 everywhere; region/income → median of the members' dims; a country → that country's dims."""
434 + import json
435 +
436 + import numpy as np
437 +
438 + if not reference:
439 + return None
440 + ref = reference.strip().lower()
441 + if ref == "world":
442 + return {"kind": "world", "id": "world", "label": "World median", "dims": {k: 50.0 for k in dims}}
443 + if ref in ("region", "income"):
444 + g = None
445 + for grp in snap.groups().values():
446 + if grp.get("kind") == ref and (grp.get("wb_code") or "").upper() == ((c.get("region_wb") if ref == "region" else c.get("income_group")) or "").upper():
447 + g = grp
448 + break
449 + if g is None:
450 + return None
451 + members = [m for m in snap.group_members(g["id"]) if m != c["id"]]
452 + if not members:
453 + return None
454 + rows = snap.query(f"SELECT dims FROM country_dna WHERE country_id IN ({','.join('?' * len(members))})", members)
455 + acc: dict[str, list[float]] = {}
456 + for r in rows:
457 + raw = r["dims"]
458 + d = json.loads(raw) if isinstance(raw, str) else dict(raw)
459 + for k, v in d.items():
460 + if v is not None:
461 + acc.setdefault(k, []).append(float(v))
462 + return {"kind": ref, "id": g["id"], "label": f"{g.get('name')} median",
463 + "dims": {k: (float(np.median(acc[k])) if acc.get(k) else None) for k in dims}}
464 + other = resolve_country(snap, ref)
465 + od, _ = _dna_dims(snap, other["id"])
466 + return {"kind": "country", "id": other["id"], "label": other.get("short_name"), "dims": {k: od.get(k) for k in dims}}
467 +
468 +
469 +@router.get("/{id}/dna", response_model=schemas.DNAResponse, summary="Country DNA (9 percentile dimensions) with optional reference profile")
470 +def get_country_dna(id: str, reference: str | None = Query(None, description="world | region | income | <ISO3 or slug>"),
471 + snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
472 + c = resolve_country(snap, id)
473 + dims, year_ref = _dna_dims(snap, c["id"])
431 474 dimensions = [{"id": k, "label": DNA_LABELS.get(k, (k.replace("_", " ").title(), None))[0],
432 475 "indicator": DNA_LABELS.get(k, (None, None))[1], "value": v} for k, v in dims.items()]
433 − return {"meta": meta_block(snap), "country": country_card(c), "dims": dims, "year_ref": row.get("year_ref") if row else None,
434 − "dimensions": dimensions}
476 + return {"meta": meta_block(snap), "country": country_card(c), "dims": dims, "year_ref": year_ref, "dimensions": dimensions,
477 + "reference": _dna_reference(snap, c, reference, dims), "note": "Percentile profile among all countries, not an overall country score."}
478 +
479 +
480 +# ------------------------------------------------------------------------------------------------ story (API 1.1)
481 +STORY_INDICATORS = ["population", "gdp-per-capita-ppp", "life-expectancy", "fertility-rate", "urban-population-share", "co2-per-capita",
482 + "internet-users", "renewable-electricity-share", "general-government-gross-debt-pct-gdp", "median-age",
483 + "energy-use-per-capita", "unemployment-rate"]
484 +STORY_FALLBACK = {"gdp-per-capita-ppp": "gdp-per-capita"}
485 +STORY_MIN_POINTS = 10
486 +STORY_MAX_ITEMS = 10
487 +
488 +
489 +@router.get("/{id}/story", response_model=schemas.StoryResponse, summary="How a country changed: long-run indicators, templated text")
490 +def get_country_story(id: str, snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
491 + from countryatlas import stats
492 + from countryatlas.api.formatting import format_change
493 + from countryatlas.api.routers.analytics import delta_mode
494 +
495 + c = resolve_country(snap, id)
496 + wanted: list[str] = []
497 + for s in STORY_INDICATORS:
498 + if s in snap.indicators():
499 + wanted.append(s)
500 + elif STORY_FALLBACK.get(s) in snap.indicators():
501 + wanted.append(STORY_FALLBACK[s])
502 + ph = ",".join("?" * len(wanted))
503 + rows = snap.query(
504 + f"""SELECT o.indicator_id, o.year, o.value, o.source_id, o.source_dataset, o.source_series_code, o.retrieved_at, o.source_updated_at
505 + FROM observations o JOIN indicators i ON i.id = o.indicator_id
506 + WHERE o.country_id = ? AND o.indicator_id IN ({ph}) AND o.frequency = 'A' AND NOT o.is_forecast AND o.value IS NOT NULL
507 + AND coalesce(o.status, 'imported') <> 'quarantined' ORDER BY o.indicator_id, o.year""",
508 + [c["id"], *wanted],
509 + )
510 + by_ind: dict[str, list[dict[str, Any]]] = {}
511 + for r in rows:
512 + by_ind.setdefault(r["indicator_id"], []).append(r)
513 + ranks = {(r[0], int(r[1])): (int(r[2]), int(r[3])) for r in snap.query_rows(
514 + f"SELECT indicator_id, year, rank, n FROM rankings WHERE country_id = ? AND indicator_id IN ({ph})", [c["id"], *wanted])}
515 + items = []
516 + since: int | None = None
517 + for iid in wanted:
518 + rs = by_ind.get(iid, [])
519 + if len(rs) < STORY_MIN_POINTS:
520 + continue
521 + ind = merged_indicator(snap.indicators()[iid])
522 + vals = [float(r["value"]) for r in rs]
523 + years = [int(r["year"]) for r in rs]
524 + # start from the first year with data (≥ 1960 preferred when the series is longer)
525 + start = next((i for i, y in enumerate(years) if y >= 1960), 0)
526 + vals, years, rs = vals[start:], years[start:], rs[start:]
527 + if len(vals) < STORY_MIN_POINTS:
528 + continue
529 + first, last = vals[0], vals[-1]
530 + y0, y1 = years[0], years[-1]
531 + since = y0 if since is None else min(since, y0)
532 + delta = last - first
533 + pct = stats.yoy(last, first)
534 + cg = stats.cagr(first, last, y1 - y0) if delta_mode(ind) == "pct" else None
535 + peak_i, trough_i = max(range(len(vals)), key=lambda i: vals[i]), min(range(len(vals)), key=lambda i: vals[i])
536 + mode = delta_mode(ind)
537 + name = ind.get("short_name") or ind.get("name")
538 + verb = "rose" if delta > 0 else "fell" if delta < 0 else "was unchanged"
539 + ch = format_change(delta, pct, ind)
540 + if mode == "points":
541 + tail = f"({ch})" if ch else ""
542 + else:
543 + cg_txt = f", {'+' if cg >= 0 else '−'}{abs(cg):.1f} % a year" if cg is not None else ""
544 + tail = f"({ch}{cg_txt})" if ch else ""
545 + text = f"{c['short_name']}: {name or iid} {verb} from {format_value(first, ind)} in {y0} to {format_value(last, ind)} in {y1} {tail}".strip() + "."
546 + prov = build_provenance(snap, iid, rs[-1].get("source_id"), rs[-1].get("source_dataset"), rs[-1].get("source_series_code"),
547 + rs[-1].get("retrieved_at"), rs[-1].get("source_updated_at"), c.get("iso2"))
548 + rf, rl = ranks.get((iid, y0)), ranks.get((iid, y1))
549 + items.append({
550 + "indicator": indicator_card(ind),
551 + "first": {"year": y0, "value": first, "formatted": format_value(first, ind)},
552 + "last": {"year": y1, "value": last, "formatted": format_value(last, ind)},
553 + "change_abs": delta, "change_pct": pct, "cagr": cg,
554 + "peak": {"year": years[peak_i], "value": vals[peak_i]}, "trough": {"year": years[trough_i], "value": vals[trough_i]},
555 + "rank_first": {"rank": rf[0], "n": rf[1], "year": y0} if rf else None,
556 + "rank_last": {"rank": rl[0], "n": rl[1], "year": y1} if rl else None,
557 + "series": [[y, v] for y, v in zip(years, vals, strict=True)], "text": text, "provenance": prov,
558 + })
559 + if len(items) >= STORY_MAX_ITEMS:
560 + break
561 + return {"meta": meta_block(snap), "country": country_card(c), "since": since, "items": items}
modified src/countryatlas/api/routers/health.py +1 −1
@@ -11,7 +11,7 @@ from countryatlas.api.db import get_database
11 11
12 12 router = APIRouter(tags=["health"])
13 13
14 −API_VERSION = "0.1.0"
14 +API_VERSION = "1.1.0"
15 15
16 16
17 17 @router.get("/health", response_model=schemas.HealthResponse, summary="Health / snapshot status")
modified src/countryatlas/api/routers/indicators.py +157 −0
@@ -3,6 +3,7 @@ from __future__ import annotations
3 3
4 4 from typing import Any
5 5
6 +import numpy as np
6 7 from fastapi import APIRouter, Depends, Query
7 8
8 9 from countryatlas.api import schemas
@@ -14,6 +15,7 @@ from countryatlas.api.common import (
14 15 merged_indicator,
15 16 meta_block,
16 17 quantile_breaks,
18 + resolve_country,
17 19 resolve_group,
18 20 resolve_indicator,
19 21 )
@@ -339,3 +341,158 @@ def get_indicator_trend(
339 341 r["retrieved_at"], r["source_updated_at"]), n_values=r["n"]) for r in prov_rows],
340 342 "note": "Aggregates are computed across member countries present in the snapshot (no World Bank aggregate series is stored).",
341 343 }
344 +
345 +
346 +# ------------------------------------------------------------------------------------------------ analytics (API 1.1)
347 +from countryatlas import stats as _stats
348 +from countryatlas.api.errors import bad_request as _bad_request
349 +from countryatlas.api.routers import analytics as _an
350 +
351 +NOTE_CORRELATION = "Correlation does not imply causation. Descriptive cross-sectional statistics on latest values."
352 +
353 +
354 +def _log_matrix(snap: Snapshot):
355 + """Per-snapshot: cross-section values transformed for association (log10 for level series), aligned with CrossSection."""
356 +
357 + def load():
358 + cs = _an.cross_section(snap)
359 + m = np.full_like(cs.values, np.nan)
360 + logs: dict[str, bool] = {}
361 + for iid, j in cs.iidx.items():
362 + ind = merged_indicator(snap.indicators()[iid])
363 + v, _ = cs.col(iid)
364 + use_log = _an.auto_log(ind, v)
365 + logs[iid] = use_log
366 + with np.errstate(divide="ignore", invalid="ignore"):
367 + m[:, j] = np.where(v > 0, np.log10(v), np.nan) if use_log else v
368 + return m, logs
369 +
370 + return snap._cached("assoc_matrix", load)
371 +
372 +
373 +@router.get("/{slug}/related", response_model=schemas.RelatedResponse, summary="Statistically related indicators (descriptive)")
374 +def get_indicator_related(slug: str, limit: int = Query(12, ge=1, le=60), min_n: int = Query(40, ge=5),
375 + snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
376 + ind = resolve_indicator(snap, slug)
377 + cs = _an.cross_section(snap)
378 + m, logs = _log_matrix(snap)
379 + j = cs.iidx.get(ind["id"])
380 + items = []
381 + if j is not None:
382 + x = m[:, j]
383 + raw_x, _ = cs.col(ind["id"])
384 + for other, k in cs.iidx.items():
385 + if other == ind["id"]:
386 + continue
387 + y = m[:, k]
388 + ok = np.isfinite(x) & np.isfinite(y)
389 + if ok.sum() < min_n:
390 + continue
391 + raw_y, years_y = cs.col(other)
392 + rho, n = _stats.spearman(raw_x[ok].tolist(), raw_y[ok].tolist())
393 + r, _ = _stats.pearson(x[ok].tolist(), y[ok].tolist())
394 + if rho is None:
395 + continue
396 + o = merged_indicator(snap.indicators()[other])
397 + yrs = years_y[ok]
398 + items.append({"indicator": indicator_card(o), "pearson": r, "spearman": rho, "n": n,
399 + "year": int(np.nanmax(yrs)) if len(yrs) else None, "log_x": logs[ind["id"]], "log_y": logs[other],
400 + "direction": "positive" if rho >= 0 else "negative"})
401 + items.sort(key=lambda i: -abs(i["spearman"] or 0))
402 + year_used = int(cs.max_year[j]) if j is not None and np.isfinite(cs.max_year[j]) else None
403 + return {"meta": meta_block(snap), "indicator": indicator_card(ind), "year_used": year_used, "n_candidates": len(items),
404 + "items": items[:limit], "note": NOTE_CORRELATION}
405 +
406 +
407 +@router.get("/{slug}/distribution", response_model=schemas.DistributionResponse, summary="Distribution across countries for a year")
408 +def get_indicator_distribution(slug: str, year: int | None = Query(None, ge=1800, le=2100), highlight: str | None = Query(None),
409 + bins: int = Query(20, ge=5, le=60), snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
410 + ind = resolve_indicator(snap, slug)
411 + year_used = year if year is not None else latest_common_year(snap, ind["id"])[0]
412 + vals = _an._nearest_values(snap, ind["id"], year_used, None) if year_used is not None else {}
413 + ids = sorted(vals)
414 + arr = np.array([vals[c][0] for c in ids], dtype=float)
415 + use_log = _an.auto_log(ind, arr)
416 + fin = arr[np.isfinite(arr)]
417 +
418 + def pct(q: float) -> float | None:
419 + return float(np.percentile(fin, q)) if len(fin) else None
420 +
421 + st = {"min": float(fin.min()) if len(fin) else None, "p10": pct(10), "p25": pct(25), "median": pct(50),
422 + "mean": float(fin.mean()) if len(fin) else None, "p75": pct(75), "p90": pct(90), "max": float(fin.max()) if len(fin) else None}
423 + # group medians
424 + def group_medians(kind: str) -> list[dict[str, Any]]:
425 + out = []
426 + for g in snap.groups().values():
427 + if g.get("kind") != kind:
428 + continue
429 + mem = set(snap.group_members(g["id"]))
430 + gv = [vals[c][0] for c in ids if c in mem]
431 + if len(gv) >= 2:
432 + out.append({"group": {"id": g["id"], "slug": g.get("slug"), "name": g.get("name"), "kind": g.get("kind"), "wb_code": g.get("wb_code"),
433 + "n_members": g.get("n_members")}, "median": float(np.median(gv)), "n": len(gv)})
434 + out.sort(key=lambda x: -(x["median"] or 0))
435 + return out
436 +
437 + by_region, by_income = group_medians("region"), group_medians("income")
438 + hl = None
439 + if highlight:
440 + c = resolve_country(snap, highlight)
441 + v = vals.get(c["id"])
442 + if v is not None:
443 + pr = _stats.percentile_rank(arr.tolist())
444 + i = ids.index(c["id"])
445 + desc = ind.get("higher_is_better") is not False
446 + rk = _stats.rank(arr.tolist(), descending=desc)[i]
447 + reg = next((g for g in by_region if g["group"]["wb_code"] == c.get("region_wb")), None)
448 + inc = next((g for g in by_income if (g["group"]["wb_code"] or "").upper() == (c.get("income_group") or "").upper()), None)
449 + hl = {"country": country_card(c), "value": v[0], "year": v[1], "percentile": pr[i], "rank": rk, "n": len(fin),
450 + "region": reg["group"] if reg else None, "region_median": reg["median"] if reg else None,
451 + "income": inc["group"] if inc else None, "income_median": inc["median"] if inc else None}
452 + else:
453 + hl = {"country": country_card(c), "value": None, "year": None, "percentile": None, "rank": None, "n": len(fin), "region": None,
454 + "region_median": None, "income": None, "income_median": None}
455 + return {"meta": meta_block(snap), "indicator": indicator_card(ind), "year": year, "year_used": year_used, "n": len(fin), "log": use_log,
456 + "histogram": _stats.histogram(fin.tolist(), bins=bins, log=use_log), "stats": st, "highlight": hl,
457 + "by_region": by_region, "by_income": by_income, "provenance": _an.indicator_provenances(snap).get(ind["id"])}
458 +
459 +
460 +@router.get("/{slug}/frames", response_model=schemas.FramesResponse, summary="Multi-year map frames (time machine)")
461 +def get_indicator_frames(slug: str, from_: int | None = Query(None, alias="from", ge=1750, le=2100), to: int | None = Query(None, ge=1750, le=2100),
462 + step: int = Query(1, ge=1, le=10), group: str = Query("world"), snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
463 + ind = resolve_indicator(snap, slug)
464 + g, members = _an.group_members_or_all(snap, group)
465 + where = ["o.indicator_id = ?", "o.frequency = 'A'", "NOT o.is_forecast", "o.value IS NOT NULL", "coalesce(o.status, 'imported') <> 'quarantined'",
466 + "coalesce(c.kind, 'country') = 'country'"]
467 + params: list[Any] = [ind["id"]]
468 + if members is not None:
469 + if not members:
470 + where.append("FALSE")
471 + else:
472 + where.append(f"o.country_id IN ({','.join('?' * len(members))})")
473 + params += members
474 + if from_ is not None:
475 + where.append("o.year >= ?")
476 + params.append(from_)
477 + if to is not None:
478 + where.append("o.year <= ?")
479 + params.append(to)
480 + rows = snap.query_rows(f"SELECT o.country_id, o.year, o.value FROM observations o JOIN countries c ON c.id = o.country_id WHERE {' AND '.join(where)}", params)
481 + by_year: dict[int, dict[str, float]] = {}
482 + for cid, yr, v in rows:
483 + by_year.setdefault(int(yr), {})[cid] = float(v)
484 + years = sorted(y for y, d in by_year.items() if len(d) >= 20)
485 + if step > 1:
486 + years = [y for y in years if (y - years[0]) % step == 0] if years else []
487 + years = years[-80:]
488 + ids = sorted({c for y in years for c in by_year[y]})
489 + values = {c: [by_year[y].get(c) for y in years] for c in ids}
490 + pooled = [v for y in years for v in by_year[y].values()]
491 + breaks = quantile_breaks(pooled, 6)
492 + if from_ is not None and to is not None and from_ > to:
493 + raise _bad_request("`from` must be <= `to`.")
494 + return {"meta": meta_block(snap), "indicator": indicator_card(ind),
495 + "group": {"id": g["id"], "slug": g.get("slug"), "name": g.get("name"), "kind": g.get("kind"), "wb_code": g.get("wb_code"), "n_members": g.get("n_members")},
496 + "years": years, "values": values,
497 + "legend": {"min": min(pooled) if pooled else None, "max": max(pooled) if pooled else None, "breaks": breaks, "n_classes": len(breaks) + 1},
498 + "n_by_year": [len(by_year[y]) for y in years], "provenance": _an.indicator_provenances(snap).get(ind["id"])}
added src/countryatlas/api/routers/quality.py +202 −0
@@ -0,0 +1,202 @@
1 +"""Data quality and freshness (API 1.1): /indicators/{slug}/quality, /countries/{id}/quality, /updates.
2 +
3 +Badges vocabulary (shared): fresh · historical · sparse · limited-coverage · stale · flagged · forecast. No file paths, hosts
4 +or secrets are ever returned.
5 +"""
6 +from __future__ import annotations
7 +
8 +from typing import Any
9 +
10 +import numpy as np
11 +from fastapi import APIRouter, Depends, Query
12 +
13 +from countryatlas.api import schemas
14 +from countryatlas.api.common import country_card, indicator_card, merged_indicator, meta_block, resolve_country, resolve_indicator
15 +from countryatlas.api.db import Snapshot, get_snapshot
16 +from countryatlas.api.provenance import _iso
17 +from countryatlas.api.routers.analytics import country_ids, indicator_provenances, latest_common_year
18 +from countryatlas.api.routers.indicators import indicator_summary
19 +
20 +router = APIRouter(tags=["quality"])
21 +
22 +SPARSE_POINTS = 10
23 +FRESH_LAG = 1
24 +STALE_LAG = 3
25 +FLAGGED_SHARE = 0.05
26 +
27 +
28 +def reference_year(snap: Snapshot) -> int:
29 + return snap._cached("reference_year", lambda: int(snap.scalar(
30 + "SELECT max(l.year) FROM latest l JOIN countries c ON c.id = l.country_id WHERE coalesce(c.kind, 'country') = 'country' "
31 + "AND l.indicator_id IN ('population', 'gdp', 'life-expectancy')") or 0))
32 +
33 +
34 +def _badges(*, latest_year: int | None, first_year: int | None, points: float | None, coverage_pct: float | None, flagged_share: float,
35 + forecast: bool, ref: int) -> list[str]:
36 + out: list[str] = []
37 + if latest_year is not None and latest_year >= ref - FRESH_LAG:
38 + out.append("fresh")
39 + if first_year is not None and first_year <= 1970:
40 + out.append("historical")
41 + if points is not None and points < SPARSE_POINTS:
42 + out.append("sparse")
43 + if coverage_pct is not None and coverage_pct < 50:
44 + out.append("limited-coverage")
45 + if latest_year is not None and latest_year <= ref - STALE_LAG:
46 + out.append("stale")
47 + if flagged_share > FLAGGED_SHARE:
48 + out.append("flagged")
49 + if forecast:
50 + out.append("forecast")
51 + return out
52 +
53 +
54 +@router.get("/indicators/{slug}/quality", response_model=schemas.IndicatorQualityResponse, summary="Coverage / freshness / continuity of an indicator")
55 +def indicator_quality(slug: str, snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
56 + ind = resolve_indicator(snap, slug)
57 + total = len(country_ids(snap))
58 + ref = reference_year(snap)
59 + per_country = snap.query(
60 + """SELECT o.country_id, count(*) AS n, min(o.year) AS y0, max(o.year) AS y1,
61 + sum(CASE WHEN o.status = 'warning' THEN 1 ELSE 0 END) AS n_warn,
62 + max(CASE WHEN o.is_forecast THEN 1 ELSE 0 END) AS has_fc
63 + FROM observations o JOIN countries c ON c.id = o.country_id JOIN indicators i ON i.id = o.indicator_id
64 + WHERE o.indicator_id = ? AND o.frequency = i.frequency AND coalesce(c.kind, 'country') = 'country' AND o.value IS NOT NULL
65 + GROUP BY 1""",
66 + [ind["id"]],
67 + )
68 + actual = [r for r in per_country]
69 + n_countries = len(actual)
70 + years_rows = snap.query_rows(
71 + """SELECT o.year, count(*) FROM observations o JOIN countries c ON c.id = o.country_id JOIN indicators i ON i.id = o.indicator_id
72 + WHERE o.indicator_id = ? AND o.frequency = i.frequency AND NOT o.is_forecast AND o.value IS NOT NULL
73 + AND coalesce(c.kind, 'country') = 'country' GROUP BY 1 ORDER BY 1""",
74 + [ind["id"]],
75 + )
76 + latest_years = [int(r["y1"]) for r in actual]
77 + points = [int(r["n"]) for r in actual]
78 + n_warn = sum(int(r["n_warn"]) for r in actual)
79 + n_obs = sum(points)
80 + first_year = min((int(r["y0"]) for r in actual), default=None)
81 + last_year = max(latest_years, default=None)
82 + coverage = round(100.0 * n_countries / max(total, 1), 1)
83 + med_points = float(np.median(points)) if points else None
84 + prov = indicator_provenances(snap).get(ind["id"])
85 + src_rows = snap.query(
86 + "SELECT source_id, source_dataset, source_series_code, max(retrieved_at) AS ra, max(source_updated_at) AS su, count(*) AS n "
87 + "FROM observations WHERE indicator_id = ? GROUP BY 1, 2, 3 ORDER BY n DESC", [ind["id"]])
88 + from countryatlas.api.provenance import build_provenance
89 +
90 + sources = [dict(build_provenance(snap, ind["id"], r["source_id"], r["source_dataset"], r["source_series_code"], r["ra"], r["su"]), n_values=r["n"])
91 + for r in src_rows]
92 + return {
93 + "meta": meta_block(snap), "indicator": indicator_card(ind), "n_countries": n_countries, "n_countries_total": total, "coverage_pct": coverage,
94 + "first_year": first_year, "last_year": last_year, "latest_common_year": latest_common_year(snap, ind["id"]),
95 + "n_years": len(years_rows), "years_with_50plus": sum(1 for _, n in years_rows if n >= 50),
96 + "median_points_per_country": med_points, "sparse_countries": sum(1 for p in points if p < SPARSE_POINTS),
97 + "stale_countries": sum(1 for y in latest_years if y <= ref - STALE_LAG), "flagged_values": n_warn,
98 + "sources": sources,
99 + "badges": _badges(latest_year=last_year, first_year=first_year, points=med_points, coverage_pct=coverage,
100 + flagged_share=(n_warn / n_obs) if n_obs else 0.0, forecast=any(int(r["has_fc"]) for r in actual), ref=ref),
101 + "provenance": prov,
102 + }
103 +
104 +
105 +@router.get("/countries/{id}/quality", response_model=schemas.CountryQualityResponse, summary="Per-indicator data quality for a country")
106 +def country_quality(id: str, snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
107 + c = resolve_country(snap, id)
108 + ref = reference_year(snap)
109 + rows = snap.query(
110 + """SELECT o.indicator_id, count(*) AS n, min(o.year) AS y0, max(o.year) AS y1, count(DISTINCT o.year) AS ny,
111 + sum(CASE WHEN o.status = 'warning' THEN 1 ELSE 0 END) AS n_warn,
112 + arg_max(o.status, o.period) AS status, arg_max(o.source_id, o.period) AS source,
113 + max(o.source_updated_at) AS su, max(o.retrieved_at) AS ra, max(CASE WHEN o.is_forecast THEN 1 ELSE 0 END) AS has_fc
114 + FROM observations o JOIN indicators i ON i.id = o.indicator_id
115 + WHERE o.country_id = ? AND o.frequency = i.frequency AND NOT o.is_forecast AND o.value IS NOT NULL GROUP BY 1 ORDER BY 1""",
116 + [c["id"]],
117 + )
118 + items = []
119 + n_fresh = n_stale = n_sparse = n_flagged = 0
120 + latest = None
121 + for r in rows:
122 + ind = merged_indicator(snap.indicators()[r["indicator_id"]])
123 + y0, y1, n = int(r["y0"]), int(r["y1"]), int(r["n"])
124 + expected = y1 - y0 + 1 if ind.get("frequency", "A") == "A" else n
125 + missing = max(0, expected - int(r["ny"]))
126 + cont = round(100.0 * int(r["ny"]) / expected, 1) if expected else None
127 + badges = _badges(latest_year=y1, first_year=y0, points=n, coverage_pct=None, flagged_share=int(r["n_warn"]) / n if n else 0.0,
128 + forecast=bool(int(r["has_fc"])), ref=ref)
129 + n_fresh += "fresh" in badges
130 + n_stale += "stale" in badges
131 + n_sparse += "sparse" in badges
132 + n_flagged += "flagged" in badges
133 + latest = y1 if latest is None else max(latest, y1)
134 + items.append({"indicator": indicator_card(ind), "latest_year": y1, "first_year": y0, "n_points": n, "expected_points": expected,
135 + "missing_years": missing, "continuity_pct": cont, "status": r["status"], "source": r["source"],
136 + "source_updated_at": _iso(r["su"]), "retrieved_at": _iso(r["ra"]), "badges": badges})
137 + total = len(snap.indicators())
138 + return {"meta": meta_block(snap), "country": country_card(c),
139 + "summary": {"n_indicators": total, "n_with_data": len(items), "coverage_pct": round(100.0 * len(items) / max(total, 1), 1),
140 + "latest_year": latest, "n_fresh": n_fresh, "n_stale": n_stale, "n_sparse": n_sparse, "n_flagged": n_flagged},
141 + "items": items}
142 +
143 +
144 +@router.get("/updates", response_model=schemas.UpdatesResponse, summary="Freshness dashboard: sources, runs, changed values")
145 +def updates(runs_limit: int = Query(20, ge=1, le=100), snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
146 + run_id = snap.run_id
147 + ref = reference_year(snap)
148 + changed = {r[0]: (int(r[1]), int(r[2])) for r in snap.query_rows(
149 + "SELECT coalesce(new_source_id, old_source_id) AS s, count(*), count(DISTINCT country_id) FROM observation_revisions WHERE run_id = ? GROUP BY 1",
150 + [run_id])}
151 + src_stats = {r["source_id"]: r for r in snap.query(
152 + """SELECT source_id, count(*) AS n_obs, count(DISTINCT indicator_id) AS n_ind, count(DISTINCT source_dataset) AS n_ds,
153 + max(CASE WHEN NOT is_forecast THEN year END) AS latest_year, max(retrieved_at) AS ra, max(source_updated_at) AS su
154 + FROM observations GROUP BY 1""")}
155 + run_status = {r["connector"]: r for r in snap.query(
156 + """SELECT connector, arg_max(status, coalesce(finished_at, started_at)) AS status, max(finished_at) AS finished,
157 + sum(CASE WHEN status IN ('failed', 'quarantined') THEN 1 ELSE 0 END) AS n_failed, count(*) AS n
158 + FROM import_runs GROUP BY 1""")}
159 + sources = []
160 + for sid, s in snap.sources().items():
161 + st = src_stats.get(sid)
162 + rs = run_status.get(sid)
163 + if st is None and rs is None:
164 + continue
165 + latest_year = int(st["latest_year"]) if st and st.get("latest_year") is not None else None
166 + if rs is None:
167 + status = "unknown"
168 + elif rs["status"] == "failed" or (rs["n"] and rs["n_failed"] == rs["n"]):
169 + status = "failed"
170 + elif rs["n_failed"] or rs["status"] in ("partial", "quarantined"):
171 + status = "partial"
172 + elif latest_year is not None and latest_year <= ref - STALE_LAG:
173 + status = "stale"
174 + else:
175 + status = "ok"
176 + ch = changed.get(sid, (0, 0))
177 + sources.append({
178 + "source": {"id": sid, "name": s.get("name"), "organization": s.get("organization"), "url": s.get("url"), "licence": s.get("licence")},
179 + "status": status, "last_success_at": _iso(s.get("last_success_at")), "last_retrieved_at": _iso(st["ra"]) if st else None,
180 + "source_updated_at": _iso(st["su"]) if st else None, "n_datasets": int(st["n_ds"]) if st else 0,
181 + "n_indicators": int(st["n_ind"]) if st else 0, "n_observations": int(st["n_obs"]) if st else 0, "latest_year": latest_year,
182 + "values_changed": ch[0], "countries_affected": ch[1],
183 + "runs": {"n": int(rs["n"]) if rs else 0, "n_failed": int(rs["n_failed"]) if rs else 0},
184 + })
185 + sources.sort(key=lambda s: -s["n_observations"])
186 + runs = snap.query(
187 + "SELECT run_id, connector, dataset, started_at, finished_at, status, rows_valid, warnings, errors, message FROM import_runs "
188 + "ORDER BY coalesce(finished_at, started_at) DESC NULLS LAST LIMIT ?", [runs_limit])
189 + for r in runs:
190 + r["started_at"], r["finished_at"] = _iso(r.get("started_at")), _iso(r.get("finished_at"))
191 + if r.get("message"):
192 + r["message"] = str(r["message"])[:200]
193 + inds = [merged_indicator(i) for i in snap.indicators().values() if i.get("latest_source_updated_at") is not None]
194 + inds.sort(key=lambda i: _iso(i.get("latest_source_updated_at")) or "", reverse=True)
195 + total_changed = int(snap.scalar("SELECT count(*) FROM observation_revisions WHERE run_id = ?", [run_id]) or 0)
196 + return {
197 + "meta": meta_block(snap),
198 + "snapshot": {"run_id": run_id, "built_at": snap.built_at, "observations": int(snap.meta.get("observation_count") or snap.table_count("observations")),
199 + "indicators": len(snap.indicators()), "countries": len(country_ids(snap)), "values_changed": total_changed,
200 + "values_changed_by_source": {k: v[0] for k, v in changed.items()}, "reference_year": ref},
201 + "sources": sources, "recent_runs": runs, "indicators_recently_updated": [indicator_summary(snap, i) for i in inds[:12]],
202 + }
modified src/countryatlas/api/routers/rankings.py +46 −0
@@ -182,3 +182,49 @@ def get_ranking_history(indicator: str, countries: str = Query(..., description=
182 182 "pct_rank": clean_float(r.get("pct_rank"))})
183 183 return {"meta": meta_block(snap), "indicator": indicator_card(ind), "countries": [country_card(c) for c in cs],
184 184 "years": sorted(years), "series": series}
185 +
186 +
187 +@router.get("/{indicator}/race", response_model=schemas.RaceResponse, summary="Rank race frames (top N per year)")
188 +def get_ranking_race(indicator: str, from_: int | None = Query(None, alias="from"), to: int | None = Query(None),
189 + top: int = Query(10, ge=3, le=30), group: str = Query("world"), snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
190 + ind = resolve_indicator(snap, indicator)
191 + g = resolve_group(snap, group)
192 + members = snap.group_members(g["id"]) if g["id"] != "world" else None
193 + where, params = ["r.indicator_id = ?"], [ind["id"]]
194 + if members is not None:
195 + if not members:
196 + where.append("FALSE")
197 + else:
198 + where.append(f"r.country_id IN ({','.join('?' * len(members))})")
199 + params += members
200 + if from_ is not None:
201 + where.append("r.year >= ?")
202 + params.append(from_)
203 + if to is not None:
204 + where.append("r.year <= ?")
205 + params.append(to)
206 + direction = "ASC" if ind.get("higher_is_better") is False else "DESC"
207 + rows = snap.query_rows(
208 + f"""SELECT year, country_id, value, rk FROM (
209 + SELECT r.year, r.country_id, r.value, row_number() OVER (PARTITION BY r.year ORDER BY r.value {direction}, r.country_id) AS rk,
210 + count(*) OVER (PARTITION BY r.year) AS n
211 + FROM rankings r WHERE {' AND '.join(where)})
212 + WHERE rk <= ? AND n >= 20 ORDER BY year, rk""",
213 + [*params, top],
214 + )
215 + frames: dict[int, list[dict[str, Any]]] = {}
216 + ids: set[str] = set()
217 + max_value = None
218 + for year, cid, value, rk in rows:
219 + v = clean_float(value)
220 + frames.setdefault(int(year), []).append({"id": cid, "value": v, "rank": int(rk)})
221 + ids.add(cid)
222 + if v is not None:
223 + max_value = v if max_value is None else max(max_value, v)
224 + years = sorted(frames)
225 + from countryatlas.api.routers.analytics import indicator_provenances
226 +
227 + return {"meta": meta_block(snap), "indicator": indicator_card(ind), "group": group_card(g), "top": top, "years": years,
228 + "frames": [{"year": y, "rows": frames[y]} for y in years],
229 + "countries": {cid: country_card(snap.countries()[cid]) for cid in sorted(ids) if cid in snap.countries()},
230 + "max_value": max_value, "provenance": indicator_provenances(snap).get(ind["id"])}
modified src/countryatlas/api/routers/regions.py +94 −0
@@ -13,6 +13,7 @@ from countryatlas.api.common import (
13 13 indicator_card,
14 14 merged_indicator,
15 15 meta_block,
16 + parse_csv,
16 17 resolve_group,
17 18 resolve_indicator,
18 19 )
@@ -87,6 +88,99 @@ def list_regions(kind: str | None = Query(None), snap: Snapshot = Depends(get_sn
87 88 return {"meta": meta_block(snap), "n": len(items), "items": items}
88 89
89 90
91 +COMPARE_DEFAULT = ["population", "gdp", "gdp-per-capita", "life-expectancy", "co2-per-capita", "internet-users", "gdp-growth", "inflation"]
92 +COMPARE_HISTORY = ["population", "gdp", "gdp-per-capita", "life-expectancy"]
93 +AGG_KIND = {s[0]: s[1] for s in AGG_SPECS}
94 +
95 +
96 +def _agg_kind(ind: dict[str, Any]) -> str:
97 + from countryatlas.api.common import is_per_capita_or_share
98 +
99 + if ind["id"] in AGG_KIND:
100 + return AGG_KIND[ind["id"]]
101 + if (ind.get("aggregation") or "none") == "sum":
102 + return "sum"
103 + return "weighted_mean" if is_per_capita_or_share(ind) else "median"
104 +
105 +
106 +@router.get("/compare", response_model=schemas.RegionCompareResponse, summary="Group vs group: aggregates, world shares, history")
107 +def compare_regions(a: str = Query("g7"), b: str = Query("brics"), indicators: str | None = Query(None),
108 + snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
109 + ga, gb = resolve_group(snap, a), resolve_group(snap, b)
110 + ids = [resolve_indicator(snap, s)["id"] for s in parse_csv(indicators, limit=12)] if indicators else [i for i in COMPARE_DEFAULT if i in snap.indicators()]
111 + groups = [(ga, snap.group_members(ga["id"])), (gb, snap.group_members(gb["id"]))]
112 + world = {r[0]: float(r[1]) for r in snap.query_rows(
113 + "SELECT l.indicator_id, sum(l.value) FROM latest l JOIN countries c ON c.id = l.country_id WHERE coalesce(c.kind,'country') = 'country' "
114 + "AND l.indicator_id IN ('population', 'gdp') GROUP BY 1")}
115 + rows_out: list[dict[str, Any]] = []
116 + shares: dict[str, Any] = {}
117 + stats_by_group: dict[str, dict[str, Any]] = {}
118 + for g, members in groups:
119 + ph = ",".join("?" * len(members)) if members else "NULL"
120 + rows = snap.query(
121 + f"""WITH l AS (SELECT * FROM latest WHERE country_id IN ({ph}) AND value IS NOT NULL AND indicator_id IN ({','.join('?' * len(ids))})),
122 + pop AS (SELECT country_id, value AS pop FROM latest WHERE indicator_id = 'population' AND country_id IN ({ph}))
123 + SELECT l.indicator_id, sum(l.value) AS total, median(l.value) AS med, count(*) AS n, max(l.year) AS year,
124 + sum(l.value * pop.pop) / nullif(sum(CASE WHEN l.value IS NOT NULL THEN pop.pop END), 0) AS wmean
125 + FROM l LEFT JOIN pop ON pop.country_id = l.country_id GROUP BY 1""",
126 + [*members, *ids, *members] if members else [*ids],
127 + )
128 + stats_by_group[g["id"]] = {r["indicator_id"]: r for r in rows}
129 + pop_sum = next((clean_float(r["total"]) for r in rows if r["indicator_id"] == "population"), None)
130 + gdp_sum = next((clean_float(r["total"]) for r in rows if r["indicator_id"] == "gdp"), None)
131 + if pop_sum is None or gdp_sum is None:
132 + extra = {r[0]: float(r[1]) for r in snap.query_rows(
133 + f"SELECT indicator_id, sum(value) FROM latest WHERE country_id IN ({ph}) AND indicator_id IN ('population','gdp') GROUP BY 1", members)} if members else {}
134 + pop_sum = pop_sum if pop_sum is not None else extra.get("population")
135 + gdp_sum = gdp_sum if gdp_sum is not None else extra.get("gdp")
136 + shares[g["id"]] = {"population_share_pct": round(100 * pop_sum / world["population"], 2) if pop_sum and world.get("population") else None,
137 + "gdp_share_pct": round(100 * gdp_sum / world["gdp"], 2) if gdp_sum and world.get("gdp") else None}
138 + for iid in ids:
139 + ind = merged_indicator(snap.indicators()[iid])
140 + kind = _agg_kind(ind)
141 + label = {"sum": "Total", "median": "Median", "weighted_mean": "Population-weighted mean"}[kind]
142 + values: dict[str, Any] = {}
143 + for g, _m in groups:
144 + r = stats_by_group[g["id"]].get(iid)
145 + if r is None:
146 + values[g["id"]] = {"value": None, "formatted": None, "n": 0, "year": None}
147 + continue
148 + val = clean_float({"sum": r["total"], "median": r["med"], "weighted_mean": r["wmean"] if r["wmean"] is not None else r["med"]}[kind])
149 + values[g["id"]] = {"value": val, "formatted": format_value(val, ind), "n": int(r["n"]), "year": r["year"]}
150 + rows_out.append({"indicator": indicator_card(ind), "kind": kind, "label": f"{label} {ind.get('short_name') or ind.get('name')}", "values": values})
151 + # history (annual): per year aggregate over members with >= 60 % reporting
152 + history: dict[str, Any] = {}
153 + for iid in [i for i in COMPARE_HISTORY if i in ids]:
154 + ind = merged_indicator(snap.indicators()[iid])
155 + kind = _agg_kind(ind)
156 + per_group: dict[str, dict[int, float]] = {}
157 + all_years: set[int] = set()
158 + for g, members in groups:
159 + if not members:
160 + per_group[g["id"]] = {}
161 + continue
162 + ph = ",".join("?" * len(members))
163 + rows = snap.query(
164 + f"""SELECT o.year, median(o.value) AS med, sum(o.value) AS total, count(*) AS n,
165 + sum(o.value * p.value) / nullif(sum(CASE WHEN o.value IS NOT NULL THEN p.value END), 0) AS wmean
166 + FROM observations o
167 + LEFT JOIN observations p ON p.indicator_id = 'population' AND p.country_id = o.country_id AND p.year = o.year AND p.frequency = 'A' AND NOT p.is_forecast
168 + WHERE o.indicator_id = ? AND o.frequency = 'A' AND NOT o.is_forecast AND o.value IS NOT NULL AND o.country_id IN ({ph})
169 + GROUP BY o.year HAVING count(*) >= ? ORDER BY o.year""",
170 + [iid, *members, max(1, int(0.6 * len(members)))],
171 + )
172 + d = {}
173 + for r in rows:
174 + v = clean_float({"sum": r["total"], "median": r["med"], "weighted_mean": r["wmean"] if r["wmean"] is not None else r["med"]}[kind])
175 + if v is not None:
176 + d[int(r["year"])] = v
177 + per_group[g["id"]] = d
178 + all_years |= set(d)
179 + years = sorted(all_years)
180 + history[iid] = {"years": years, "kind": kind, **{gid: [per_group[gid].get(y) for y in years] for gid in per_group}}
181 + return {"meta": meta_block(snap), "groups": [group_card(ga), group_card(gb)], "rows": rows_out, "shares": shares, "history": history}
182 +
183 +
90 184 @router.get("/{slug}", response_model=schemas.RegionResponse, summary="Group page: members, aggregates, member ranking")
91 185 def get_region(slug: str, indicator: str = Query("gdp-per-capita", description="Indicator for the member ranking"),
92 186 sort: str | None = Query(None, pattern="^(asc|desc)$"), snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
modified src/countryatlas/api/routers/search.py +115 −0
@@ -134,12 +134,125 @@ def typed_hint(snap: Snapshot, r: dict[str, Any]) -> str:
134 134 return " · ".join(filter(None, [label, raw_hint]))
135 135
136 136
137 +# ------------------------------------------------------------------------------------------------ intents (API 1.1)
138 +INTENT_STOP = {"the", "of", "in", "and", "vs", "versus", "v", "compare", "rank", "ranking", "rankings", "map", "explore", "to", "with", "for", "by"}
139 +
140 +
141 +def _country_hit(snap: Snapshot, text: str, minsim: float = 0.9) -> dict[str, Any] | None:
142 + t = text.strip()
143 + if len(t) in (2, 3) and t.isalpha(): # ISO2 / ISO3 code ("usa", "fr")
144 + c = snap.countries().get(t.upper()) or next((x for x in snap.countries().values() if (x.get("iso2") or "").lower() == t.lower()), None)
145 + if c:
146 + return {"type": "country", "id": c["id"], "slug": c.get("slug"), "name": c.get("short_name"), "score": 1.0, "country": country_card(c)}
147 + hits = run_search(snap, t, 1, {"country"}, minsim=minsim)
148 + return hits[0] if hits else None
149 +
150 +
151 +def _indicator_hit(snap: Snapshot, text: str) -> dict[str, Any] | None:
152 + hits = run_search(snap, text, 1, {"indicator"}, minsim=0.86)
153 + return hits[0] if hits else None
154 +
155 +
156 +def _group_hit(snap: Snapshot, text: str) -> dict[str, Any] | None:
157 + t = text.strip().lower()
158 + for g in snap.groups().values():
159 + if g["id"] == "world":
160 + continue
161 + if t in {(g.get("slug") or "").lower(), (g.get("name") or "").lower(), (g.get("wb_code") or "").lower(), g["id"].lower()}:
162 + return g
163 + return None
164 +
165 +
166 +def _action(action: str, ident: str, name: str, hint: str, url: str) -> dict[str, Any]:
167 + return {"type": "action", "action": action, "id": ident, "slug": ident, "name": name, "hint": hint, "score": 1.0, "url": url,
168 + "country": None, "topic": None, "indicator": None}
169 +
170 +
171 +def _split_countries(snap: Snapshot, tokens: list[str]) -> list[dict[str, Any]]:
172 + """Left-to-right grouping of tokens into country names: an exact single word first, then 2–3-word names, then a fuzzy word."""
173 + out: list[dict[str, Any]] = []
174 + i = 0
175 + while i < len(tokens):
176 + found = None
177 + if tokens[i] not in INTENT_STOP:
178 + h = _country_hit(snap, tokens[i], minsim=0.98)
179 + if h and h["score"] >= 0.95:
180 + found, i = h, i + 1
181 + if found is None:
182 + for w in (3, 2):
183 + chunk = " ".join(tokens[i:i + w])
184 + if len(tokens[i:i + w]) < w:
185 + continue
186 + h = _country_hit(snap, chunk, minsim=0.92)
187 + if h and h["score"] >= 0.97: # (near-)exact multi-word name only — a fuzzy 3-word hit would swallow the next country
188 + found, i = h, i + w
189 + break
190 + if found is None and tokens[i] not in INTENT_STOP:
191 + h = _country_hit(snap, tokens[i], minsim=0.9)
192 + if h and h["score"] >= 0.88:
193 + found, i = h, i + 1
194 + if found is None:
195 + return []
196 + if all(found["id"] != o["id"] for o in out):
197 + out.append(found)
198 + return out
199 +
200 +
201 +def parse_intents(snap: Snapshot, q: str) -> list[dict[str, Any]]:
202 + """Deterministic intent parsing: compare X Y [Z], rank <indicator>, <indicator> <group>, <indicator> map, X vs Y."""
203 + ql = q.strip().lower()
204 + tokens = [t for t in re.split(r"[\s,/]+", ql) if t]
205 + if len(tokens) < 2:
206 + return []
207 + out: list[dict[str, Any]] = []
208 + # X vs Y | compare X Y Z
209 + if tokens[0] in ("compare",) or any(t in ("vs", "versus", "v") for t in tokens):
210 + body = [t for t in tokens if t not in ("compare", "vs", "versus", "v", "and", "with", "to")]
211 + cs = _split_countries(snap, body)
212 + if len(cs) >= 2:
213 + names = " vs ".join(c["name"] for c in cs)
214 + out.append(_action("compare", ":".join(c["id"] for c in cs), names, "Compare · side by side", "/compare/" + "/".join(c["slug"] for c in cs)))
215 + # rank <indicator>
216 + if tokens[0] in ("rank", "ranking", "rankings", "top") and len(tokens) >= 2:
217 + ind = _indicator_hit(snap, " ".join(tokens[1:]))
218 + if ind:
219 + out.append(_action("ranking", ind["id"], f"{ind['name']} ranking", "Ranking · all countries", f"/rankings/{ind['slug']}"))
220 + # <indicator> map|explore | map <indicator>
221 + if tokens[-1] in ("map", "explore") or tokens[0] in ("map", "explore"):
222 + rest = tokens[1:] if tokens[0] in ("map", "explore") else tokens[:-1]
223 + ind = _indicator_hit(snap, " ".join(rest))
224 + if ind:
225 + out.append(_action("explore", ind["id"], f"{ind['name']} on the map", "World Explorer · map + time slider", f"/explore?indicator={ind['slug']}"))
226 + # <indicator> <group> | <group> <indicator>
227 + if not out:
228 + for split in range(1, len(tokens)):
229 + for left, right in ((tokens[:split], tokens[split:]), (tokens[split:], tokens[:split])):
230 + g = _group_hit(snap, " ".join(right))
231 + if not g:
232 + continue
233 + ind = _indicator_hit(snap, " ".join(left))
234 + if ind:
235 + out.append(_action("group_ranking", f"{ind['id']}:{g['id']}", f"{ind['name']} in {g.get('name')}", f"Ranking · {g.get('name')}",
236 + f"/rankings/{ind['slug']}?group={g.get('slug') or g['id']}"))
237 + break
238 + if out:
239 + break
240 + seen = set()
241 + uniq = []
242 + for h in out:
243 + if (h["action"], h["id"]) not in seen:
244 + seen.add((h["action"], h["id"]))
245 + uniq.append(h)
246 + return uniq
247 +
248 +
137 249 @router.get("/search", response_model=schemas.SearchResponse, summary="Search countries, indicators, topics, regions, sources")
138 250 def search(q: str = Query(..., min_length=1, max_length=120), limit: int = Query(10, ge=1, le=50),
139 251 type: str | None = Query(None, description="Restrict to a type: country|indicator|topic|region|source"),
140 252 snap: Snapshot = Depends(get_snapshot)) -> dict[str, Any]:
141 253 types = {type} if type else None
142 254 hits = run_search(snap, q, limit, types)
255 + intents = parse_intents(snap, q) if not type else []
143 256 # combos: "<topic|indicator> <country>" or "<country> <topic|indicator>" → country + topic/indicator + combined hit
144 257 tokens = [t for t in re.split(r"[\s,/]+", q.strip().lower()) if t]
145 258 if len(tokens) >= 2 and not type:
@@ -183,4 +296,6 @@ def search(q: str = Query(..., min_length=1, max_length=120), limit: int = Query
183 296 seen.add(k)
184 297 merged.append(h)
185 298 hits = merged[:limit]
299 + if intents:
300 + hits = (intents + [h for h in hits if h["type"] != "action"])[: max(limit, len(intents) + 3)]
186 301 return {"meta": meta_block(snap), "q": q, "n": len(hits), "hits": hits}
modified src/countryatlas/api/schemas.py +336 −0
@@ -568,3 +568,339 @@ class HealthResponse(Base):
568 568 db_path: str | None = None
569 569 version: str | None = None
570 570 cache: dict[str, int] | None = None
571 +
572 +
573 +# ------------------------------------------------------------------------------------------------ analytics (API 1.1)
574 +class MoverLite(Base):
575 + country: CountryCard
576 + value: float | None = None
577 + ref_value: float | None = None
578 + delta: float | None = None
579 + delta_pct: float | None = None
580 + formatted: str | None = None
581 + year: int | None = None
582 +
583 +
584 +class PulseItem(Base):
585 + indicator: IndicatorCard
586 + year: int
587 + n: int
588 + n_up: int
589 + n_down: int
590 + n_flat: int
591 + share_up: float
592 + share_down: float
593 + median_change_abs: float | None = None
594 + median_change_pct: float | None = None
595 + direction_semantics: str
596 + record_highs: int
597 + record_lows: int
598 + headline: str
599 + top_up: MoverLite | None = None
600 + top_down: MoverLite | None = None
601 + convergence: dict[str, Any] | None = None
602 + provenance: Provenance | None = None
603 +
604 +
605 +class PulseResponse(Base):
606 + meta: Meta
607 + year_reference: int
608 + summary: dict[str, Any]
609 + items: list[PulseItem]
610 +
611 +
612 +class MoverItem(Base):
613 + country: CountryCard
614 + indicator: IndicatorCard
615 + kind: str
616 + year: int | None = None
617 + ref_year: int | None = None
618 + value: float | None = None
619 + ref_value: float | None = None
620 + delta: float | None = None
621 + delta_pct: float | None = None
622 + formatted: str | None = None
623 + formatted_ref: str | None = None
624 + severity: float | None = None
625 + direction: str
626 + interpretation: str | None = None
627 + headline: str | None = None
628 + provenance: Provenance | None = None
629 +
630 +
631 +class MoversResponse(Base):
632 + meta: Meta
633 + window: int
634 + category: str
635 + kind: str
636 + min_population: int | None = None
637 + filter_note: str | None = None
638 + categories: list[str]
639 + kinds: list[str]
640 + n: int
641 + items: list[MoverItem]
642 +
643 +
644 +class ExtremeRow(Base):
645 + country: CountryCard
646 + value_start: float | None = None
647 + value_end: float | None = None
648 + year_start: int | None = None
649 + year_end: int | None = None
650 + delta: float | None = None
651 + delta_pct: float | None = None
652 + formatted_start: str | None = None
653 + formatted_end: str | None = None
654 +
655 +
656 +class ExtremeFacet(Base):
657 + id: str
658 + title: str
659 + indicator: IndicatorCard
660 + direction: str
661 + metric: str
662 + rows: list[ExtremeRow]
663 + n: int
664 + provenance: Provenance | None = None
665 +
666 +
667 +class ExtremesResponse(Base):
668 + meta: Meta
669 + window: str
670 + from_year: int | None = None
671 + to_year: int | None = None
672 + min_population: int | None = None
673 + filter_note: str | None = None
674 + facets: list[ExtremeFacet]
675 +
676 +
677 +class PointCountry(Base):
678 + id: str
679 + slug: str | None = None
680 + name: str | None = None
681 + flag: str | None = None
682 + region: str | None = None
683 + income: str | None = None
684 +
685 +
686 +class ScatterPoint(PointCountry):
687 + x: float | None = None
688 + y: float | None = None
689 + size: float | None = None
690 + year_x: int | None = None
691 + year_y: int | None = None
692 +
693 +
694 +class ScatterResponse(Base):
695 + meta: Meta
696 + x: IndicatorCard
697 + y: IndicatorCard
698 + size: IndicatorCard | None = None
699 + year: int | None = None
700 + year_used: int | None = None
701 + nearest_years: int
702 + group: GroupCard
703 + n: int
704 + points: list[ScatterPoint]
705 + stats: dict[str, Any]
706 + note: str
707 +
708 +
709 +class TrajectoryResponse(Base):
710 + meta: Meta
711 + x: IndicatorCard
712 + y: IndicatorCard
713 + size: IndicatorCard | None = None
714 + group: GroupCard
715 + years: list[int]
716 + countries: list[PointCountry]
717 + series: dict[str, dict[str, list[float | None]]]
718 + domains: dict[str, Any]
719 + log_x: bool
720 + log_y: bool
721 + provenance: list[Provenance] = Field(default_factory=list)
722 +
723 +
724 +class FinderFilter(Base):
725 + indicator: IndicatorCard
726 + op: str
727 + value: float
728 + value2: float | None = None
729 + year_used: int | None = None
730 +
731 +
732 +class FinderItem(Base):
733 + country: CountryCard
734 + matched: list[str]
735 + values: dict[str, Any]
736 +
737 +
738 +class FinderResponse(Base):
739 + meta: Meta
740 + mode: str
741 + filters: list[FinderFilter]
742 + n_matching: int
743 + n_evaluated: int
744 + items: list[FinderItem]
745 +
746 +
747 +class PeerPoint(PointCountry):
748 + x: float | None = None
749 + y: float | None = None
750 + expected: float | None = None
751 + residual: float | None = None
752 + residual_z: float | None = None
753 +
754 +
755 +class PeersResponse(Base):
756 + meta: Meta
757 + x: IndicatorCard
758 + y: IndicatorCard
759 + year_used: int | None = None
760 + n: int
761 + method: str
762 + fit: dict[str, Any] | None = None
763 + points: list[PeerPoint]
764 + above: list[PeerPoint]
765 + below: list[PeerPoint]
766 + pairs: list[dict[str, str]]
767 + note: str
768 + methodology: str
769 +
770 +
771 +class RelatedItem(Base):
772 + indicator: IndicatorCard
773 + pearson: float | None = None
774 + spearman: float | None = None
775 + n: int
776 + year: int | None = None
777 + log_x: bool
778 + log_y: bool
779 + direction: str
780 +
781 +
782 +class RelatedResponse(Base):
783 + meta: Meta
784 + indicator: IndicatorCard
785 + year_used: int | None = None
786 + n_candidates: int
787 + items: list[RelatedItem]
788 + note: str
789 +
790 +
791 +class DistributionResponse(Base):
792 + meta: Meta
793 + indicator: IndicatorCard
794 + year: int | None = None
795 + year_used: int | None = None
796 + n: int
797 + log: bool
798 + histogram: dict[str, Any]
799 + stats: dict[str, float | None]
800 + highlight: dict[str, Any] | None = None
801 + by_region: list[dict[str, Any]]
802 + by_income: list[dict[str, Any]]
803 + provenance: Provenance | None = None
804 +
805 +
806 +class FramesResponse(Base):
807 + meta: Meta
808 + indicator: IndicatorCard
809 + group: GroupCard
810 + years: list[int]
811 + values: dict[str, list[float | None]]
812 + legend: MapLegend
813 + n_by_year: list[int]
814 + provenance: Provenance | None = None
815 +
816 +
817 +class IndicatorQualityResponse(Base):
818 + meta: Meta
819 + indicator: IndicatorCard
820 + n_countries: int
821 + n_countries_total: int
822 + coverage_pct: float
823 + first_year: int | None = None
824 + last_year: int | None = None
825 + latest_common_year: int | None = None
826 + n_years: int
827 + years_with_50plus: int
828 + median_points_per_country: float | None = None
829 + sparse_countries: int
830 + stale_countries: int
831 + flagged_values: int
832 + sources: list[Provenance] = Field(default_factory=list)
833 + badges: list[str]
834 +
835 +
836 +class RaceResponse(Base):
837 + meta: Meta
838 + indicator: IndicatorCard
839 + group: GroupCard
840 + top: int
841 + years: list[int]
842 + frames: list[dict[str, Any]]
843 + countries: dict[str, CountryCard]
844 + max_value: float | None = None
845 + provenance: Provenance | None = None
846 +
847 +
848 +class RegionCompareResponse(Base):
849 + meta: Meta
850 + groups: list[GroupCard]
851 + rows: list[dict[str, Any]]
852 + shares: dict[str, Any]
853 + history: dict[str, Any]
854 +
855 +
856 +class StoryItem(Base):
857 + indicator: IndicatorCard
858 + first: dict[str, Any]
859 + last: dict[str, Any]
860 + change_abs: float | None = None
861 + change_pct: float | None = None
862 + cagr: float | None = None
863 + peak: dict[str, Any] | None = None
864 + trough: dict[str, Any] | None = None
865 + rank_first: dict[str, Any] | None = None
866 + rank_last: dict[str, Any] | None = None
867 + series: list[list[float | None]]
868 + text: str
869 + provenance: Provenance | None = None
870 +
871 +
872 +class StoryResponse(Base):
873 + meta: Meta
874 + country: CountryCard
875 + since: int | None = None
876 + items: list[StoryItem]
877 +
878 +
879 +class CountryQualityItem(Base):
880 + indicator: IndicatorCard
881 + latest_year: int | None = None
882 + first_year: int | None = None
883 + n_points: int
884 + expected_points: int
885 + missing_years: int
886 + continuity_pct: float | None = None
887 + status: str | None = None
888 + source: str | None = None
889 + source_updated_at: str | None = None
890 + retrieved_at: str | None = None
891 + badges: list[str]
892 +
893 +
894 +class CountryQualityResponse(Base):
895 + meta: Meta
896 + country: CountryCard
897 + summary: dict[str, Any]
898 + items: list[CountryQualityItem]
899 +
900 +
901 +class UpdatesResponse(Base):
902 + meta: Meta
903 + snapshot: dict[str, Any]
904 + sources: list[dict[str, Any]]
905 + recent_runs: list[dict[str, Any]]
906 + indicators_recently_updated: list[IndicatorSummary]
modified src/countryatlas/connectors/base.py +35 −10
@@ -11,12 +11,12 @@ import json
11 11 import threading
12 12 import time
13 13 from abc import ABC, abstractmethod
14 −from datetime import datetime, timezone
14 +from datetime import UTC, datetime
15 15 from pathlib import Path
16 16 from typing import Any, ClassVar
17 17
18 18 import httpx
19 −from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
19 +from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
20 20
21 21 from countryatlas.config import settings
22 22 from countryatlas.models import (
@@ -57,6 +57,28 @@ class TransientHTTPError(Exception):
57 57 pass
58 58
59 59
60 +RETRY_STATUSES = (408, 425, 429, 500, 502, 503, 504)
61 +RETRY_ATTEMPTS = 5
62 +RETRY_AFTER_CAP = 120.0 # seconds; a longer Retry-After is honoured up to this cap
63 +
64 +
65 +def retry_after_seconds(value: str | None) -> float | None:
66 + """Parse a Retry-After header (delta-seconds or HTTP-date) into seconds ≥ 0, capped; None when absent/unparseable."""
67 + if not value:
68 + return None
69 + v = value.strip()
70 + if v.isdigit():
71 + return min(float(v), RETRY_AFTER_CAP)
72 + try:
73 + from email.utils import parsedate_to_datetime
74 +
75 + dt = parsedate_to_datetime(v)
76 + except Exception: # noqa: BLE001
77 + return None
78 + delta = (dt - datetime.now(UTC)).total_seconds()
79 + return min(max(delta, 0.0), RETRY_AFTER_CAP)
80 +
81 +
60 82 class Connector(ABC):
61 83 id: ClassVar[str] = ""
62 84 name: ClassVar[str] = ""
@@ -79,19 +101,22 @@ class Connector(ABC):
79 101 )
80 102
81 103 # ---------------------------------------------------------------- HTTP
104 + # Deterministic exponential backoff (2, 4, 8, 16 s … capped at 60 s, no jitter so runs are reproducible) on transient
105 + # statuses (408/425/429/5xx) and transport errors (timeouts, resets); 5 attempts. A `Retry-After` header is honoured
106 + # (up to RETRY_AFTER_CAP) before the backoff wait.
82 107 @retry(
83 108 reraise=True,
84 − stop=stop_after_attempt(5),
85 − wait=wait_exponential_jitter(initial=2, max=60),
109 + stop=stop_after_attempt(RETRY_ATTEMPTS),
110 + wait=wait_exponential(multiplier=2, min=2, max=60),
86 111 retry=retry_if_exception_type((TransientHTTPError, httpx.TransportError)),
87 112 )
88 113 def get(self, url: str, params: dict[str, Any] | None = None, headers: dict[str, str] | None = None) -> httpx.Response:
89 114 self._limiter.acquire()
90 115 r = self._client.get(url, params=params, headers=headers)
91 − if r.status_code in (429, 500, 502, 503, 504):
92 − retry_after = r.headers.get("Retry-After")
93 − if retry_after and retry_after.isdigit():
94 − time.sleep(min(int(retry_after), 120))
116 + if r.status_code in RETRY_STATUSES:
117 + wait = retry_after_seconds(r.headers.get("Retry-After"))
118 + if wait:
119 + time.sleep(wait)
95 120 raise TransientHTTPError(f"{r.status_code} {url}")
96 121 r.raise_for_status()
97 122 return r
@@ -111,7 +136,7 @@ class Connector(ABC):
111 136 dataset=dataset,
112 137 code=code,
113 138 url=str(r.url),
114 − retrieved_at=datetime.now(timezone.utc),
139 + retrieved_at=datetime.now(UTC),
115 140 status_code=r.status_code,
116 141 content_type=r.headers.get("Content-Type"),
117 142 body=r.content,
@@ -197,7 +222,7 @@ class Connector(ABC):
197 222 def apply_transform(value: float, transform: str | None) -> float:
198 223 if not transform:
199 224 return value
200 − return float(eval(transform, {"__builtins__": {}}, {"x": value})) # noqa: S307 — registry-controlled expressions
225 + return float(eval(transform, {"__builtins__": {}}, {"x": value}))
201 226
202 227 def close(self) -> None:
203 228 self._client.close()
modified src/countryatlas/pipeline/build.py +66 −3
@@ -33,6 +33,7 @@ SCHEMA_SQL = Path(__file__).resolve().parents[1] / "storage" / "schema.sql"
33 33 MIN_OBSERVATIONS_WITH_WB = 100_000
34 34 HEADLINE_MIN_COUNTRIES = 100
35 35 HEADLINE_MIN_N_WORLD = 50 # warning when a headline indicator's latest row is ranked in a pool smaller than this
36 +MIN_RETENTION_VS_PREVIOUS = 0.90 # never publish a snapshot with < 90 % of the previous snapshot's observations
36 37
37 38 # Public facts about sources whose connector module is not implemented yet (so `sources` is complete for the API).
38 39 SOURCE_DEFAULTS: dict[str, dict[str, str]] = {
@@ -287,6 +288,53 @@ def _merge_observations(con: duckdb.DuckDBPyConnection) -> tuple[int, int, int]:
287 288 return n, n_alt, n_fresher
288 289
289 290
291 +def _previous_observation_count(previous_db: Path) -> int | None:
292 + """`observation_count` of the live snapshot (its `meta` first, else a count), None when there is no previous snapshot."""
293 + if not previous_db.exists():
294 + return None
295 + try:
296 + prev = duckdb.connect(str(previous_db), read_only=True)
297 + except Exception as e: # noqa: BLE001
298 + log.warning("previous snapshot unreadable (%s) — retention check skipped", e)
299 + return None
300 + try:
301 + row = prev.execute("SELECT value FROM meta WHERE key = 'observation_count'").fetchone()
302 + if row and str(row[0]).isdigit():
303 + return int(row[0])
304 + return int(prev.execute("SELECT count(*) FROM observations").fetchone()[0])
305 + except Exception as e: # noqa: BLE001
306 + log.warning("previous snapshot count unavailable (%s)", e)
307 + return None
308 + finally:
309 + prev.close()
310 +
311 +
312 +def _source_health(con: duckdb.DuckDBPyConnection) -> dict[str, dict[str, int]]:
313 + """Per connector: ok / partial / failed / quarantined spec counts from the latest import_runs."""
314 + out: dict[str, dict[str, int]] = {}
315 + for connector, status, n in con.execute(
316 + "SELECT connector, status, count(*) FROM import_runs GROUP BY 1, 2 ORDER BY 1, 2"
317 + ).fetchall():
318 + out.setdefault(connector, {"ok": 0, "partial": 0, "failed": 0, "quarantined": 0})
319 + out[connector][status if status in out[connector] else "failed"] += int(n)
320 + return out
321 +
322 +
323 +def _write_build_log(run_id: str, counts: dict[str, Any], warnings: list[str], files: list[Path], duration_s: float,
324 + published: bool, error: str | None = None) -> None:
325 + """`logs/build-<run_id>.json`: counts, warnings, staging files used, source health — for audit and the admin console."""
326 + try:
327 + settings.logs_dir.mkdir(parents=True, exist_ok=True)
328 + doc = {
329 + "run_id": run_id, "finished_at": datetime.now(UTC).isoformat(), "published": published, "error": error,
330 + "duration_s": round(duration_s, 1), "counts": counts, "warnings": warnings,
331 + "staging_files": [f"{p.parent.name}/{p.name}" for p in files],
332 + }
333 + (settings.logs_dir / f"build-{run_id}.json").write_text(json.dumps(doc, indent=1, default=str))
334 + except Exception as e: # noqa: BLE001
335 + log.warning("could not write build log: %s", e)
336 +
337 +
290 338 def _carry_revisions(con: duckdb.DuckDBPyConnection, previous_db: Path, run_id: str) -> tuple[int, int]:
291 339 """Compare with the previous live snapshot; record changed values and copy the old revisions table."""
292 340 if not previous_db.exists():
@@ -355,12 +403,15 @@ def _load_runs_and_issues(con: duckdb.DuckDBPyConnection) -> tuple[int, int]:
355 403
356 404
357 405 # ------------------------------------------------------------------------------------------------ integrity + swap
358 −def _integrity(con: duckdb.DuckDBPyConnection, files: list[Path], strict: bool) -> list[str]:
406 +def _integrity(con: duckdb.DuckDBPyConnection, files: list[Path], strict: bool, previous_count: int | None = None) -> list[str]:
359 407 problems: list[str] = []
360 408 n_obs = con.execute("SELECT count(*) FROM observations").fetchone()[0]
361 409 has_wb = any(p.parent.name == "worldbank" for p in files)
362 410 if has_wb and n_obs < MIN_OBSERVATIONS_WITH_WB:
363 411 problems.append(f"only {n_obs} observations although World Bank staging is present (< {MIN_OBSERVATIONS_WITH_WB})")
412 + if previous_count and n_obs < MIN_RETENTION_VS_PREVIOUS * previous_count:
413 + problems.append(f"only {n_obs} observations vs {previous_count} in the previous snapshot "
414 + f"({n_obs / previous_count:.1%} < {MIN_RETENTION_VS_PREVIOUS:.0%}) — refusing to publish a shrunken snapshot")
364 415 staged_indicators = {r[0] for r in con.execute("SELECT DISTINCT indicator_id FROM staging_all").fetchall()}
365 416 cov = dict(con.execute("SELECT indicator_id, count(*) FROM latest GROUP BY indicator_id").fetchall())
366 417 for slug in registry.topics()["headline"]:
@@ -414,6 +465,11 @@ def build(run_id: str | None = None, strict: bool = True, swap: bool = True) ->
414 465 for o in orphans: # spec removed/renamed in the registry since the file was staged → must not leak into the snapshot
415 466 log.warning("ignoring orphan staging file (no matching source spec in the registry): %s", o)
416 467 log.info("build %s: %d staging files (%d orphans ignored) → %s", run_id, len(files), len(orphans), build_path)
468 + by_conn: dict[str, int] = {}
469 + for f in files:
470 + by_conn[f.parent.name] = by_conn.get(f.parent.name, 0) + 1
471 + log.info("build %s: staging files per connector: %s", run_id, ", ".join(f"{k}={v}" for k, v in sorted(by_conn.items())))
472 + previous_count = _previous_observation_count(settings.db_path)
417 473 counts: dict[str, int] = {}
418 474 warnings: list[str] = []
419 475 con = duckdb.connect(str(build_path))
@@ -478,16 +534,20 @@ def build(run_id: str | None = None, strict: bool = True, swap: bool = True) ->
478 534 _connector_meta())
479 535 counts["indicators_with_data"] = con.execute("SELECT count(DISTINCT indicator_id) FROM observations").fetchone()[0]
480 536 counts["countries_with_data"] = con.execute("SELECT count(DISTINCT country_id) FROM observations").fetchone()[0]
481 − warnings = _integrity(con, files, strict)
537 + warnings = _integrity(con, files, strict, previous_count)
482 538 for w in warnings:
483 539 log.warning("integrity: %s", w)
540 + health = _source_health(con)
541 + counts["specs_failed"] = sum(h["failed"] + h["quarantined"] for h in health.values())
484 542 derived.write_meta(con, {
485 543 "schema_version": SCHEMA_VERSION, "build_run_id": run_id, "observation_count": counts["observations"],
544 + "previous_observation_count": previous_count if previous_count is not None else "",
486 545 "observations_alt_count": counts["observations_alt"], "indicator_count": counts["indicators_with_data"],
487 546 "country_count": counts["countries_with_data"], "latest_count": counts["latest"],
488 547 "rankings_count": counts["rankings"], "changes_count": counts["changes"], "events_count": counts["events"],
489 548 "insights_count": counts["insights"], "similarity_count": counts["similarity"],
490 549 "staging_files": len(files), "connectors": ",".join(sorted({p.parent.name for p in files})),
550 + "source_health": json.dumps(health, sort_keys=True),
491 551 "registry_dir": str(settings.registry_dir), "repo": str(ROOT),
492 552 "build_duration_s": f"{time.monotonic() - t_start:.1f}",
493 553 "integrity_warnings": " | ".join(warnings),
@@ -495,7 +555,7 @@ def build(run_id: str | None = None, strict: bool = True, swap: bool = True) ->
495 555 con.execute("CHECKPOINT")
496 556 con.close()
497 557 _timer("meta + checkpoint", t0)
498 − except Exception:
558 + except Exception as exc:
499 559 try:
500 560 con.close()
501 561 except Exception as e: # noqa: BLE001
@@ -503,6 +563,8 @@ def build(run_id: str | None = None, strict: bool = True, swap: bool = True) ->
503 563 build_path.unlink(missing_ok=True)
504 564 build_path.with_suffix(".duckdb.wal").unlink(missing_ok=True)
505 565 log.exception("build %s FAILED — live database untouched", run_id)
566 + _write_build_log(run_id, counts, warnings, files, time.monotonic() - t_start, published=False,
567 + error=f"{type(exc).__name__}: {exc}")
506 568 raise
507 569
508 570 snap: Path | None = None
@@ -511,6 +573,7 @@ def build(run_id: str | None = None, strict: bool = True, swap: bool = True) ->
511 573 log.info("build %s swapped into %s (snapshot %s)", run_id, settings.db_path, snap.name)
512 574 dur = time.monotonic() - t_start
513 575 log.info("build %s done in %.1fs: %s", run_id, dur, json.dumps(counts))
576 + _write_build_log(run_id, counts, warnings, files, dur, published=swap)
514 577 return BuildResult(run_id=run_id, db_path=settings.db_path if swap else build_path, snapshot_path=snap, counts=counts,
515 578 duration_s=dur, warnings=warnings)
516 579
modified src/countryatlas/pipeline/changes.py +109 −1
@@ -1,7 +1,13 @@
1 1 """Deterministic change / event detectors (ARCHITECTURE §7) with templated English headlines — no LLM anywhere.
2 2
3 3 `changes` = detectors evaluated at the LATEST period of each country×indicator series
4 −`events` = detectors evaluated over the WHOLE history (yoy jumps/drops, records after a ≥5-year gap, sign flips)
4 +`events` = detectors evaluated over the WHOLE history (yoy jumps/drops, records after a ≥5-year gap, sign flips,
5 + one structural break per series)
6 +
7 +Change detection 2.0 kinds (countryatlas.stats): `structural_break` (single mean shift, gain ≥ 0.5 of the total sum of
8 +squares, |shift| ≥ 1.5 sd, segments ≥ 5 years; `changes` only when the break is within the last 10 years), `trend_reversal`
9 +(three yearly moves of one sign after three of the opposite sign, cumulative move ≥ floor), `volatility_spike` (sd of the
10 +last 5 yearly differences ≥ 3 × sd of the previous 15 and ≥ floor). Severities 0.45–0.7 before the importance weight.
5 11
6 12 Working scale: percent-like indicators are analysed on absolute differences ("points"); positive level series
7 13 (GDP, population, emissions…) on log-differences (reported as % change); everything else on absolute differences.
@@ -21,6 +27,7 @@ from typing import Any
21 27 import numpy as np
22 28 import polars as pl
23 29
30 +from countryatlas import stats
24 31 from countryatlas.pipeline.format import delta_mode, fmt_delta, fmt_value
25 32 from countryatlas.registry import Indicator
26 33
@@ -43,6 +50,15 @@ EVENT_MIN_SEVERITY = 0.35 # events below this are noise and are dropped
43 50 EVENTS_PER_INDICATOR_YEAR = 3 # timeline cap per (country, indicator, year)
44 51 RECORD_GAP_YEARS = 5
45 52 SIGN_FLIP_HINTS = ("growth", "balance", "net-migration", "inflation", "change")
53 +# --- change detection 2.0 (structural break, trend reversal, volatility spike) ---
54 +BREAK_MIN_SEGMENT = 5 # years on each side of a mean shift
55 +BREAK_MIN_GAIN = 0.5 # share of the total sum of squares explained by the split
56 +BREAK_MIN_SHIFT_RATIO = 1.5 # |shift| / sd(series)
57 +BREAK_RECENT_YEARS = 10 # `changes` only when the break is within the last 10 years
58 +REVERSAL_RUN = 3 # 3 yearly moves of one sign after 3 of the opposite sign
59 +VOL_RECENT = 5 # sd of the last 5 yearly diffs …
60 +VOL_BASELINE = 15 # … vs the sd of the previous 15
61 +VOL_RATIO = 3.0
46 62
47 63 EVENT_COLUMNS = [
48 64 "id", "country_id", "indicator_id", "kind", "period", "year", "value", "ref_value", "delta", "delta_pct",
@@ -332,9 +348,96 @@ def detect_changes(s: Series, ind: Indicator) -> list[dict[str, Any]]:
332 348 f"{name} has fallen faster for three consecutive years, reaching {fmt_value(cur, ind)} in {year}.",
333 349 {"diffs": [float(x) for x in dd], "mode": mode}, 3, "changes")
334 350 )
351 +
352 + # --- change detection 2.0 -----------------------------------------------------------------------------------
353 + brk = _structural_break(s, ind, w)
354 + if brk is not None and year - brk["year"] <= BREAK_RECENT_YEARS:
355 + out.append(_break_row(s, ind, brk, "changes"))
356 + rev = _trend_reversal(s, ind, v, floor, floor_abs, mode)
357 + if rev is not None:
358 + out.append(rev)
359 + vol = _volatility_spike(s, ind, v, d, floor, floor_abs, mode)
360 + if vol is not None:
361 + out.append(vol)
335 362 return out
336 363
337 364
365 +# ------------------------------------------------------------------------------------------------ detectors 2.0
366 +def _structural_break(s: Series, ind: Indicator, w: np.ndarray) -> dict[str, Any] | None:
367 + """Single mean shift on the working-scale series (stats.structural_break): gain ≥ 0.5, |shift| ≥ 1.5 sd, segments ≥ 5.
368 + Returns {index, year, before, after (raw-scale means), gain, shift_ratio} or None. Monotone series are skipped (a trend
369 + is not a level shift)."""
370 + if len(w) < 2 * BREAK_MIN_SEGMENT or not np.all(np.isfinite(w)) or _is_monotone(s.values):
371 + return None
372 + b = stats.structural_break([float(x) for x in w], min_segment=BREAK_MIN_SEGMENT)
373 + if b is None or b["gain"] < BREAK_MIN_GAIN or b["shift_ratio"] < BREAK_MIN_SHIFT_RATIO:
374 + return None
375 + i = int(b["index"])
376 + raw = s.values
377 + return {"index": i, "year": int(s.years[i]), "before": float(np.nanmean(raw[:i])), "after": float(np.nanmean(raw[i:])),
378 + "gain": float(b["gain"]), "shift_ratio": float(b["shift_ratio"])}
379 +
380 +
381 +def _break_row(s: Series, ind: Indicator, brk: dict[str, Any], scope: str) -> dict[str, Any]:
382 + name = ind.display_name
383 + i = brk["index"]
384 + sev = min(0.7, 0.45 + 0.1 * (brk["gain"] - BREAK_MIN_GAIN) / 0.5 + 0.05 * min(3.0, brk["shift_ratio"] - BREAK_MIN_SHIFT_RATIO))
385 + direction = "higher" if brk["after"] > brk["before"] else "lower"
386 + headline = (f"{name} shifted to a {direction} level around {brk['year']}: average {fmt_value(brk['before'], ind)} before, "
387 + f"{fmt_value(brk['after'], ind)} since.")
388 + return _row(s, ind, "structural_break", i, brk["before"], sev, headline,
389 + {"break_year": brk["year"], "before_mean": brk["before"], "after_mean": brk["after"], "gain": round(brk["gain"], 3),
390 + "shift_ratio": round(brk["shift_ratio"], 2), "min_segment": BREAK_MIN_SEGMENT, "min_gain": BREAK_MIN_GAIN,
391 + "min_shift_ratio": BREAK_MIN_SHIFT_RATIO, "n_points": len(s.values)}, len(s.values) - i, scope)
392 +
393 +
394 +def _trend_reversal(s: Series, ind: Indicator, v: np.ndarray, floor: float, floor_abs: bool, mode: str) -> dict[str, Any] | None:
395 + """stats.persistent_reversal(run=3): the cumulative move over the last 3 years must also clear the indicator floor."""
396 + direction = stats.persistent_reversal([float(x) for x in v], run=REVERSAL_RUN)
397 + if direction is None:
398 + return None
399 + i = len(v) - 1
400 + cur, ref = float(v[i]), float(v[i - REVERSAL_RUN])
401 + mag = abs(cur - ref) if (floor_abs or mode != "relative") else abs(np.log(cur / ref)) if cur > 0 and ref > 0 else 0.0
402 + if not (mag >= floor > 0):
403 + return None
404 + name = ind.display_name
405 + now_verb, before_verb = ("risen", "falling") if direction == "up" else ("fallen", "rising")
406 + headline = (f"{name} has {now_verb} for three consecutive years after {before_verb} for three, reaching "
407 + f"{fmt_value(cur, ind)} in {int(s.years[i])}.")
408 + return _row(s, ind, "trend_reversal", i, ref, 0.55, headline,
409 + {"direction": direction, "run": REVERSAL_RUN, "turning_year": int(s.years[i - REVERSAL_RUN]), "mode": mode},
410 + REVERSAL_RUN, "changes")
411 +
412 +
413 +def _volatility_spike(s: Series, ind: Indicator, v: np.ndarray, d: np.ndarray, floor: float, floor_abs: bool,
414 + mode: str) -> dict[str, Any] | None:
415 + """sd of the last 5 yearly (working-scale) differences ≥ 3 × sd of the previous 15, and ≥ the indicator floor."""
416 + if len(d) < VOL_RECENT + VOL_BASELINE or not np.all(np.isfinite(d[-(VOL_RECENT + VOL_BASELINE):])):
417 + return None
418 + recent, base = d[-VOL_RECENT:], d[-(VOL_RECENT + VOL_BASELINE):-VOL_RECENT]
419 + sd_r, sd_b = float(np.std(recent, ddof=1)), float(np.std(base, ddof=1))
420 + if sd_b <= 0 or sd_r < VOL_RATIO * sd_b:
421 + return None
422 + floor_w = floor
423 + if floor_abs and mode == "relative":
424 + ref = float(np.nanmedian(np.abs(v[-VOL_RECENT - 1:])))
425 + floor_w = floor / ref if ref else 0.0
426 + if not (sd_r >= floor_w > 0):
427 + return None
428 + i = len(v) - 1
429 + name = ind.display_name
430 + ratio = sd_r / sd_b
431 + sev = min(0.7, 0.45 + 0.05 * min(5.0, ratio - VOL_RATIO))
432 + swing = fmt_delta(float(np.max(np.abs(np.diff(v[-VOL_RECENT - 1:])))), None if mode != "relative" else
433 + float(np.max(np.abs(recent)) * 100.0), ind)
434 + headline = (f"{name} has become unusually volatile: yearly moves over {int(s.years[i - VOL_RECENT])}–{int(s.years[i])} were "
435 + f"{ratio:.1f}× as dispersed as in the previous {VOL_BASELINE} years (largest swing {swing}).")
436 + return _row(s, ind, "volatility_spike", i, float(v[i - VOL_RECENT]), sev, headline,
437 + {"sd_recent": sd_r, "sd_baseline": sd_b, "ratio": round(ratio, 2), "recent_years": VOL_RECENT,
438 + "baseline_years": VOL_BASELINE, "min_ratio": VOL_RATIO, "mode": mode}, VOL_RECENT, "changes")
439 +
440 +
338 441 # ------------------------------------------------------------------------------------------------ events (history)
339 442 def detect_events(s: Series, ind: Indicator) -> list[dict[str, Any]]:
340 443 n = len(s.values)
@@ -401,6 +504,11 @@ def detect_events(s: Series, ind: Indicator) -> list[dict[str, Any]]:
401 504 f"{name} turned {direction} in {int(s.years[i])} ({fmt_value(float(v[i]), ind)}).",
402 505 {"direction": direction}, 1, "events"))
403 506
507 + # Structural break anywhere in the history (one per series at most)
508 + brk = _structural_break(s, ind, w)
509 + if brk is not None:
510 + out.append(_break_row(s, ind, brk, "events"))
511 +
404 512 if len(out) > MAX_EVENTS_PER_SERIES:
405 513 out.sort(key=lambda r: r["severity"], reverse=True)
406 514 out = out[:MAX_EVENTS_PER_SERIES]
modified src/countryatlas/pipeline/fetch.py +6 −5
@@ -25,7 +25,7 @@ from countryatlas.models import ImportRun, IndicatorSourceSpec, RawPayload
25 25 from countryatlas.pipeline import new_run_id
26 26 from countryatlas.pipeline.staging import (
27 27 latest_raw_payloads,
28 − read_previous_rows,
28 + read_previous_frame,
29 29 rows_to_frame,
30 30 spec_paths,
31 31 write_issues,
@@ -33,7 +33,7 @@ from countryatlas.pipeline.staging import (
33 33 write_parquet_atomic,
34 34 write_run,
35 35 )
36 −from countryatlas.pipeline.validate import validate_frame
36 +from countryatlas.pipeline.validate import PreviousStats, validate_frame
37 37 from countryatlas.registry import indicators_by_id, source_specs
38 38
39 39 log = logging.getLogger(__name__)
@@ -109,10 +109,11 @@ def process_spec(connector: Connector, spec: IndicatorSourceSpec, run_id: str, m
109 109 )
110 110 write_issues(spec, report.issues)
111 111 return _finish(spec, run)
112 − # 4. generic validation
112 + # 4. generic validation (schema, units, partial download, null spike, vintage shift, bounds, years, jumps, stale)
113 113 frame = rows_to_frame(rows)
114 − prev_rows = read_previous_rows(spec) if mode == "fetch" else None
115 − gv = validate_frame(frame, ind, previous_rows=prev_rows)
114 + prev_frame = read_previous_frame(spec) if mode == "fetch" else None
115 + previous = PreviousStats.from_frame(prev_frame) if prev_frame is not None else None
116 + gv = validate_frame(frame, ind, previous=previous)
116 117 issues = list(report.issues) + gv.issues
117 118 run.warnings = gv.warnings + report.warnings
118 119 run.errors = gv.errors + report.errors
modified src/countryatlas/pipeline/similarity.py +6 −1
@@ -4,7 +4,8 @@ Inputs are the `latest` values (non-forecast, non-quarantined latest observation
4 4 transformed (log / log1p), z-scored across countries (nan-aware), and compared with a weighted Euclidean distance over
5 5 the features both countries have (rescaled to the mode's full weight; a pair needs ≥ 50 % of the weight in common).
6 6 score = 100·exp(−d/d0), d0 = median of all pairwise distances of the mode. Top 12 peers stored, with per-feature
7 −contributions (share of the squared distance) so every score is explainable.
7 +contributions (share of the squared distance, z-scores AND the raw latest values `value_a` / `value_b`) so every score is
8 +explainable in plain numbers.
8 9 """
9 10 from __future__ import annotations
10 11
@@ -76,6 +77,8 @@ def compute_similarity(latest_df: pl.DataFrame, countries: list[str]) -> pl.Data
76 77 names = [f["indicator"] if not f.get("per") else f"{f['indicator']}/{f['per']}" for f in feats]
77 78 weights = np.array([float(f.get("weight", 1.0)) for f in feats])
78 79 m = np.column_stack([_feature_vector(latest, countries, f) for f in feats]) # (n_countries, n_features)
80 + # raw values (after the optional `per` ratio, BEFORE the log transform) so the UI can quote "US$55k vs US$65k"
81 + raw = np.column_stack([_feature_vector(latest, countries, {**f, "transform": "none"}) for f in feats])
79 82 present = np.isfinite(m)
80 83 coverage = (present * weights).sum(axis=1) / weights.sum()
81 84 eligible = coverage >= MIN_FEATURE_COVERAGE
@@ -117,6 +120,8 @@ def compute_similarity(latest_df: pl.DataFrame, countries: list[str]) -> pl.Data
117 120 names[k]: {
118 121 "z_a": round(float(z[a, k]), 3) if present[a, k] else None,
119 122 "z_b": round(float(z[b, k]), 3) if present[b, k] else None,
123 + "value_a": float(raw[a, k]) if present[a, k] and np.isfinite(raw[a, k]) else None,
124 + "value_b": float(raw[b, k]) if present[b, k] and np.isfinite(raw[b, k]) else None,
120 125 "weight": float(weights[k]),
121 126 "contribution": round(float(contrib_raw[k] / total), 4) if shared[a_pos, b_pos, k] else None,
122 127 }
modified src/countryatlas/pipeline/staging.py +12 −0
@@ -137,6 +137,18 @@ def read_previous_rows(spec: IndicatorSourceSpec) -> int | None:
137 137 return None
138 138
139 139
140 +def read_previous_frame(spec: IndicatorSourceSpec) -> pl.DataFrame | None:
141 + """Key columns + value of the previous staging file for this spec (for null-spike / vintage-shift checks)."""
142 + p = spec_paths(spec)["parquet"]
143 + if not p.exists():
144 + return None
145 + try:
146 + return pl.read_parquet(p, columns=["country_id", "period", "frequency", "value"])
147 + except Exception as e: # noqa: BLE001
148 + log.warning("previous staging file unreadable (%s): %s", p, e)
149 + return None
150 +
151 +
140 152 def list_staging_files(connector: str | None = None) -> list[Path]:
141 153 base = settings.staging_dir
142 154 if not base.exists():
modified src/countryatlas/pipeline/validate.py +112 −15
@@ -1,13 +1,18 @@
1 1 """Generic, deterministic validation rules (ARCHITECTURE §6) applied to one staging frame (one spec).
2 2
3 3 Row-level outcomes only ever *flag* (status column); nothing is deleted:
4 − out_of_bounds value outside registry `bounds` → status quarantined
5 − extreme_jump |Δ| > jump_threshold × 1.4826·MAD(country diffs), ≥5 points → status warning
6 − stale the country's latest period ended > stale_after_days ago → status stale (that latest row only)
4 + out_of_bounds value outside registry `bounds` or non-finite → status quarantined
5 + impossible_year year < 1750 or > current year + 10 → status quarantined
6 + extreme_jump |Δ| > jump_threshold × 1.4826·MAD(country diffs), ≥5 points → status warning
7 + stale the country's latest period ended > stale_after_days ago → status stale (that latest row only)
7 8 Dataset-level outcomes (quarantine_dataset=True → the pipeline keeps the previous staging file):
8 − duplicate same key twice (connector.validate already errors on this)
9 − unit_mismatch rows whose `unit` differs from the registry unit
9 + schema_change a required column is missing or its dtype is incompatible with the staging schema
10 + duplicate same key twice (connector.validate already errors on this)
11 + unit_mismatch rows whose `unit` differs from the registry unit
10 12 partial_download rows < 30 % of the previous staging file for the same spec
13 +Dataset-level warnings (kept, logged, surfaced in validation_issues):
14 + null_spike share of null values > 3 × the previous run's share AND > 5 %
15 + vintage_shift median |relative change| of the keys shared with the previous staging file > 25 %
11 16 """
12 17 from __future__ import annotations
13 18
@@ -28,15 +33,48 @@ MIN_POINTS_FOR_JUMP = 5
28 33 MAX_ROW_ISSUES = 2000 # per code per spec, to keep validation_issues manageable
29 34 LOG_FLOOR = 0.10 # log-diff floor ≈ 10 % relative change
30 35 RANGE_FLOOR = 0.02 # absolute-diff floor = 2 % of the country's series range
36 +MIN_YEAR = 1750
37 +FUTURE_YEARS = 10 # years beyond the current year still accepted (projections)
38 +NULL_SPIKE_RATIO = 3.0
39 +NULL_SPIKE_MIN_SHARE = 0.05
40 +VINTAGE_SHIFT_MAX = 0.25 # median |rel. change| over overlapping keys
41 +VINTAGE_MIN_OVERLAP = 20
31 42 # formats whose level series grow multiplicatively → compare on log-differences when the registry lower bound is ≥ 0
32 43 LOG_FORMATS = {"currency", "number", "tonnes", "kwh", "per_1000", "per_100k", "per_million", "km", "ha"}
33 44
45 +REQUIRED_COLUMNS: dict[str, tuple[type[pl.DataType], ...]] = {
46 + "country_id": (pl.Utf8,),
47 + "indicator_id": (pl.Utf8,),
48 + "period": (pl.Date,),
49 + "year": (pl.Int8, pl.Int16, pl.Int32, pl.Int64, pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64),
50 + "frequency": (pl.Utf8,),
51 + "value": (pl.Float32, pl.Float64, pl.Int32, pl.Int64),
52 + "unit": (pl.Utf8,),
53 + "source_id": (pl.Utf8,),
54 +}
55 +
34 56
35 57 def uses_log_diffs(indicator: Indicator) -> bool:
36 58 lo = (indicator.bounds or [None, None])[0]
37 59 return indicator.format in LOG_FORMATS and lo is not None and float(lo) >= 0
38 60
39 61
62 +@dataclass
63 +class PreviousStats:
64 + """What the previous staging file for the same spec looked like (for partial / null-spike / vintage-shift checks)."""
65 +
66 + rows: int
67 + null_share: float = 0.0
68 + values: pl.DataFrame | None = None # country_id, period, frequency, value
69 +
70 + @classmethod
71 + def from_frame(cls, df: pl.DataFrame) -> PreviousStats:
72 + n = df.height
73 + nulls = int(df.get_column("value").null_count()) if "value" in df.columns and n else 0
74 + cols = [c for c in ("country_id", "period", "frequency", "value") if c in df.columns]
75 + return cls(rows=n, null_share=(nulls / n) if n else 0.0, values=df.select(cols) if len(cols) == 4 else None)
76 +
77 +
40 78 @dataclass
41 79 class GenericValidation:
42 80 frame: pl.DataFrame
@@ -66,24 +104,52 @@ def _period_end(freq: str) -> pl.Expr:
66 104 return p.dt.offset_by("1y") - pl.duration(days=1)
67 105
68 106
107 +def check_schema(df: pl.DataFrame) -> list[str]:
108 + """Names of required columns that are missing or have an incompatible dtype (empty = OK)."""
109 + problems: list[str] = []
110 + for col, dtypes in REQUIRED_COLUMNS.items():
111 + if col not in df.columns:
112 + problems.append(f"missing column {col}")
113 + continue
114 + dt = df.schema[col]
115 + if not any(dt == d for d in dtypes):
116 + problems.append(f"column {col} has dtype {dt}")
117 + return problems
118 +
119 +
69 120 def validate_frame(
70 121 df: pl.DataFrame,
71 122 indicator: Indicator,
72 123 previous_rows: int | None = None,
73 124 now: datetime | None = None,
74 125 check_partial: bool = True,
126 + previous: PreviousStats | None = None,
75 127 ) -> GenericValidation:
76 − """Apply the generic rules and return the frame with updated `status` + issues."""
128 + """Apply the generic rules and return the frame with updated `status` + issues.
129 +
130 + `previous` (stats of the previous staging file) enables the partial-download, null-spike and vintage-shift checks;
131 + `previous_rows` is kept for backward compatibility (partial-download only).
132 + """
77 133 now = now or datetime.now(UTC)
78 134 issues: list[ValidationIssue] = []
79 135 quarantine = False
80 136 message: str | None = None
81 137 ind_id = indicator.slug
82 138 n = df.height
139 + if previous is not None and previous_rows is None:
140 + previous_rows = previous.rows
83 141
84 142 if n == 0:
85 143 return GenericValidation(frame=df, issues=issues, quarantine_dataset=False, message="no rows")
86 144
145 + # --- dataset-level: schema -----------------------------------------------------------------------------------
146 + schema_problems = check_schema(df)
147 + if schema_problems:
148 + message = "schema change: " + "; ".join(schema_problems)
149 + issues.append(ValidationIssue(severity="error", code="schema_change", message=message, indicator_id=ind_id))
150 + log.error("%s: %s", ind_id, message)
151 + return GenericValidation(frame=df, issues=issues, quarantine_dataset=True, message=message)
152 +
87 153 # --- dataset-level: unit mismatch -------------------------------------------------------------------------
88 154 units = df.get_column("unit").drop_nulls().unique().to_list()
89 155 bad_units = [u for u in units if u != indicator.unit]
@@ -109,7 +175,27 @@ def validate_frame(
109 175 country_id=r["country_id"], period=r["period"])
110 176 )
111 177
112 − # --- row-level: bounds -------------------------------------------------------------------------------------
178 + # --- dataset-level warnings vs the previous vintage: null spike, vintage shift -----------------------------
179 + if previous is not None:
180 + null_share = df.get_column("value").null_count() / n
181 + if null_share > NULL_SPIKE_MIN_SHARE and null_share > NULL_SPIKE_RATIO * previous.null_share:
182 + msg = (f"null spike: {null_share:.1%} of values are null (previous run {previous.null_share:.1%})")
183 + issues.append(ValidationIssue(severity="warning", code="null_spike", message=msg, indicator_id=ind_id))
184 + log.warning("%s: %s", ind_id, msg)
185 + if previous.values is not None and previous.values.height:
186 + joined = df.select("country_id", "period", "frequency", "value").join(
187 + previous.values.rename({"value": "_prev"}), on=["country_id", "period", "frequency"], how="inner"
188 + ).filter(pl.col("value").is_not_null() & pl.col("_prev").is_not_null() & (pl.col("_prev") != 0))
189 + if joined.height >= VINTAGE_MIN_OVERLAP:
190 + rel = ((pl.col("value") - pl.col("_prev")).abs() / pl.col("_prev").abs())
191 + med = float(joined.select(rel.median()).item())
192 + if med > VINTAGE_SHIFT_MAX:
193 + msg = (f"vintage shift: median |relative change| of {joined.height} overlapping values is {med:.1%} "
194 + f"(> {VINTAGE_SHIFT_MAX:.0%}) — source revised or changed definition?")
195 + issues.append(ValidationIssue(severity="warning", code="vintage_shift", message=msg, indicator_id=ind_id))
196 + log.warning("%s: %s", ind_id, msg)
197 +
198 + # --- row-level: bounds + impossible years --------------------------------------------------------------------
113 199 lo, hi = (indicator.bounds or [None, None])[:2]
114 200 oob = pl.lit(False)
115 201 if lo is not None:
@@ -117,7 +203,8 @@ def validate_frame(
117 203 if hi is not None:
118 204 oob = oob | (pl.col("value") > float(hi))
119 205 oob = oob | ~pl.col("value").is_finite()
120 − df = df.with_columns(oob.alias("_oob"))
206 + bad_year = (pl.col("year") < MIN_YEAR) | (pl.col("year") > now.year + FUTURE_YEARS)
207 + df = df.with_columns(oob.fill_null(True).alias("_oob"), bad_year.fill_null(False).alias("_bad_year"))
121 208
122 209 # --- row-level: extreme jumps (per country series, robust MAD of first differences) -----------------------
123 210 # Level series that grow multiplicatively (GDP, population, emissions…) are compared on log-differences so that a
@@ -158,26 +245,33 @@ def validate_frame(
158 245 df = df.with_columns(stale.alias("_stale"))
159 246
160 247 # --- statuses: quarantined > warning > stale > imported -----------------------------------------------------
248 + quarantined = pl.col("_oob") | pl.col("_bad_year")
161 249 df = df.with_columns(
162 − pl.when(pl.col("_oob"))
250 + pl.when(quarantined)
163 251 .then(pl.lit("quarantined"))
164 252 .when(pl.col("_jump"))
165 253 .then(pl.lit("warning"))
166 254 .when(pl.col("_stale"))
167 255 .then(pl.lit("stale"))
168 256 .otherwise(pl.lit("imported"))
169 − .alias("status")
257 + .alias("status"),
258 + quarantined.alias("_q"),
170 259 )
171 − n_q = int(df.get_column("_oob").sum())
172 − n_w = int((df.get_column("_jump") & ~df.get_column("_oob")).sum())
173 − n_s = int((df.get_column("_stale") & ~df.get_column("_oob") & ~df.get_column("_jump")).sum())
260 + n_q = int(df.get_column("_q").sum())
261 + n_w = int((df.get_column("_jump") & ~df.get_column("_q")).sum())
262 + n_s = int((df.get_column("_stale") & ~df.get_column("_q") & ~df.get_column("_jump")).sum())
174 263
175 264 for r in df.filter(pl.col("_oob")).head(MAX_ROW_ISSUES).iter_rows(named=True):
176 265 issues.append(
177 266 ValidationIssue(severity="warning", code="out_of_bounds", indicator_id=ind_id, country_id=r["country_id"],
178 267 period=r["period"], message=f"value {r['value']!r} outside bounds [{lo}, {hi}]")
179 268 )
180 − for r in df.filter(pl.col("_jump") & ~pl.col("_oob")).head(MAX_ROW_ISSUES).iter_rows(named=True):
269 + for r in df.filter(pl.col("_bad_year") & ~pl.col("_oob")).head(MAX_ROW_ISSUES).iter_rows(named=True):
270 + issues.append(
271 + ValidationIssue(severity="warning", code="impossible_year", indicator_id=ind_id, country_id=r["country_id"],
272 + period=r["period"], message=f"year {r['year']} outside [{MIN_YEAR}, {now.year + FUTURE_YEARS}]")
273 + )
274 + for r in df.filter(pl.col("_jump") & ~pl.col("_q")).head(MAX_ROW_ISSUES).iter_rows(named=True):
181 275 issues.append(
182 276 ValidationIssue(severity="warning", code="extreme_jump", indicator_id=ind_id, country_id=r["country_id"],
183 277 period=r["period"],
@@ -194,7 +288,10 @@ def validate_frame(
194 288 ValidationIssue(severity="warning", code="stale", indicator_id=ind_id,
195 289 message=f"dataset latest period {latest_period} is older than {stale_days} days")
196 290 )
291 + if n_q:
292 + log.warning("%s: %d rows quarantined (out of bounds / impossible year) — kept for audit, excluded from derived tables",
293 + ind_id, n_q)
197 294
198 − out = df.drop(["_oob", "_jump", "_diff", "_thr", "_stale"])
295 + out = df.drop(["_oob", "_bad_year", "_q", "_jump", "_diff", "_thr", "_stale"])
199 296 return GenericValidation(frame=out, issues=issues, quarantine_dataset=quarantine, message=message,
200 297 n_quarantined=n_q, n_warning=n_w, n_stale=n_s)
added tests/api/test_analytics.py +207 −0
@@ -0,0 +1,207 @@
1 +"""Analytics endpoints (API 1.1) on the synthetic fixture (8 countries × 14 indicators, 1990–2024)."""
2 +from __future__ import annotations
3 +
4 +import pytest
5 +
6 +from tests.api.conftest import assert_meta, assert_provenance, value
7 +
8 +
9 +def test_pulse(get):
10 + body = get("/pulse").json()
11 + assert_meta(body)
12 + assert body["year_reference"] == 2024
13 + # fixture: 8 countries only, below the 30-country pool threshold → no items but a valid envelope
14 + assert body["items"] == [] and body["summary"]["n_countries_reporting"] == 0
15 + assert "n_record_highs" in body["summary"]
16 +
17 +
18 +def test_movers_window_1_and_kinds(get):
19 + body = get("/movers?window=1&limit=10").json()
20 + assert body["window"] == 1 and body["categories"][0] == "all" and "structural" in body["kinds"]
21 + assert body["n"] == len(body["items"]) > 0
22 + it = body["items"][0]
23 + for k in ("country", "indicator", "kind", "value", "delta", "direction", "interpretation", "headline", "provenance"):
24 + assert k in it
25 + assert it["direction"] in ("up", "down")
26 + assert_provenance(it["provenance"])
27 + rec = get("/movers?window=1&kind=record&limit=50").json()
28 + assert rec["items"] and all(i["kind"] in ("record_high", "record_low") for i in rec["items"])
29 + imp = get("/movers?window=1&kind=improvement&limit=50").json()
30 + assert all(i["interpretation"] == "improvement" for i in imp["items"])
31 + econ = get("/movers?window=1&category=economic&limit=50").json()
32 + assert all(i["indicator"]["topic"] in ("economy", "government", "trade", "income") for i in econ["items"])
33 + get("/movers?window=3", status=400)
34 + get("/movers?category=weird", status=400)
35 +
36 +
37 +def test_movers_multi_year_window_needs_pool(get):
38 + body = get("/movers?window=10&limit=10").json() # 8 countries < 20 → empty but valid
39 + assert body["window"] == 10 and body["items"] == []
40 + body = get("/movers?window=5&kind=record").json()
41 + assert body["items"] == []
42 +
43 +
44 +def test_extremes_envelope(get):
45 + body = get("/extremes?window=10").json()
46 + assert body["window"] == "10" and body["filter_note"] == "Countries above 1M inhabitants"
47 + assert body["facets"] == [] # every facet needs ≥ 30 countries in the window
48 + get("/extremes?window=7", status=400)
49 +
50 +
51 +def test_scatter(get):
52 + body = get("/scatter?x=gdp-per-capita&y=life-expectancy&year=2024&log_x=true").json()
53 + assert_meta(body)
54 + assert body["year_used"] == 2024 and body["n"] == 8 and body["size"]["id"] == "population"
55 + p = next(pt for pt in body["points"] if pt["id"] == "CAN")
56 + assert p["x"] == pytest.approx(value("CAN", "gdp-per-capita", 2024)) and p["y"] == pytest.approx(value("CAN", "life-expectancy", 2024))
57 + assert p["size"] == pytest.approx(value("CAN", "population", 2024)) and p["year_x"] == 2024
58 + st = body["stats"]
59 + assert st["n"] == 8 and -1 <= st["pearson"] <= 1 and -1 <= st["spearman"] <= 1 and st["ols"]["slope"] is not None
60 + assert st["log_x"] is True and st["log_y"] is False
61 + assert "causation" in body["note"]
62 + none = get("/scatter?x=gdp-per-capita&y=life-expectancy&size=none").json()
63 + assert none["size"] is None and none["stats"]["log_x"] is False and none["year_used"] == 2024 # auto: ratio max/min < 50 in the fixture
64 + get("/scatter?x=gdp&y=gdp", status=400)
65 + get("/scatter?x=gdp&y=nope", status=404)
66 + grp = get("/scatter?x=gdp-per-capita&y=life-expectancy&group=g7").json()
67 + assert grp["group"]["id"] == "g7" and grp["n"] == 5
68 +
69 +
70 +def test_trajectory(get):
71 + body = get("/trajectory?x=gdp-per-capita&y=life-expectancy&from=2000&to=2005").json()
72 + assert body["years"] == [2000, 2001, 2002, 2003, 2004, 2005]
73 + assert len(body["countries"]) == 8 and set(body["series"]) == {c["id"] for c in body["countries"]}
74 + s = body["series"]["IND"]
75 + assert len(s["x"]) == 6 and s["x"][0] == pytest.approx(value("IND", "gdp-per-capita", 2000))
76 + assert s["size"][0] == pytest.approx(value("IND", "population", 2000))
77 + assert body["domains"]["x"][0] <= min(v for v in s["x"]) and body["log_x"] is True # pooled x spans > 50× → log
78 + assert body["provenance"]
79 + auto = get("/trajectory?x=gdp-per-capita&y=life-expectancy").json()
80 + assert auto["years"][0] == 1990 and auto["years"][-1] == 2024 # < 50 countries → every year with data
81 + get("/trajectory?x=gdp&y=gdp", status=400)
82 +
83 +
84 +def test_finder(get):
85 + body = get("/finder?f=gdp-per-capita:gt:40000&f=population:gt:100000000").json()
86 + assert body["mode"] == "and" and [f["indicator"]["id"] for f in body["filters"]] == ["gdp-per-capita", "population"]
87 + assert body["filters"][0]["op"] == "gt" and body["filters"][0]["value"] == 40000 and body["filters"][0]["year_used"] == 2024
88 + ids = [i["country"]["id"] for i in body["items"]]
89 + assert ids == ["JPN", "USA"] # rich (2024 gpc > 40k) and populous; sorted by gpc desc (JPN 69k > USA 65k in the fixture)
90 + assert body["n_matching"] == 2 and body["n_evaluated"] == 8
91 + assert body["items"][0]["matched"] == ["gdp-per-capita", "population"]
92 + assert_provenance(body["items"][0]["values"]["gdp-per-capita"]["provenance"])
93 + any_ = get("/finder?f=gdp-per-capita:gt:40000&f=population:gt:1000000000&mode=or").json()
94 + assert "IND" in [i["country"]["id"] for i in any_["items"]] # India by population only
95 + between = get("/finder?f=life-expectancy:between:60..70&sort=population:asc").json()
96 + assert all(60 <= i["values"]["life-expectancy"]["value"] <= 70 for i in between["items"])
97 + reg = get("/finder?f=population:gt:1&region=g7").json()
98 + assert reg["n_matching"] == 5
99 + get("/finder", status=400)
100 + get("/finder?f=gdp-per-capita:huge:1", status=400)
101 + get("/finder?f=nope:gt:1", status=404)
102 +
103 +
104 +def test_peers(get):
105 + body = get("/peers?y=life-expectancy&x=gdp-per-capita&log_x=true").json()
106 + assert body["year_used"] == 2024 and body["n"] == 8 and body["method"] == "theil-sen"
107 + assert body["fit"]["log_x"] is True and body["fit"]["residual_scale"] is not None
108 + pts = {p["id"]: p for p in body["points"]}
109 + assert pts["CAN"]["expected"] is not None and pts["CAN"]["residual"] == pytest.approx(pts["CAN"]["y"] - pts["CAN"]["expected"])
110 + assert body["above"][0]["residual_z"] >= body["above"][-1]["residual_z"] and body["below"][0]["residual_z"] <= body["below"][-1]["residual_z"]
111 + assert body["pairs"] and "descriptive" in body["note"]
112 + ols = get("/peers?y=life-expectancy&x=gdp-per-capita&method=ols").json()
113 + assert ols["fit"]["r2"] is not None
114 + get("/peers?y=gdp&x=gdp", status=400)
115 +
116 +
117 +def test_related_distribution_frames(get):
118 + rel = get("/indicators/life-expectancy/related?min_n=5").json()
119 + assert rel["indicator"]["id"] == "life-expectancy" and rel["items"] and "causation" in rel["note"]
120 + assert all(i["indicator"]["id"] != "life-expectancy" for i in rel["items"])
121 + assert abs(rel["items"][0]["spearman"]) >= abs(rel["items"][-1]["spearman"])
122 + assert get("/indicators/life-expectancy/related").json()["items"] == [] # min_n 40 > 8 countries
123 +
124 + d = get("/indicators/life-expectancy/distribution?highlight=JPN").json()
125 + assert d["year_used"] == 2024 and d["n"] == 8 and sum(d["histogram"]["counts"]) == 8 and len(d["histogram"]["edges"]) == 21
126 + assert d["stats"]["min"] <= d["stats"]["median"] <= d["stats"]["max"]
127 + assert d["highlight"]["country"]["id"] == "JPN" and d["highlight"]["rank"] == 1 and d["highlight"]["percentile"] == pytest.approx(100.0)
128 + assert d["by_income"] and d["by_region"]
129 + assert_provenance(d["provenance"])
130 + lg = get("/indicators/gdp-per-capita/distribution?bins=5").json()
131 + assert lg["log"] is False and len(lg["histogram"]["counts"]) == 5 # auto-log needs a > 50× spread; the fixture's 2024 spread is ~25×
132 +
133 + fr = get("/indicators/gdp-per-capita/frames?from=2000&to=2003").json()
134 + assert fr["years"] == [] or fr["years"][0] >= 2000 # fixture has 8 countries (< 20 per year) → no frames
135 + assert fr["legend"]["n_classes"] >= 1 and isinstance(fr["values"], dict)
136 +
137 +
138 +def test_quality_and_updates(get):
139 + q = get("/indicators/life-expectancy/quality").json()
140 + assert q["n_countries"] == 8 and q["n_countries_total"] == 8 and q["coverage_pct"] == 100.0
141 + assert q["first_year"] == 1990 and q["last_year"] == 2024 and q["years_with_50plus"] == 0
142 + assert "fresh" in q["badges"] and "historical" not in q["badges"] and q["sources"]
143 + get("/indicators/nope/quality", status=404)
144 + cq = get("/countries/canada/quality").json()
145 + assert cq["country"]["id"] == "CAN" and cq["summary"]["n_with_data"] == 14 and cq["summary"]["latest_year"] == 2024
146 + it = next(i for i in cq["items"] if i["indicator"]["id"] == "government-debt-pct-gdp")
147 + assert it["first_year"] == 1995 and it["n_points"] == 30 and it["missing_years"] == 0 and it["continuity_pct"] == 100.0
148 + assert "fresh" in it["badges"]
149 + u = get("/updates").json()
150 + assert_meta(u)
151 + assert u["snapshot"]["observations"] > 0 and u["snapshot"]["values_changed"] == 0
152 + ids = {s["source"]["id"]: s for s in u["sources"]}
153 + assert ids["worldbank"]["status"] == "ok" and ids["who"]["status"] == "failed" and ids["owid"]["status"] == "partial"
154 + assert ids["worldbank"]["n_observations"] > 0 and ids["worldbank"]["latest_year"] == 2024
155 + assert len(u["recent_runs"]) == 6 and all("raw_path" not in r for r in u["recent_runs"])
156 + assert u["indicators_recently_updated"]
157 +
158 +
159 +def test_race_and_regions_compare(get):
160 + r = get("/rankings/gdp/race?top=3&from=2020").json()
161 + assert r["years"] == [] or r["frames"][0]["year"] >= 2020 # fixture rankings have 8 countries (< 20) → no frames
162 + assert r["top"] == 3 and r["indicator"]["id"] == "gdp" and r["group"]["id"] == "world"
163 + get("/rankings/nope/race", status=404)
164 + rc = get("/regions/compare?a=g7&b=brics").json()
165 + assert [g["id"] for g in rc["groups"]] == ["g7", "brics"]
166 + row = next(x for x in rc["rows"] if x["indicator"]["id"] == "population")
167 + assert row["kind"] == "sum" and row["values"]["g7"]["n"] == 5 and row["values"]["brics"]["n"] == 2
168 + assert rc["shares"]["g7"]["population_share_pct"] + rc["shares"]["brics"]["population_share_pct"] < 100.01
169 + h = rc["history"]["gdp-per-capita"]
170 + assert h["kind"] == "weighted_mean" and h["years"][0] == 1990 and len(h["g7"]) == len(h["years"])
171 + get("/regions/compare?a=nope&b=g7", status=404)
172 +
173 +
174 +def test_story_and_dna_reference(get):
175 + s = get("/countries/canada/story").json()
176 + assert s["country"]["id"] == "CAN" and s["since"] == 1990 and s["items"]
177 + pop = next(i for i in s["items"] if i["indicator"]["id"] == "population")
178 + assert pop["first"]["year"] == 1990 and pop["last"]["year"] == 2024 and pop["change_pct"] > 0 and pop["cagr"] == pytest.approx(1.0, abs=0.05)
179 + assert pop["rank_first"]["year"] == 1990 and pop["rank_last"]["rank"] >= 1 and len(pop["series"]) == 35
180 + assert pop["text"].startswith("Canada: Population") and "1990" in pop["text"] and "2024" in pop["text"]
181 + assert_provenance(pop["provenance"])
182 + le = next(i for i in s["items"] if i["indicator"]["id"] == "life-expectancy")
183 + assert le["cagr"] is None and "yrs" in le["text"]
184 + d = get("/countries/canada/dna?reference=world").json()
185 + assert d["reference"]["kind"] == "world" and set(d["reference"]["dims"].values()) == {50.0}
186 + d = get("/countries/canada/dna?reference=income").json()
187 + assert d["reference"]["kind"] == "income" and d["reference"]["id"] == "hic" and d["reference"]["dims"]["income"] is not None
188 + d = get("/countries/canada/dna?reference=fra").json()
189 + assert d["reference"]["kind"] == "country" and d["reference"]["id"] == "FRA"
190 + assert get("/countries/canada/dna").json()["reference"] is None
191 + get("/countries/canada/dna?reference=xxx", status=404)
192 +
193 +
194 +def test_search_intents(get):
195 + hits = get("/search?q=compare%20canada%20usa").json()["hits"]
196 + assert hits[0]["type"] == "action" and hits[0]["action"] == "compare" and hits[0]["url"] == "/compare/canada/united-states"
197 + hits = get("/search?q=canada%20vs%20france%20vs%20japan").json()["hits"]
198 + assert hits[0]["url"] == "/compare/canada/france/japan"
199 + hits = get("/search?q=rank%20gdp").json()["hits"]
200 + assert hits[0]["action"] == "ranking" and hits[0]["url"] == "/rankings/gdp"
201 + hits = get("/search?q=inflation%20g7").json()["hits"]
202 + assert hits[0]["action"] == "group_ranking" and hits[0]["url"] == "/rankings/inflation?group=g7"
203 + hits = get("/search?q=life%20expectancy%20map").json()["hits"]
204 + assert hits[0]["action"] == "explore" and hits[0]["url"] == "/explore?indicator=life-expectancy"
205 + # plain queries are unchanged
206 + assert get("/search?q=canada").json()["hits"][0]["type"] == "country"
207 + assert all(h["type"] != "action" for h in get("/search?q=gdp&type=indicator").json()["hits"])
modified tests/test_changes.py +73 −0
@@ -127,3 +127,76 @@ def test_driver_over_frame() -> None:
127 127 assert ch.filter(pl.col("kind") == "yoy_drop").height == 2
128 128 assert set(ch.columns) >= {"id", "country_id", "indicator_id", "kind", "headline", "severity", "detected_at"}
129 129 assert ev.height >= 2
130 +
131 +
132 +# ------------------------------------------------------------------------------------------------ change detection 2.0
133 +def test_structural_break_in_changes_and_events() -> None:
134 + ind = indicators_by_id()["unemployment-rate"]
135 + # 12 years around 5 % (with small noise so the series is not monotone), then 8 years around 12 % → break in 2012
136 + vals = [5.0, 5.2, 4.9, 5.1, 5.0, 5.3, 4.8, 5.1, 5.0, 5.2, 4.9, 5.1, 12.0, 12.2, 11.9, 12.1, 12.0, 12.3, 11.8, 12.1]
137 + ch = detect_changes(_series("unemployment-rate", vals, 2000), ind) # last year 2019, break 2012 → within 10 years
138 + brk = [r for r in ch if r["kind"] == "structural_break"]
139 + assert len(brk) == 1
140 + b = brk[0]
141 + assert b["year"] == 2012 and "shifted to a higher level around 2012" in b["headline"]
142 + assert 0.45 <= b["severity"] <= 0.7
143 + import json as _json
144 +
145 + d = b["detail"] if isinstance(b["detail"], dict) else _json.loads(b["detail"])
146 + assert d["break_year"] == 2012 and d["gain"] >= 0.5 and d["shift_ratio"] >= 1.5 and d["min_segment"] == 5
147 + ev = detect_events(_series("unemployment-rate", vals, 2000), ind)
148 + assert len([r for r in ev if r["kind"] == "structural_break"]) == 1
149 + # an old break (25 years ago) stays an event but is not a recent change
150 + old = vals + [12.0, 12.2, 11.9, 12.1, 12.0, 12.3, 11.8, 12.1, 12.0, 12.2, 11.9, 12.1, 12.0, 12.3, 11.8]
151 + ch_old = detect_changes(_series("unemployment-rate", old, 1988), ind) # last year 2022, break 2000 → 22 years ago
152 + assert not any(r["kind"] == "structural_break" for r in ch_old)
153 + assert any(r["kind"] == "structural_break" for r in detect_events(_series("unemployment-rate", old, 1988), ind))
154 + # a smooth trend is not a level shift
155 + trend = [5.0 + 0.3 * k for k in range(20)]
156 + assert not any(r["kind"] == "structural_break" for r in detect_changes(_series("unemployment-rate", trend, 2000), ind))
157 +
158 +
159 +def test_trend_reversal() -> None:
160 + ind = indicators_by_id()["unemployment-rate"] # floor 1 point
161 + vals = [8.0, 8.0, 8.0, 9.0, 8.0, 7.0, 6.0, 7.0, 8.0, 9.0] # 3 falls (9→6) then 3 rises (6→9): +3 points ≥ floor
162 + ch = detect_changes(_series("unemployment-rate", vals, 2010), ind)
163 + rev = [r for r in ch if r["kind"] == "trend_reversal"]
164 + assert len(rev) == 1
165 + assert rev[0]["headline"].startswith("Unemployment has risen for three consecutive years after falling for three")
166 + assert rev[0]["severity"] == pytest.approx(0.55) and rev[0]["ref_value"] == 6.0
167 + down = [3.0, 3.0, 3.0, 2.0, 3.0, 4.0, 5.0, 4.0, 3.0, 2.0]
168 + r2 = [r for r in detect_changes(_series("unemployment-rate", down, 2010), ind) if r["kind"] == "trend_reversal"]
169 + assert len(r2) == 1 and "has fallen for three consecutive years after rising for three" in r2[0]["headline"]
170 + # too small a swing (0.3 points) is silent
171 + tiny = [8.0, 8.0, 8.0, 8.3, 8.2, 8.1, 8.0, 8.1, 8.2, 8.3]
172 + assert not any(r["kind"] == "trend_reversal" for r in detect_changes(_series("unemployment-rate", tiny, 2010), ind))
173 +
174 +
175 +def test_volatility_spike() -> None:
176 + ind = indicators_by_id()["inflation"] # floor 2 points
177 + calm = [2.0 + 0.1 * ((-1) ** k) for k in range(16)] # 15 tiny diffs (±0.2)
178 + wild = [2.0, 9.0, 1.0, 8.0, 0.0] # 5 large swings
179 + ch = detect_changes(_series("inflation", calm + wild, 2005), ind)
180 + vol = [r for r in ch if r["kind"] == "volatility_spike"]
181 + assert len(vol) == 1
182 + assert "unusually volatile" in vol[0]["headline"] and 0.45 <= vol[0]["severity"] <= 0.7
183 + import json as _json
184 +
185 + d = vol[0]["detail"] if isinstance(vol[0]["detail"], dict) else _json.loads(vol[0]["detail"])
186 + assert d["ratio"] >= 3.0 and d["recent_years"] == 5 and d["baseline_years"] == 15
187 + # same pattern but the recent swings are below the 2-point floor → silent
188 + small = calm + [2.0, 2.5, 2.0, 2.6, 2.1]
189 + assert not any(r["kind"] == "volatility_spike" for r in detect_changes(_series("inflation", small, 2005), ind))
190 + # too short a history → silent
191 + assert not any(r["kind"] == "volatility_spike" for r in detect_changes(_series("inflation", wild * 2, 2015), ind))
192 +
193 +
194 +def test_new_kinds_flow_through_driver_with_weights() -> None:
195 + inds = indicators_by_id()
196 + vals = [5.0, 5.2, 4.9, 5.1, 5.0, 5.3, 4.8, 5.1, 5.0, 5.2, 4.9, 5.1, 12.0, 12.2, 11.9, 12.1, 12.0, 12.3, 11.8, 12.1]
197 + rows = [{"country_id": "CAN", "indicator_id": "unemployment-rate", "period": date(2006 + k, 1, 1), "year": 2006 + k, "value": v}
198 + for k, v in enumerate(vals)]
199 + ch, ev = compute_changes_and_events(pl.DataFrame(rows), inds, headline_ids={"unemployment-rate"}, now=datetime(2026, 9, 1, tzinfo=UTC))
200 + kinds = set(ch["kind"].to_list())
201 + assert "structural_break" in kinds
202 + assert ev.filter(pl.col("kind") == "structural_break").height == 1
added tests/test_pipeline_reliability.py +125 −0
@@ -0,0 +1,125 @@
1 +"""Pipeline reliability: HTTP retries/backoff honouring Retry-After, country mapping, never publishing a shrunken snapshot,
2 +source-health meta and build logs."""
3 +from __future__ import annotations
4 +
5 +import json
6 +
7 +import duckdb
8 +import httpx
9 +import pytest
10 +import respx
11 +
12 +from countryatlas.config import settings
13 +from countryatlas.connectors import base as base_mod
14 +from countryatlas.connectors.base import Connector, retry_after_seconds
15 +from countryatlas.pipeline import build as build_mod
16 +from countryatlas.registry import lookup
17 +from tests.test_build import _tiny_staging
18 +
19 +
20 +class _Dummy(Connector):
21 + id = "worldbank"
22 + name = "dummy"
23 + rate_per_minute = 100_000
24 +
25 + def fetch(self, spec): # pragma: no cover - not used
26 + raise NotImplementedError
27 +
28 + def normalize(self, raw, spec): # pragma: no cover - not used
29 + return []
30 +
31 +
32 +def test_retry_after_parsing() -> None:
33 + assert retry_after_seconds(None) is None and retry_after_seconds("abc") is None
34 + assert retry_after_seconds("7") == 7.0
35 + assert retry_after_seconds("999") == base_mod.RETRY_AFTER_CAP
36 + assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT") == 0.0 # in the past → no wait
37 +
38 +
39 +@respx.mock
40 +def test_get_retries_on_transient_statuses_then_succeeds(monkeypatch: pytest.MonkeyPatch) -> None:
41 + sleeps: list[float] = []
42 + monkeypatch.setattr(base_mod.time, "sleep", lambda s: sleeps.append(s)) # Retry-After sleep
43 + # tenacity's wait: make it instant but record the computed deterministic delays
44 + waits: list[float] = []
45 + orig_wait = _Dummy.get.retry.wait
46 +
47 + def fake_wait(retry_state):
48 + w = orig_wait(retry_state)
49 + waits.append(w)
50 + return 0
51 +
52 + monkeypatch.setattr(_Dummy.get.retry, "wait", fake_wait)
53 + route = respx.get("https://example.test/x").mock(side_effect=[
54 + httpx.Response(429, headers={"Retry-After": "3"}),
55 + httpx.Response(503),
56 + httpx.Response(200, json={"ok": True}),
57 + ])
58 + c = _Dummy()
59 + r = c.get("https://example.test/x")
60 + assert r.status_code == 200 and route.call_count == 3
61 + assert [s for s in sleeps if s] == [3.0] # Retry-After honoured once (zero-length tenacity naps filtered)
62 + assert waits == [2.0, 4.0] # deterministic exponential backoff, no jitter
63 + c.close()
64 +
65 +
66 +@respx.mock
67 +def test_get_gives_up_after_five_attempts(monkeypatch: pytest.MonkeyPatch) -> None:
68 + monkeypatch.setattr(_Dummy.get.retry, "wait", lambda rs: 0)
69 + route = respx.get("https://example.test/down").mock(return_value=httpx.Response(502))
70 + c = _Dummy()
71 + with pytest.raises(base_mod.TransientHTTPError):
72 + c.get("https://example.test/down")
73 + assert route.call_count == base_mod.RETRY_ATTEMPTS
74 + c.close()
75 + # a 404 is not retried
76 + route2 = respx.get("https://example.test/missing").mock(return_value=httpx.Response(404))
77 + c2 = _Dummy()
78 + with pytest.raises(httpx.HTTPStatusError):
79 + c2.get("https://example.test/missing")
80 + assert route2.call_count == 1
81 + c2.close()
82 +
83 +
84 +def test_country_mapping_drops_aggregates_and_unknowns() -> None:
85 + lk = lookup()
86 + assert lk.any("CAN") == "CAN" and lk.any("ca") == "CAN" and lk.any("124") == "CAN"
87 + assert lk.any("Viet Nam") == "VNM" and lk.any("Türkiye") == "TUR" and lk.any("Kosovo") == "XKX"
88 + for agg in ("WLD", "OED", "EUU", "HIC", "EU27_2020", "OAVG"):
89 + assert lk.from_iso3(agg) is None
90 + assert lk.any("Atlantis") is None and lk.any("ZZZ") is None and lk.any("999") is None
91 + assert lk.from_iso2("EL") == "GRC" and lk.from_iso2("UK") == "GBR" and lk.from_iso2("XK") == "XKX"
92 +
93 +
94 +@pytest.mark.usefixtures("data_dir")
95 +def test_shrunken_snapshot_is_never_published_and_meta_has_health() -> None:
96 + _tiny_staging()
97 + first = build_mod.build(run_id="20260101T000000Z", strict=False)
98 + con = duckdb.connect(str(settings.db_path), read_only=True)
99 + meta = dict(con.execute("SELECT key, value FROM meta").fetchall())
100 + con.close()
101 + assert meta["previous_observation_count"] == "" # no previous snapshot
102 + health = json.loads(meta["source_health"])
103 + assert health["worldbank"]["ok"] >= 5 and set(health["worldbank"]) == {"ok", "partial", "failed", "quarantined"}
104 + log_path = settings.logs_dir / f"build-{first.run_id}.json"
105 + assert log_path.exists()
106 + doc = json.loads(log_path.read_text())
107 + assert doc["published"] is True and doc["counts"]["observations"] > 0 and any(f.startswith("worldbank/") for f in doc["staging_files"])
108 + before = settings.db_path.stat()
109 + # second build sees the previous count
110 + second = build_mod.build(run_id="20260102T000000Z", strict=False)
111 + con = duckdb.connect(str(settings.db_path), read_only=True)
112 + meta2 = dict(con.execute("SELECT key, value FROM meta").fetchall())
113 + con.close()
114 + assert int(meta2["previous_observation_count"]) == first.counts["observations"] and second.counts["observations"] == first.counts["observations"]
115 + after = settings.db_path.stat()
116 + # drop most staging files → < 90 % retention → strict integrity failure, live DB untouched, build log says not published
117 + for f in sorted(settings.staging_dir.glob("*/*.parquet"))[:-1]:
118 + f.unlink()
119 + with pytest.raises(build_mod.IntegrityError, match="refusing to publish"):
120 + build_mod.build(run_id="20260103T000000Z", strict=True)
121 + assert settings.db_path.stat().st_ino == after.st_ino and settings.db_path.stat().st_mtime_ns == after.st_mtime_ns
122 + assert not list(settings.build_dir.glob("atlas-20260103*"))
123 + failed_log = json.loads((settings.logs_dir / "build-20260103T000000Z.json").read_text())
124 + assert failed_log["published"] is False and "refusing to publish" in (failed_log["error"] or "")
125 + assert before.st_ino != after.st_ino or before.st_mtime_ns != after.st_mtime_ns # the second build did publish
modified tests/test_validate.py +58 −0
@@ -53,3 +53,61 @@ def test_stale_flag_on_latest_row_only() -> None:
53 53 gv = validate_frame(df, ind, now=datetime(2026, 9, 1, tzinfo=UTC))
54 54 statuses = gv.frame.sort("period")["status"].to_list()
55 55 assert statuses[-1] == "stale" and statuses[:-1] == ["imported"] * 4
56 +
57 +
58 +def test_impossible_years_are_quarantined_not_deleted() -> None:
59 + ind = indicators_by_id()["gdp-per-capita"]
60 + df = _rows("gdp-per-capita", ind.unit, {"CAN": [100.0, 101.0, 102.0, 103.0, 104.0, 105.0]}, 2020) # 2020–2025
61 + bad = pl.DataFrame({"country_id": ["CAN", "CAN"], "indicator_id": ["gdp-per-capita"] * 2, "period": [date(1700, 1, 1), date(2099, 1, 1)],
62 + "year": [1700, 2099], "frequency": ["A", "A"], "value": [50.0, 500.0], "unit": [ind.unit] * 2,
63 + "source_id": ["worldbank"] * 2, "source_dataset": ["WDI"] * 2, "source_series_code": ["X"] * 2,
64 + "is_estimate": [False] * 2, "is_forecast": [False] * 2, "retrieved_at": [datetime(2026, 1, 1, tzinfo=UTC).replace(tzinfo=None)] * 2,
65 + "source_updated_at": [None, None], "status": ["imported"] * 2, "metadata": [None, None]}).cast(df.schema)
66 + gv = validate_frame(pl.concat([df, bad]), ind, now=datetime(2026, 9, 1, tzinfo=UTC))
67 + assert gv.frame.height == 8 # nothing deleted
68 + by_year = dict(zip(gv.frame["year"].to_list(), gv.frame["status"].to_list(), strict=True))
69 + assert by_year[1700] == "quarantined" and by_year[2099] == "quarantined" and by_year[2024] == "imported"
70 + assert gv.n_quarantined == 2 and not gv.quarantine_dataset
71 + assert {i.code for i in gv.issues} >= {"impossible_year"}
72 + # a projection 6 years out is fine
73 + assert dict(zip(gv.frame["year"].to_list(), gv.frame["status"].to_list(), strict=True))[2025] == "imported"
74 +
75 +
76 +def test_duplicate_keys_quarantine_dataset() -> None:
77 + ind = indicators_by_id()["gdp-per-capita"]
78 + df = _rows("gdp-per-capita", ind.unit, {"CAN": [1.0, 2.0, 3.0]})
79 + gv = validate_frame(pl.concat([df, df.tail(1)]), ind)
80 + assert gv.quarantine_dataset and any(i.code == "duplicate" for i in gv.issues)
81 + assert gv.frame.height == 4 # rows kept for audit
82 +
83 +
84 +def test_schema_change_quarantines_dataset() -> None:
85 + ind = indicators_by_id()["gdp-per-capita"]
86 + df = _rows("gdp-per-capita", ind.unit, {"CAN": [1.0, 2.0, 3.0]})
87 + gv = validate_frame(df.drop("period"), ind)
88 + assert gv.quarantine_dataset and gv.issues[0].code == "schema_change" and "period" in gv.issues[0].message
89 + gv2 = validate_frame(df.with_columns(pl.col("value").cast(pl.Utf8)), ind)
90 + assert gv2.quarantine_dataset and "dtype" in gv2.issues[0].message
91 +
92 +
93 +def test_null_spike_and_vintage_shift_are_warnings_only() -> None:
94 + from countryatlas.pipeline.validate import PreviousStats
95 +
96 + ind = indicators_by_id()["gdp-per-capita"]
97 + series = {c: [100.0 + k for k in range(25)] for c in ("CAN", "FRA", "DEU")}
98 + prev = _rows("gdp-per-capita", ind.unit, series)
99 + # null spike: 20 % of values null now vs 0 % before
100 + cur = prev.with_columns(pl.when(pl.col("year") % 5 == 0).then(None).otherwise(pl.col("value")).alias("value"))
101 + gv = validate_frame(cur, ind, previous=PreviousStats.from_frame(prev))
102 + assert not gv.quarantine_dataset and any(i.code == "null_spike" for i in gv.issues)
103 + # vintage shift: every overlapping value 40 % higher → warning, dataset kept
104 + shifted = prev.with_columns((pl.col("value") * 1.4).alias("value"))
105 + gv2 = validate_frame(shifted, ind, previous=PreviousStats.from_frame(prev))
106 + codes = {i.code for i in gv2.issues}
107 + assert "vintage_shift" in codes and not gv2.quarantine_dataset and gv2.frame.height == shifted.height
108 + # a 3 % revision is normal
109 + gv3 = validate_frame(prev.with_columns((pl.col("value") * 1.03).alias("value")), ind, previous=PreviousStats.from_frame(prev))
110 + assert not any(i.code in ("vintage_shift", "null_spike") for i in gv3.issues)
111 + # partial download via PreviousStats too
112 + gv4 = validate_frame(prev.head(5), ind, previous=PreviousStats.from_frame(prev))
113 + assert gv4.quarantine_dataset and any(i.code == "partial_download" for i in gv4.issues)
56 114