Claim-first data architecture
Since 2026-09-12 every numerical figure DataCenterIndex extracts is stored as a claim before it can become a displayed value. A claim is one assertion by one document about one subject, with its scope, its semantics, the sentence that supports it, the parser that produced it and the field-level authority of its source. Reconciliation then decides which claim (if any) populates a column. Nothing is deleted when sources disagree.
Claim
| column | meaning |
|---|---|
subject_type, subject_id |
facility / campus / project / operator / market / country |
predicate |
what is asserted — it_capacity_mw, critical_power_mw, utility_capacity_mw, grid_connection_mw, current_power_mw, planned_power_mw, ultimate_campus_mw, phase_mw, project_investment_usd, campus_investment_usd, company_investment_usd, country_program_usd, multi_year_capex_usd, deal_value_usd, status, … |
value, value_text, unit |
the figure (MW, USD) or text |
scope, scope_reason |
building · facility · campus · metro · country · portfolio · company · unknown, and the rule that decided |
evidence_text, evidence_start, evidence_end |
the supporting sentence and its offsets in the parsed text — no sentence, no numerical claim (structured spec tables record the extraction method instead) |
source_id, url, document_id, published_at, retrieved_at |
where and when |
authority_tier |
A–E for THIS field (see below) |
parser_name, parser_version, run_id |
who extracted it (reprocessing + rollback) |
status |
current · superseded (same source, same page, new value) · rejected (review / rollback) · review (critical sanity flag) · unscoped (figure describes a portfolio, a company, a country, a metro, or its scope is unknown) |
Scope rules
- Only site scopes (
building,facility,campus) may populate a facility's or project's capacity / investment columns. Portfolio, company, country, metro and unknown scopes are stored and shown in the evidence drawer, never summed. - Scope is classified deterministically from the sentence (
packages/core/src/claims.tsclassifyScope): company → portfolio → country → metro → campus → building → facility → the record's own scope when the sentence has no signal (structured parsers) → unknown. - A campus figure is never written to a building record; a building's figure never stands for its campus.
Capacity semantics
IT load, critical power, utility capacity, grid connection, current power, planned capacity, ultimate build-out and
phase capacity are different predicates and different columns (it_capacity_mw, total_power_mw, planned_power_mw,
utility_capacity_mw, grid_connection_mw, ultimate_campus_mw). The displayed figure carries capacity_scope and
capacity_semantics so the UI can say what "300 MW" means. When a sentence gives no cue the predicate defaults from the
field/record status and the claim is flagged mw_semantics_default (info).
Field-level authority
One universal source rank is wrong. FIELD_AUTHORITY in packages/core/src/claims.ts gives a tier per field:
| field | A | B | C | D | E |
|---|---|---|---|---|---|
| capacity | utility, government, filing | operator, cloud provider | registry, secondary | news | dataset, community |
| geometry | government, community (OSM) | registry, operator, dataset | cloud provider, utility, filing | secondary | news |
| interconnection | registry (PeeringDB) | operator | community, dataset | secondary, news, government, filing | utility, cloud |
| status | operator, government, filing, cloud provider | utility, registry | secondary, news | dataset, community | — |
| investment | filing, government | operator, cloud provider | utility, secondary | news | registry, dataset, community |
Estimates and LLM-only extractions drop one tier; a human review raises one.
Sanity engines
capacitySanity / investmentSanity run on every claim. Blocking flags (blocks: true) keep the figure out of the
columns: non-site scope, campus figure on a building, ≥ 20 GW (market statistic), invalid value, deal value / capex /
country programme as a project investment, > $500 B. Non-blocking critical flags (single site > 1 000 MW without a campus
designation, > $50 B on one site, money and MW sharing a number in one sentence, 5× change) assign the figure but open a
quality_flags row prioritised by impact (reviewPriority).
Announcement classification (projects)
Before a news article can create a project, classifyProjectEvent labels it: NEW_BUILD, EXPANSION,
CONSTRUCTION_START, PERMIT, LAND_ACQUISITION, GRID_CONNECTION (physical — may create / modify a project);
POWER_AGREEMENT, FINANCING, ACQUISITION, PARTNERSHIP, CUSTOMER_AGREEMENT (associated — attach an event to an
identifiable project, never create one); EXECUTIVE_APPOINTMENT, SUSTAINABILITY, PRODUCT_NEWS,
GENERAL_COMPANY_NEWS, UNKNOWN (never a project). Creation additionally needs the evidence threshold: (explicit
name or operator) + location + a development verb. Company domiciles ("Denver-based", "headquartered in") are stripped
before locating the project.
Lifecycle state machine
projectTransition(from, to): forward moves along rumored → proposed → announced → permitting → approved →
under_construction → partially_operational → operational are accepted; delayed / cancelled are side branches;
leaving delayed resumes; a backward move needs a source that outranks the stored one, otherwise it is flagged
(status_backward). Stage dates (permit_filed_on, approved_on, construction_started_on, opened_on) are set from
the announcement that moved the stage.
Winners, history, rollback
provenance.is_winner marks the observation backing each displayed value; provenance.run_id, claims.run_id,
events.run_id and document_versions.run_id tie every change to a connector run (POST /api/admin/runs/:id/rollback).
Capacity history = winner changes + claims + capacity_changed events. Daily entity_snapshots keep global totals,
per-operator and per-country totals, rankings and stage counts for "as of" views and the automated regression checks
(facilities −5 %, known MW +20 %, one operator +10 GW/day, one connector > 500 records/day, > 200 location changes/day).
Containment (campus vs building)
facilities.parent_facility_id + record_scope (building / facility / campus). The matcher's rule:campus-vs-building
now links instead of flagging a duplicate. Every aggregate uses FACILITY_VIEW (apps/worker/src/rankings.ts): a campus
whose buildings publish figures contributes no MW itself; a building without a figure under a campus with one is
"covered"; a campus with buildings is not counted as an extra facility.
Debugging
pnpm dci trace <doc-id|url> (or GET /api/admin/documents/:id/trace) shows every stage for one document: raw fetch →
parsed text → structured records → normalized entities → claims with scope/semantics/evidence → match candidates →
dry-run reconciliation → resulting changes. pnpm dci quality runs the sweep, pnpm dci snapshot the snapshots +
regression checks, pnpm dci gaps the data-gap counts, pnpm dci quarantine <id> on puts a connector in preview mode.