spb/airiskindex Public

1# AI Risk Index — Methodology v1.0.0 (DRAFT)23> Status: **draft, unreleased**. This document is the source of truth for the scoring4> methodology (see CLAUDE.md §9). Code in `packages/scoring` links to the section5> anchors below; if code and this document disagree, stop and flag it.6>7> Related work and evidence base: `docs/research/01-existing-indices.md` (indices),8> `02-data-sources.md` (data), `04-landscape-and-evidence.md` (adoption & barriers evidence).910---1112## 1. Principles <a name="principles"></a>13141. **Task-based, not occupation-based.** Occupations are bundles of tasks with very15   different AI exposure; whole-occupation scoring (Frey & Osborne 2013) has a poor16   empirical record. Occupation scores are always *derived* from task scores17   (Arntz, Gregory & Zierahn 2016; Eloundou et al. 2024).182. **Three concepts, never collapsed** (see §5):19   - **Exposure** — AI is technically capable of performing the task.20   - **Substitution** — AI actually replaces the human performing it. This is the21     headline composite.22   - **Augmentation** — AI assists the human, increasing productivity.23   Realized labor-market effects concentrate in *automation-classified* usage, not24   augmentation (Brynjolfsson, Chandar & Chen 2025), so conflating the three is not25   just imprecise — it is empirically wrong.263. **Uncertainty is part of the score.** Every published score carries27   `score_low / score / score_high`. Interval width is driven primarily by28   disagreement between independent LLM raters (single-model ratings show up to a29   19× spread in headline statistics across frontier models — Yin, Vu & Persico 2026)30   plus human-calibration error on the expert anchor set.314. **Fully reproducible.** Weights, formulas, prompt versions, rater model IDs and32   per-model ratings are all published. Every score traces to a stored33   `score_runs` row; historical runs are immutable.345. **Versioned.** Any change to weights, formulas, or rater prompts bumps35   `INDEX_VERSION` (semver) with a changelog entry and regenerated sensitivity outputs.3637## 2. Data <a name="data"></a>3839| Input | Source | Role |40|---|---|---|41| Occupations & task statements | O*NET 30.x (O*NET-SOC 2019 taxonomy), CC BY 4.0 | Unit of analysis (~18k tasks, ~900 data-level occupations) |42| Task importance weights | O*NET Task Ratings (IM scale 1–5) | Aggregation weights (§6) |43| EU/France occupations | ESCO v1.2.x + official ESCO↔O*NET crosswalk; ROME 4.0 | Crosswalked scores (crosswalk loss documented per occupation) |44| Wages | BLS OEWS (latest May release); Eurostat SES; INSEE | `cost_ratio` denominator |45| Adoption data | Census BTOS AI supplement, Ramp AI Index, Anthropic Economic Index (Hugging Face), Challenger reports | `adoption_velocity` inputs (§4.5) |46| Human anchor ratings | Expert Delphi panel (`data/derived/expert_overrides/`) | LLM-rater calibration (§3) |4748Raw dumps are immutable (`data/raw/`, gitignored); all transformations go through49`apps/etl`; derived artifacts commit only manifests (hashes + row counts).5051## 3. Task rating (LLM-as-evaluator) <a name="task-rating"></a>5253Each O*NET task statement is rated on a **5-point scale** per dimension (§4) by a54**panel of ≥2 (target 3) frontier LLMs** (`RATER_MODELS`), using versioned rubric55prompts (`apps/worker/src/raters/prompts/`). Per task × dimension:5657- `rating_mid` = mean of panel ratings;58- `rating_low` / `rating_high` = min / max of panel ratings (rater-disagreement band).5960Rubric anchors follow the citable standards: automatability uses the Eloundou et al.61"≥50% time saving at equal quality" threshold, decomposed into named criteria62(SML-style multi-criterion rubric); ratings require structured justifications and are63stored with model ID, prompt version and raw response (full audit trail).6465**Calibration:** 5% of ratings are sampled for human review; panel-vs-human66disagreement > 1 point routes the task to the expert queue. Expert overrides replace67the LLM band for that task and are flagged in the API output.6869## 4. Dimensions <a name="dimensions"></a>7071Composite weights live in `packages/scoring/src/weights.ts` and are published at72`/api/v1/methodology`. Weights sum to 1.0.7374| Dimension | Key | Weight | Orientation | Rubric anchor (rating of 5 means…) |75|---|---|---|---|---|76| Task automatability | `automatability` | 0.35 | direct | Current AI (incl. tooling/agents) can do the task with ≥50% time saving at equal quality |77| Current technical feasibility | `feasibility` | 0.20 | direct | Deployed products demonstrably perform this task reliably today (not merely conceivable) |78| Cost of substitution vs. wage | `cost_ratio` | 0.15 | direct | AI cost per task-equivalent ≪ loaded human wage for the same output |79| Adoption barriers | `barriers` | 0.20 | **inverted** | Strong barriers: licensing/liability/regulation/human-contact requirements block substitution |80| Sector adoption velocity | `adoption_velocity` | 0.10 | direct | Occupation's dominant sectors adopt AI fast and deep (agents in production, measured displacement) |8182Orientation: a *direct* dimension's higher rating increases substitution pressure; an83*inverted* dimension's higher rating decreases it. `barriers` is the only inverted84dimension in v1. Orientation is encoded once, in `packages/scoring/src/weights.ts`,85next to the weights.8687Additionally, **`augmentation`** is rated per task on the same 5-point scale (does AI88assist the human on this task, raising productivity without replacing them?). It is89**not** part of the substitution composite; it feeds the augmentation sub-score (§5).9091### 4.4 `barriers` components <a name="barriers"></a>92Rated against five named criteria (see research doc 04): licensing/authorization93requirement (0.30), liability & error-cost asymmetry (0.25), regulatory-process94coverage (0.20, jurisdiction-specific), human-contact requirement (0.15),95organizational friction (0.10). Pre-2022 null results (OECD 2021; Acemoglu et al.962022) show barriers dominate short-run outcomes — hence the 0.20 weight.9798### 4.5 `adoption_velocity` components <a name="adoption-velocity"></a>99Grounded in *measured* adoption, not forecasts: sector AI-use rate employment-weighted100(0.35), sector adoption momentum (0.20), agentic deployment depth (0.20), realized101displacement intensity (0.15), occupation-level usage intensity from the Anthropic102Economic Index (0.10). Refreshed each index release; sources are public and dated.103104## 5. Scoring formulas <a name="formulas"></a>105106All formulas are implemented, pure and deterministic, in `packages/scoring`.107Ratings `r ∈ [1,5]` normalize to pressure `p ∈ [0,1]`:108109```110p = (r − 1) / 4            (direct dimensions)111p = 1 − (r − 1) / 4        (inverted dimensions)112```113114Per task:115116```117substitution_task = 100 · Σ_d  w_d · p_d                     (all five dimensions)118exposure_task     = 100 · (w_auto·p_auto + w_feas·p_feas) / (w_auto + w_feas)119augmentation_task = 100 · p_augmentation120```121122Confidence bounds: `score_low` is computed with each dimension's123pressure-minimizing rating bound (for direct dimensions the low rating; for inverted124dimensions the **high** rating), `score_high` symmetrically. Bounds are therefore125worst/best-case envelopes over rater disagreement, and `low ≤ score ≤ high` always126holds.127128## 6. Aggregation to occupations <a name="aggregation"></a>129130Occupation scores are the **importance-weighted mean** of task scores, weights from131O*NET Task Ratings importance (IM, 1–5), normalized to sum to 1 within the132occupation. Applied identically to `low`, `score` and `high`. Tasks lacking133importance ratings receive the occupation-mean importance.134135The API additionally reports the share of tasks with `substitution_task ≥ 70`136("highly exposed task share") — this, not the composite alone, is the preferred137headline in UI copy ("X% of tasks in this occupation are highly exposed").138139## 7. Versioning & runs <a name="versioning"></a>140141- `INDEX_VERSION` (semver) in `packages/scoring/src/version.ts`.142- MAJOR: formula/weight changes. MINOR: data-source version bumps (new O*NET release,143  new adoption data). PATCH: recomputation with refreshed adoption inputs, prompt144  clarifications that don't change the rubric semantics.145- Every computation writes a `score_runs` row (index version, prompt version, rater146  models). Old runs stay queryable forever; the API serves the latest by default and147  any run on request.148149## 8. Validation & sensitivity <a name="validation"></a>150151Published with every MAJOR/MINOR release under `docs/methodology/sensitivity/`:1521531. **Convergent validity:** Spearman correlation of our occupation scores against154   Felten AIOE, Eloundou β, and ILO WP140 gradients (ρ ≈ 0.84 between independent155   modern methodologies is the reference bar).1562. **Rater stability:** cross-model agreement distribution; occupations with the157   widest bands flagged in-product.1583. **Outcome tracking:** correlation against the Stanford "Canaries" dashboard159   (entry-level employment in exposed occupations) and AEI usage shares — exposure160   indices individually explain <11% of realized unemployment risk (Frank, Ahn &161   Moro 2025), so we report outcome tracking honestly rather than claiming prediction.1624. **Weight sensitivity:** composite rank stability under ±25% perturbation of each163   weight.164165## 9. Known limitations <a name="limitations"></a>166167- LLM raters co-evolve with the technology they measure (the "ruler" problem);168  multi-model panels bound but do not eliminate this.169- Sector-level adoption inputs are priors, corrected by occupation-level usage data.170- ESCO/ROME crosswalked scores inherit crosswalk loss; flagged per occupation.171- Scores describe *tasks as currently constituted*; occupations reorganize.172- This index measures exposure and substitution *pressure*, not certainty of job173  loss. Product copy follows the adaptation-not-doom tone guide accordingly.174