# Raw Data Sources for the AI Risk Index ETL Pipeline

**Research date:** 2026-08-05 · **Author:** Claude (web research pass)
**Scope:** every raw source `apps/etl` needs, with current versions, exact URLs, licensing, schema notes, and how each feeds `data/raw/` → `data/derived/`.

> ⚠️ **Headline finding:** the project docs assume **O*NET 29.x**; the current production release is **O*NET 30.3 (May 2026)**, and **O*NET 31.0 lands in late August 2026** with a modernized Content Model. The 30.x series **renamed key files** (Technology Skills → *Software Skills*; Skills split into *Essential Skills* / *Transferable Skills*), which is a breaking change for any ETL written against 29.x file names. Pin one release per `INDEX_VERSION` and record it in the manifest.

---

## Master table

| # | Source | Current version / vintage | Primary URL | Format | License | Update cadence | Pipeline role |
|---|--------|---------------------------|-------------|--------|---------|----------------|---------------|
| 1 | O*NET Database | **30.3** (May 2026); 31.0 due late Aug 2026 | https://www.onetcenter.org/database.html | Excel/CSV/JSON, SQL (MySQL/SQL Server/Oracle), text | CC BY 4.0 (attribution: USDOL/ETA) | Quarterly (Feb/May/Aug/Nov-Dec) | Core: occupations, tasks, ratings → `data/raw/onet/` |
| 2 | O*NET Web Services | API v2.0 (serves 30.3) | https://services.onetcenter.org/ | REST/JSON | CC BY 4.0 + ToS | Always-current | Optional live lookups; not for bulk ETL |
| 3 | SOC 2018 taxonomy | 2018 (next revision ~2028) | https://www.bls.gov/soc/2018/ | XLSX/PDF | Public domain | ~Decennial | Occupation code spine (via O*NET-SOC 2019) |
| 4 | ESCO classification | **v1.2.1** (2025-12-10) | https://esco.ec.europa.eu/en/use-esco/download | CSV, SKOS/RDF, API | Free reuse (Decision 2011/833/EU) w/ attribution | ~Annual point releases | EU/FR occupation layer → `data/raw/esco/` |
| 5 | ESCO↔O*NET crosswalk (official) | 2022 report; CSV updated 2023-08 (ESCO v1.1 base) | https://esco.ec.europa.eu/en/use-esco/other-crosswalks | CSV | Same as ESCO | Irregular | Map O*NET-SOC → ESCO → `data/raw/esco/` |
| 6 | ROME 4.0 (France Travail) | Update of 2026-06-18; next Oct 2026 | https://www.data.gouv.fr/datasets/repertoire-operationnel-des-metiers-et-des-emplois-rome | CSV/XLSX (+ API) | Licence Ouverte 2.0 | ≥2×/year | FR occupation layer → `data/raw/rome/` |
| 7 | BLS OEWS wages+employment | **May 2025** (released 2026-05-15) | https://www.bls.gov/oes/tables.htm | XLSX in ZIP | Public domain | Annual (~May) | `cost_ratio` wages, employment weights → `data/raw/bls/oews/` |
| 8 | BLS Employment Projections | **2024–34** (released 2025-08-28); 2025–35 due ~Sep 2026 | https://www.bls.gov/emp/data/occupational-data.htm | XLSX | Public domain | Annual | Adoption-velocity / outlook context → `data/raw/bls/ep/` |
| 9 | Eurostat SES earnings | Ref. year **2022** (4-yearly; next 2026 ref, pub ~2028) | https://ec.europa.eu/eurostat/databrowser/view/earn_ses_hourly | TSV/SDMX API | CC BY 4.0 | Every 4 years | EU wages by ISCO-08 → `data/raw/eurostat/` |
| 10 | Eurostat LFS employment | Annual, latest 2025 | https://ec.europa.eu/eurostat/databrowser/view/lfsa_egai2d | TSV/SDMX API | CC BY 4.0 | Annual | EU employment by ISCO 2-digit → `data/raw/eurostat/` |
| 11 | INSEE salaires (France) | 2023 consolidated (Base Tous salariés, DSN) | https://www.insee.fr/fr/statistiques (Base Tous salariés) | XLSX/CSV | Open (Insee reuse) | Annual (~2-yr lag) | FR wages by PCS → `data/raw/insee/` |
| 12 | Census BTOS AI supplement | Collected 2025-11-17→2026-02-08; released **2026-04-23** | https://www.census.gov/hfp/btos/data_downloads | XLSX/CSV | Public domain (experimental) | Biweekly core; supplements episodic | `adoption_velocity` (US, by sector/state/size) → `data/raw/btos/` |
| 13 | Eurostat ICT-in-enterprises AI | Survey year **2025** (19.95% EU firms use AI) | Datasets `isoc_eb_ai`, `isoc_eb_ain2` | TSV/SDMX API | CC BY 4.0 | Annual (Dec/Jan) | `adoption_velocity` (EU, by NACE) → `data/raw/eurostat/` |
| 14 | Ramp AI Index | June 2026 (55.0% adoption) | https://ramp.com/data | CSV download + charts | Free; check Ramp terms | Monthly | High-frequency US adoption signal → `data/raw/ramp/` |
| 15 | Anthropic Economic Index | 6th release (**2026-06-26**, "Cadences") | https://huggingface.co/datasets/Anthropic/EconomicIndex | CSV (HF dataset) | Data CC-BY; code MIT | ~Quarterly | Observed task-level AI usage; rater calibration → `data/raw/aei/` |
| 16 | Stanford AI Index | **2026 edition** (9th, Apr 2026, 423 pp) | https://hai.stanford.edu/assets/files/ai_index_report_2026.pdf | PDF + public data appendix | Free (attribution) | Annual (~April) | Context stats for methodology doc / barriers |
| 17 | Felten AIOE | 2021 + GenAI variants | https://github.com/AIOE-Data/AIOE | XLSX | Free w/ citation | Static | Benchmark exposure scores → `data/raw/benchmarks/` |
| 18 | Eloundou "GPTs are GPTs" | Science 2024 replication | https://github.com/openai/GPTs-are-GPTs | CSV | MIT | Static | Benchmark exposure scores → `data/raw/benchmarks/` |
| 19 | Webb (2020) AI exposure | 2020 | https://www.michaelwebb.co (on request) | — | On request | Static | Optional benchmark (no public bulk file) |
| 20 | ILOSTAT | Continuous; incl. "employment by GenAI exposure" tables | https://ilostat.ilo.org | CSV/API | CC BY 4.0 | Continuous | Intl. harmonized employment; ILO GenAI exposure scores |

---

## 1. O*NET Database (core input)

- **Current release:** **O*NET 30.3**, May 2026 (source: https://www.onetcenter.org/db_releases.html). Release train since the project was scoped: 29.2 (Feb 2025) → 29.3 (May 2025) → 30.0 (Aug 2025) → 30.1 (Dec 2025) → 30.2 (Feb 2026, new four-level Job Zones) → 30.3 (May 2026, **modernized Content Model** + Specific Interests) → **31.0 expected late August 2026**.
- **Downloads:** https://www.onetcenter.org/database.html — formats: tabular (Excel/CSV/JSON), SQL loads for MySQL/PostgreSQL/MariaDB, SQL Server, Oracle, plus RDF. Full Excel archive: `https://www.onetcenter.org/dl_files/database/db_30_3_excel.zip` (individual files linked from the same page; per-format data dictionary at https://www.onetcenter.org/dictionary/30.3/excel/).
- **Files the ETL needs (30.3 row counts):**
  - `Occupation Data` — 1,016 rows (O*NET-SOC code, title, description)
  - `Task Statements` — 18,796 rows (task_id, task, task type, incumbents responding)
  - `Task Ratings` — 161,559 rows (importance IM, relevance RL, frequency FT scales, with N, SE, CI bounds — feeds our task weights **and** our own CI propagation)
  - `Tasks to DWAs`, `DWA Reference`, `IWA Reference` — task ↔ detailed/intermediate work activity links
  - `Work Activities` — 73,308 rows (GWA ratings)
  - `Abilities` — 92,976 · `Knowledge` — 59,004 · `Work Context` — 297,676
  - **Renamed in 30.x:** `Technology Skills` → **`Software Skills`** (31,821 rows); the old `Skills` file is now **`Essential Skills`** (17,880) + **`Transferable Skills`** (44,700). ETL loaders and any code referencing "Technology Skills" must be updated.
- **License:** **CC BY 4.0**. Required attribution: credit the "O*NET 30.3 Database by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA), used under the CC BY 4.0 license." Put this in the public methodology page and API `/api/v1/methodology` metadata.
- **Taxonomy:** O*NET-SOC **2019** taxonomy (built on SOC 2018): 1,016 occupation titles of which **923 are data-collection-level**; code format `XX-XXXX.XX` (SOC 6-digit + 2-digit O*NET suffix). ~891 occupations already updated in 2026 YTD.
- **O*NET Web Services:** register (free) at https://services.onetcenter.org/developer/signup; API v2.0 reference at https://services.onetcenter.org/reference. **No hard rate limit**, but ToS gives per-second/per-day guidance; on `429`, retry after ≥200 ms. Always serves the latest DB — good for spot checks, **not** for reproducible scoring (use pinned dumps).
- **SOC 2018:** 867 detailed occupations → 459 broad → 98 minor → 23 major groups; code `XX-XXXX`. Definitions/structure files at https://www.bls.gov/soc/2018/ (XLSX). Public domain.
- **Pipeline:** fetch script downloads the pinned release ZIP into `data/raw/onet/db_30_3/`, verifies SHA-256, records release + hash in the derived manifest. Never edit in place.

## 2. ESCO + crosswalks + ROME (EU/France layer)

- **ESCO v1.2.1** (last update 2025-12-10). Portal: https://esco.ec.europa.eu/en · downloads (per-language CSV + "language independent" files, SKOS/RDF, Local API): https://esco.ec.europa.eu/en/use-esco/download (free account/selection flow). ~3,000 occupations mapped to ISCO-08; **13,939 skills** in v1.2.1. Web API: https://esco.ec.europa.eu/en/use-esco (base `https://ec.europa.eu/esco/api`).
- **License:** free reuse under Commission Decision 2011/833/EU; must publish the ESCO acknowledgement statement (see FAQ: https://esco.ec.europa.eu/en/about-esco/faq).
- **ESCO↔O*NET official crosswalk** (co-created EC + USDOL, AI-assisted with human validation):
  - Page: https://esco.ec.europa.eu/en/use-esco/other-crosswalks (also listed at https://www.onetcenter.org/crosswalks.html)
  - CSV: `https://esco.ec.europa.eu/system/files/2023-08/ONET_%28Occupations%29_0_updated.csv`
  - Two published variants: (a) exact/narrow/broad/close matches (QA'd, USDOL-validated); (b) same + "related" matches (**lower quality — not validated**; exclude from scoring joins by default).
  - Technical report: `https://esco.ec.europa.eu/system/files/2022-12/ONET%20ESCO%20Technical%20Report.pdf`. Built against **ESCO v1.1 / O*NET-SOC 2019** (the O*NET Web Services crosswalk endpoint still states ESCO v1.1.0) — re-verify concept URIs against v1.2.1 during ETL; unmatched URIs go to a QA report.
  - Bonus: **ESCO↔NACE crosswalk** now available: `https://esco.ec.europa.eu/system/files/2026-02/ESCO-NACE%20rev.%202.1%20crosswalk.xlsx` (useful for joining Eurostat sector-level AI adoption to occupations).
- **ESCO↔ROME:** no single public EC file. France Travail maintains ROME↔ESCO correspondence tables under the EURES obligation (each member state maps its national classification to ESCO); the ROME open-data bundle includes correspondence referentials, and EURES member-state mapping tables are listed on the ESCO portal ("EURES Countries Mapping Tables"). Validate coverage during ETL and fall back to ROME→ISCO-08→ESCO if a direct table is missing for some fiches.
- **ROME 4.0 (France Travail):**
  - Open data: https://www.data.gouv.fr/datasets/repertoire-operationnel-des-metiers-et-des-emplois-rome — multiple referential files (arborescence, compétences, contextes, mobilité). **Licence Ouverte / Open Licence 2.0.** Last update 2026-06-18; **next update announced for Oct 2026**; ≥2 updates/year.
  - Also via API on https://francetravail.io (ROME 4.0 APIs, OAuth key) and mirrored on https://www.francetravail.org/opendata/.
  - Code format: 1 letter + 4 digits (e.g., `M1607`); ~600 fiches métiers organized by 14 domaines.
  - Pipeline: `data/raw/rome/` with the data.gouv.fr resource URLs + version date in the manifest.

## 3. Wage data

- **BLS OEWS — May 2025** (released **2026-05-15**; next: May 2026 data in spring 2027). ~830 SOC occupations; employment, mean/median hourly & annual wages, wage percentiles (10/25/50/75/90), by nation/state/MSA/industry.
  - Tables hub: https://www.bls.gov/oes/tables.htm → ZIPs `oesm25nat.zip` (national), `oesm25st.zip` (states), `oesm25ma.zip` (metro), national-by-industry files (served from `https://www.bls.gov/oes/special-requests/…`; BLS blocks non-browser user agents — use a browser UA in the fetch script). Field layout documented in each ZIP's `field_descriptions` sheet.
  - License: US government work, public domain. Cadence: annual.
  - Pipeline: national file feeds `cost_ratio` (wage denominator, stored as **integer cents + USD** per repo convention) and employment weights; join key = SOC 2018 6-digit → O*NET-SOC 2019 (strip `.XX` suffix / use O*NET-SOC↔SOC crosswalk).
- **Eurostat SES (Structure of Earnings Survey):** 4-yearly, latest reference year **2022** (published 2024–25; next ref-year 2026 published ~2028). Datasets: `earn_ses_hourly` (and monthly/annual variants) — mean/median hourly earnings by **ISCO-08 2-digit** × NACE × country. Databrowser: https://ec.europa.eu/eurostat/databrowser/view/earn_ses_hourly · bulk via SDMX API `https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/earn_ses_hourly?format=TSV`. License CC BY 4.0. Occupation resolution is only 2-digit ISCO — EU `cost_ratio` will be coarser than US; flag in methodology.
- **France:** INSEE *Base Tous salariés* (from DSN, formerly DADS): salaire net **EQTP** by PCS (up to 4-digit), sector, sex; latest consolidated 2023 (see Insee Première n°1938 for 2021: https://www.insee.fr/fr/statistiques/6799523; séries longues: https://www.insee.fr/fr/statistiques/8660332). DARES publishes wage/employment "portraits statistiques des métiers" by FAP. Requires PCS↔ROME/ISCO crosswalk (INSEE publishes PCS↔ISCO tables) — France-specific wage joins are a v2 concern.

## 4. Employment counts & projections

- **OEWS employment** (same May 2025 files as §3) — primary US employment weights.
- **BLS Employment Projections 2024–34** (released 2025-08-28; **2025–35 edition expected ~Sept 2026** — recheck before ingesting): https://www.bls.gov/emp/data/occupational-data.htm — "All occupational tables in a single file (XLSX)", National Employment Matrix 2024/2034, occupational separations, plus the new **skills data tables** (importance of skills by occupation). Public domain. Feeds adoption-velocity priors and UI "outlook" context. State-level: https://projectionscentral.org/longterm (REST + download).
- **Eurostat LFS:** `lfsa_egai2d` — employed persons by detailed occupation (ISCO-08 2-digit), annual: https://ec.europa.eu/eurostat/databrowser/view/lfsa_egai2d (SDMX API as above). CC BY 4.0.
- **ILOSTAT:** https://ilostat.ilo.org — harmonized employment by ISCO level 2 (annual/quarterly) across countries, CSV bulk + API, CC BY 4.0. Notably now publishes **"Employment by sex and generative AI exposure"** tables (based on the ILO/Gmyrek GenAI occupational exposure scores) — both a benchmark and a ready-made employment-by-exposure aggregate.

## 5. AI adoption data (`adoption_velocity` dimension)

- **US Census BTOS AI supplement:** third AI supplement collected **2025-11-17 → 2026-02-08**, released **2026-04-23** (press: https://www.census.gov/newsroom/press-releases/2026/btos-apr-23.html). Measures firm AI use overall and — new this cycle — **by worker tasks and business functions**, split by NAICS sector, state, and firm size. Data hub: https://www.census.gov/hfp/btos/data (Downloads tab: https://www.census.gov/hfp/btos/data_downloads, incl. historical; API tab available). Core biweekly BTOS also carries a recurring "AI use in last two weeks" item (wording revised Nov 2025 — treat as a series break). ~1.2M-business sample; experimental data product; public domain.
- **Eurostat — AI in enterprises:** datasets **`isoc_eb_ai`** (enterprises using AI technologies) and **`isoc_eb_ain2`** (by purpose/technology × NACE), survey year **2025**: 19.95% of EU enterprises use AI (55.03% of large firms); Statistics Explained article (updated Dec 2025, next Dec 2026): https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Use_of_artificial_intelligence_in_enterprises. Join to occupations via sector (NACE) using the ESCO-NACE crosswalk (§2). CC BY 4.0.
- **Ramp AI Index (Ramp Economics Lab):** https://ramp.com/data — monthly AI adoption among US firms from card/bill-pay spend of 70k+ businesses (June 2026: **55.0%**, +0.8 pp MoM), broken out by size and sector, downloadable; methodology: https://econlab.substack.com/p/how-ramp-data-works. Now shifting to *intensity* tracking. Free/open resource; confirm redistribution terms before committing derived aggregates.
- **Anthropic Economic Index:** HF dataset **`Anthropic/EconomicIndex`** (https://huggingface.co/datasets/Anthropic/EconomicIndex). Release folders: `release_2025_02_10` (initial O*NET task mappings, automation vs augmentation), `release_2025_03_27` (cluster-level, thinking-mode fractions per O*NET task), `release_2025_09_15` (geography + 1P API), `release_2026_01_15` ("economic primitives"), `release_2026_03_24` ("learning curves"), `release_2026_06_26` ("Cadences", monthly aggregates), plus `labor_market_impacts/`. **Data CC-BY (repo metadata lists MIT for code).** This is the single most direct empirical input for calibrating our LLM-rater `automatability`/`feasibility` scores against observed usage — it is keyed to **O*NET task statements**, same spine as ours.
- **Stanford AI Index 2026** (9th edition, April 2026, 423 pp): report PDF https://hai.stanford.edu/assets/files/ai_index_report_2026.pdf; public data appendix downloadable from the HAI AI Index page. Key 2026 stats: 53% population-level GenAI adoption; $172B est. US consumer surplus. Use for methodology narrative and `barriers`/`adoption_velocity` context, not row-level joins. Annual.

## 6. Published occupation-level exposure benchmarks

These are **validation benchmarks** (`data/raw/benchmarks/`), not scoring inputs — our methodology must remain independently reproducible.

| Dataset | What it is | URL / file | License |
|---|---|---|---|
| **Felten–Raj–Seamans AIOE** | AI Occupational Exposure by 6-digit SOC (10 AI applications × 52 O*NET abilities); + Language-Modeling and Image-Generation GenAI variants, industry (AIIE) & geography | https://github.com/AIOE-Data/AIOE → `AIOE_DataAppendix.xlsx`, `Language Modeling AIOE and AIIE.xlsx`, `Image Generation AIOE and AIIE.xlsx` | Free; citation required (SMJ 42(12):2195–2217, 2021) |
| **Eloundou et al. "GPTs are GPTs"** (Science 2024) | Task- and occupation-level LLM exposure (α=E1, β=E1+0.5·E2, γ=E1+E2; human + GPT-4 ratings) on O*NET tasks | https://github.com/openai/GPTs-are-GPTs → `occ_level.csv` (occupation), task-level in `data/` | MIT |
| **Webb (2020)** | Patent-text-based AI exposure by SOC | https://www.michaelwebb.co/webb_ai.pdf — **data on request only** (no public bulk file) | On request |
| **Anthropic AEI task exposure** | Observed Claude usage mapped to O*NET tasks (see §5) | HF `Anthropic/EconomicIndex` | CC-BY |
| **ILO GenAI exposure (Gmyrek et al.)** | ISCO-08 occupation GenAI exposure gradients, used in ILOSTAT employment tables | via https://ilostat.ilo.org | CC BY 4.0 |
| Context | Yale Budget Lab compared 7 exposure metrics (large disagreement on *most*-exposed, agreement on *least*-exposed); Brookings (Mar 2026) methodology uses AIOE + OEWS | brookings.edu methods PDF | — |

Use for the sensitivity analyses in `docs/methodology/sensitivity/` (rank correlations of our composite vs AIOE / GPTs-are-GPTs / AEI).

---

## Ingestion order & manifest plan

Order respects join dependencies (occupation spine first, then attributes, then joins):

1. **SOC 2018 structure** (`data/raw/soc/`) — code spine, 867 detailed occupations.
2. **O*NET 30.3 full dump** (`data/raw/onet/db_30_3/`) — pin release; load order: Occupation Data → Task Statements → Task Ratings → Tasks-to-DWAs/IWAs → Work Activities → Abilities/Knowledge → Software/Essential/Transferable Skills → Work Context. *Decision needed before build:* stay on 30.3 or wait for 31.0 (late Aug 2026) — either way, `INDEX_VERSION` notes the O*NET release.
3. **OEWS May 2025** (`data/raw/bls/oews/`) — wages (→ integer cents) + employment; join on SOC 6-digit.
4. **BLS EP 2024–34** (`data/raw/bls/ep/`) — projections + skills tables.
5. **ESCO v1.2.1** (`data/raw/esco/classification/`) + **ESCO↔O*NET crosswalk** (`data/raw/esco/crosswalk/`) — validate URI coverage vs v1.2.1; exclude "related" matches from scoring joins.
6. **ROME 4.0** (`data/raw/rome/`) + ROME↔ESCO correspondence; fallback path ROME→ISCO-08→ESCO.
7. **Eurostat**: `earn_ses_hourly` (SES 2022), `lfsa_egai2d` (LFS), `isoc_eb_ai(n2)` (AI adoption) via SDMX API pulls (`data/raw/eurostat/`).
8. **Adoption signals**: BTOS AI supplement, Ramp AI Index, AEI releases (`data/raw/btos|ramp|aei/`).
9. **Benchmarks**: AIOE, GPTs-are-GPTs, ILO GenAI exposure (`data/raw/benchmarks/`) — validation only.

**Manifest rules** (per repo convention — raw payloads are gitignored, manifests committed): for every fetched artifact record `{source, version_label, source_url, fetch_date, sha256, row_count, license, attribution_string}` in `data/derived/manifests/<source>.json`. The fetch script must fail loudly if a pinned URL 404s or the hash changes (BLS and O*NET replace files in place across releases). Re-verify before each `INDEX_VERSION` bump: O*NET quarterly page, OEWS annual page (mid-May), EP annual page (late Aug/Sep), ESCO portal (point releases), ROME (Oct 2026), BTOS supplement announcements, AEI HF repo.

**Attribution block for the public methodology page:** O*NET (USDOL/ETA, CC BY 4.0) · ESCO (© European Union, ESCO acknowledgement statement) · ROME (France Travail, Licence Ouverte 2.0) · BLS/Census (public domain, cite program + vintage) · Eurostat (CC BY 4.0) · Anthropic Economic Index (CC-BY) · benchmark papers cited per their requirements.