# Raw Data Sources for the AI Risk Index ETL Pipeline

Research date: 2026-08-05 · Author: Claude (web research pass) Scope: every raw source apps/etl needs, with current versions, exact URLs, licensing, schema notes, and how each feeds data/raw/ → data/derived/.

⚠️ Headline finding: the project docs assume O*NET 29.x; the current production release is O*NET 30.3 (May 2026), and O*NET 31.0 lands in late August 2026 with a modernized Content Model. The 30.x series renamed key files (Technology Skills → Software Skills; Skills split into Essential Skills / Transferable Skills), which is a breaking change for any ETL written against 29.x file names. Pin one release per INDEX_VERSION and record it in the manifest.

# Master table

# 1. O*NET Database (core input)

• Current release: O*NET 30.3, May 2026 (source: https://www.onetcenter.org/db_releases.html). Release train since the project was scoped: 29.2 (Feb 2025) → 29.3 (May 2025) → 30.0 (Aug 2025) → 30.1 (Dec 2025) → 30.2 (Feb 2026, new four-level Job Zones) → 30.3 (May 2026, modernized Content Model + Specific Interests) → 31.0 expected late August 2026.
• Downloads: https://www.onetcenter.org/database.html — formats: tabular (Excel/CSV/JSON), SQL loads for MySQL/PostgreSQL/MariaDB, SQL Server, Oracle, plus RDF. Full Excel archive: https://www.onetcenter.org/dl_files/database/db_30_3_excel.zip (individual files linked from the same page; per-format data dictionary at https://www.onetcenter.org/dictionary/30.3/excel/).
• Files the ETL needs (30.3 row counts):
  • Occupation Data — 1,016 rows (O*NET-SOC code, title, description)
  • Task Statements — 18,796 rows (task_id, task, task type, incumbents responding)
  • Task Ratings — 161,559 rows (importance IM, relevance RL, frequency FT scales, with N, SE, CI bounds — feeds our task weights and our own CI propagation)
  • Tasks to DWAs, DWA Reference, IWA Reference — task ↔ detailed/intermediate work activity links
  • Work Activities — 73,308 rows (GWA ratings)
  • Abilities — 92,976 · Knowledge — 59,004 · Work Context — 297,676
  • Renamed in 30.x: Technology Skills → Software Skills (31,821 rows); the old Skills file is now Essential Skills (17,880) + Transferable Skills (44,700). ETL loaders and any code referencing "Technology Skills" must be updated.
• License: CC BY 4.0. Required attribution: credit the "O*NET 30.3 Database by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA), used under the CC BY 4.0 license." Put this in the public methodology page and API /api/v1/methodology metadata.
• Taxonomy: ONET-SOC 2019 taxonomy (built on SOC 2018): 1,016 occupation titles of which 923 are data-collection-level; code format XX-XXXX.XX (SOC 6-digit + 2-digit ONET suffix). ~891 occupations already updated in 2026 YTD.
• O*NET Web Services: register (free) at https://services.onetcenter.org/developer/signup; API v2.0 reference at https://services.onetcenter.org/reference. No hard rate limit, but ToS gives per-second/per-day guidance; on 429, retry after ≥200 ms. Always serves the latest DB — good for spot checks, not for reproducible scoring (use pinned dumps).
• SOC 2018: 867 detailed occupations → 459 broad → 98 minor → 23 major groups; code XX-XXXX. Definitions/structure files at https://www.bls.gov/soc/2018/ (XLSX). Public domain.
• Pipeline: fetch script downloads the pinned release ZIP into data/raw/onet/db_30_3/, verifies SHA-256, records release + hash in the derived manifest. Never edit in place.

# 2. ESCO + crosswalks + ROME (EU/France layer)

• ESCO v1.2.1 (last update 2025-12-10). Portal: https://esco.ec.europa.eu/en · downloads (per-language CSV + "language independent" files, SKOS/RDF, Local API): https://esco.ec.europa.eu/en/use-esco/download (free account/selection flow). ~3,000 occupations mapped to ISCO-08; 13,939 skills in v1.2.1. Web API: https://esco.ec.europa.eu/en/use-esco (base https://ec.europa.eu/esco/api).
• License: free reuse under Commission Decision 2011/833/EU; must publish the ESCO acknowledgement statement (see FAQ: https://esco.ec.europa.eu/en/about-esco/faq).
• ESCO↔O*NET official crosswalk (co-created EC + USDOL, AI-assisted with human validation):
  • Page: https://esco.ec.europa.eu/en/use-esco/other-crosswalks (also listed at https://www.onetcenter.org/crosswalks.html)
  • CSV: https://esco.ec.europa.eu/system/files/2023-08/ONET_%28Occupations%29_0_updated.csv
  • Two published variants: (a) exact/narrow/broad/close matches (QA'd, USDOL-validated); (b) same + "related" matches (lower quality — not validated; exclude from scoring joins by default).
  • Technical report: https://esco.ec.europa.eu/system/files/2022-12/ONET%20ESCO%20Technical%20Report.pdf. Built against ESCO v1.1 / O*NET-SOC 2019 (the O*NET Web Services crosswalk endpoint still states ESCO v1.1.0) — re-verify concept URIs against v1.2.1 during ETL; unmatched URIs go to a QA report.
  • Bonus: ESCO↔NACE crosswalk now available: https://esco.ec.europa.eu/system/files/2026-02/ESCO-NACE%20rev.%202.1%20crosswalk.xlsx (useful for joining Eurostat sector-level AI adoption to occupations).
• ESCO↔ROME: no single public EC file. France Travail maintains ROME↔ESCO correspondence tables under the EURES obligation (each member state maps its national classification to ESCO); the ROME open-data bundle includes correspondence referentials, and EURES member-state mapping tables are listed on the ESCO portal ("EURES Countries Mapping Tables"). Validate coverage during ETL and fall back to ROME→ISCO-08→ESCO if a direct table is missing for some fiches.
• ROME 4.0 (France Travail):
  • Open data: https://www.data.gouv.fr/datasets/repertoire-operationnel-des-metiers-et-des-emplois-rome — multiple referential files (arborescence, compétences, contextes, mobilité). Licence Ouverte / Open Licence 2.0. Last update 2026-06-18; next update announced for Oct 2026; ≥2 updates/year.
  • Also via API on https://francetravail.io (ROME 4.0 APIs, OAuth key) and mirrored on https://www.francetravail.org/opendata/.
  • Code format: 1 letter + 4 digits (e.g., M1607); ~600 fiches métiers organized by 14 domaines.
  • Pipeline: data/raw/rome/ with the data.gouv.fr resource URLs + version date in the manifest.

# 3. Wage data

• BLS OEWS — May 2025 (released 2026-05-15; next: May 2026 data in spring 2027). ~830 SOC occupations; employment, mean/median hourly & annual wages, wage percentiles (10/25/50/75/90), by nation/state/MSA/industry.
  • Tables hub: https://www.bls.gov/oes/tables.htm → ZIPs oesm25nat.zip (national), oesm25st.zip (states), oesm25ma.zip (metro), national-by-industry files (served from https://www.bls.gov/oes/special-requests/…; BLS blocks non-browser user agents — use a browser UA in the fetch script). Field layout documented in each ZIP's field_descriptions sheet.
  • License: US government work, public domain. Cadence: annual.
  • Pipeline: national file feeds cost_ratio (wage denominator, stored as integer cents + USD per repo convention) and employment weights; join key = SOC 2018 6-digit → ONET-SOC 2019 (strip .XX suffix / use ONET-SOC↔SOC crosswalk).
• Eurostat SES (Structure of Earnings Survey): 4-yearly, latest reference year 2022 (published 2024–25; next ref-year 2026 published ~2028). Datasets: earn_ses_hourly (and monthly/annual variants) — mean/median hourly earnings by ISCO-08 2-digit × NACE × country. Databrowser: https://ec.europa.eu/eurostat/databrowser/view/earn_ses_hourly · bulk via SDMX API https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/earn_ses_hourly?format=TSV. License CC BY 4.0. Occupation resolution is only 2-digit ISCO — EU cost_ratio will be coarser than US; flag in methodology.
• France: INSEE Base Tous salariés (from DSN, formerly DADS): salaire net EQTP by PCS (up to 4-digit), sector, sex; latest consolidated 2023 (see Insee Première n°1938 for 2021: https://www.insee.fr/fr/statistiques/6799523; séries longues: https://www.insee.fr/fr/statistiques/8660332). DARES publishes wage/employment "portraits statistiques des métiers" by FAP. Requires PCS↔ROME/ISCO crosswalk (INSEE publishes PCS↔ISCO tables) — France-specific wage joins are a v2 concern.

# 4. Employment counts & projections

• OEWS employment (same May 2025 files as §3) — primary US employment weights.
• BLS Employment Projections 2024–34 (released 2025-08-28; 2025–35 edition expected ~Sept 2026 — recheck before ingesting): https://www.bls.gov/emp/data/occupational-data.htm — "All occupational tables in a single file (XLSX)", National Employment Matrix 2024/2034, occupational separations, plus the new skills data tables (importance of skills by occupation). Public domain. Feeds adoption-velocity priors and UI "outlook" context. State-level: https://projectionscentral.org/longterm (REST + download).
• Eurostat LFS: lfsa_egai2d — employed persons by detailed occupation (ISCO-08 2-digit), annual: https://ec.europa.eu/eurostat/databrowser/view/lfsa_egai2d (SDMX API as above). CC BY 4.0.
• ILOSTAT: https://ilostat.ilo.org — harmonized employment by ISCO level 2 (annual/quarterly) across countries, CSV bulk + API, CC BY 4.0. Notably now publishes "Employment by sex and generative AI exposure" tables (based on the ILO/Gmyrek GenAI occupational exposure scores) — both a benchmark and a ready-made employment-by-exposure aggregate.

# 5. AI adoption data (adoption_velocity dimension)

• US Census BTOS AI supplement: third AI supplement collected 2025-11-17 → 2026-02-08, released 2026-04-23 (press: https://www.census.gov/newsroom/press-releases/2026/btos-apr-23.html). Measures firm AI use overall and — new this cycle — by worker tasks and business functions, split by NAICS sector, state, and firm size. Data hub: https://www.census.gov/hfp/btos/data (Downloads tab: https://www.census.gov/hfp/btos/data_downloads, incl. historical; API tab available). Core biweekly BTOS also carries a recurring "AI use in last two weeks" item (wording revised Nov 2025 — treat as a series break). ~1.2M-business sample; experimental data product; public domain.
• Eurostat — AI in enterprises: datasets isoc_eb_ai (enterprises using AI technologies) and isoc_eb_ain2 (by purpose/technology × NACE), survey year 2025: 19.95% of EU enterprises use AI (55.03% of large firms); Statistics Explained article (updated Dec 2025, next Dec 2026): https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Use_of_artificial_intelligence_in_enterprises. Join to occupations via sector (NACE) using the ESCO-NACE crosswalk (§2). CC BY 4.0.
• Ramp AI Index (Ramp Economics Lab): https://ramp.com/data — monthly AI adoption among US firms from card/bill-pay spend of 70k+ businesses (June 2026: 55.0%, +0.8 pp MoM), broken out by size and sector, downloadable; methodology: https://econlab.substack.com/p/how-ramp-data-works. Now shifting to intensity tracking. Free/open resource; confirm redistribution terms before committing derived aggregates.
• Anthropic Economic Index: HF dataset Anthropic/EconomicIndex (https://huggingface.co/datasets/Anthropic/EconomicIndex). Release folders: release_2025_02_10 (initial ONET task mappings, automation vs augmentation), release_2025_03_27 (cluster-level, thinking-mode fractions per ONET task), release_2025_09_15 (geography + 1P API), release_2026_01_15 ("economic primitives"), release_2026_03_24 ("learning curves"), release_2026_06_26 ("Cadences", monthly aggregates), plus labor_market_impacts/. Data CC-BY (repo metadata lists MIT for code). This is the single most direct empirical input for calibrating our LLM-rater automatability/feasibility scores against observed usage — it is keyed to O*NET task statements, same spine as ours.
• Stanford AI Index 2026 (9th edition, April 2026, 423 pp): report PDF https://hai.stanford.edu/assets/files/ai_index_report_2026.pdf; public data appendix downloadable from the HAI AI Index page. Key 2026 stats: 53% population-level GenAI adoption; $172B est. US consumer surplus. Use for methodology narrative and barriers/adoption_velocity context, not row-level joins. Annual.

# 6. Published occupation-level exposure benchmarks

These are validation benchmarks (data/raw/benchmarks/), not scoring inputs — our methodology must remain independently reproducible.

Use for the sensitivity analyses in docs/methodology/sensitivity/ (rank correlations of our composite vs AIOE / GPTs-are-GPTs / AEI).

# Ingestion order & manifest plan

Order respects join dependencies (occupation spine first, then attributes, then joins):

1. SOC 2018 structure (data/raw/soc/) — code spine, 867 detailed occupations.
2. O*NET 30.3 full dump (data/raw/onet/db_30_3/) — pin release; load order: Occupation Data → Task Statements → Task Ratings → Tasks-to-DWAs/IWAs → Work Activities → Abilities/Knowledge → Software/Essential/Transferable Skills → Work Context. Decision needed before build: stay on 30.3 or wait for 31.0 (late Aug 2026) — either way, INDEX_VERSION notes the O*NET release.
3. OEWS May 2025 (data/raw/bls/oews/) — wages (→ integer cents) + employment; join on SOC 6-digit.
4. BLS EP 2024–34 (data/raw/bls/ep/) — projections + skills tables.
5. ESCO v1.2.1 (data/raw/esco/classification/) + ESCO↔O*NET crosswalk (data/raw/esco/crosswalk/) — validate URI coverage vs v1.2.1; exclude "related" matches from scoring joins.
6. ROME 4.0 (data/raw/rome/) + ROME↔ESCO correspondence; fallback path ROME→ISCO-08→ESCO.
7. Eurostat: earn_ses_hourly (SES 2022), lfsa_egai2d (LFS), isoc_eb_ai(n2) (AI adoption) via SDMX API pulls (data/raw/eurostat/).
8. Adoption signals: BTOS AI supplement, Ramp AI Index, AEI releases (data/raw/btos|ramp|aei/).
9. Benchmarks: AIOE, GPTs-are-GPTs, ILO GenAI exposure (data/raw/benchmarks/) — validation only.

Manifest rules (per repo convention — raw payloads are gitignored, manifests committed): for every fetched artifact record {source, version_label, source_url, fetch_date, sha256, row_count, license, attribution_string} in data/derived/manifests/<source>.json. The fetch script must fail loudly if a pinned URL 404s or the hash changes (BLS and ONET replace files in place across releases). Re-verify before each INDEX_VERSION bump: ONET quarterly page, OEWS annual page (mid-May), EP annual page (late Aug/Sep), ESCO portal (point releases), ROME (Oct 2026), BTOS supplement announcements, AEI HF repo.

Attribution block for the public methodology page: O*NET (USDOL/ETA, CC BY 4.0) · ESCO (© European Union, ESCO acknowledgement statement) · ROME (France Travail, Licence Ouverte 2.0) · BLS/Census (public domain, cite program + vintage) · Eurostat (CC BY 4.0) · Anthropic Economic Index (CC-BY) · benchmark papers cited per their requirements.