# data/ — empirical inputs for WP 24 (“From Labor to Capital: AGI, General Equilibrium, and Asset Valuation”)

Retrieved **2026-10-27**, key-less endpoints only, no number typed from memory. Two scripts rebuild everything:

```bash
python3 -I scripts/01_download_data.py            # raw downloads -> data/raw/<source>/ + data/raw/manifest.json (idempotent; --force to refetch)
python3 -I scripts/02_build_calibration_targets.py # -> data/processed/calibration_targets.csv, data/processed/series/*.csv, bea_io_value_added_2023.csv, epoch_price_performance_fit.csv
```
Dependencies: `requests`, `pandas`, `numpy`, `openpyxl`, `xlrd` (installed 2026-10-27 for the Shiller `.xls` files), `PyMuPDF` (`fitz`, to read the LBNL PDF).
Raw files are untrusted data and are never executed. `data/raw/` and `data/processed/` are git-ignored (regenerate them); only the two READMEs are versioned.

## Layout
| Path | Content |
|---|---|
| `raw/README.md` | one block per source: what, URL, size, vintage, licence/terms |
| `raw/manifest.json` | per-file URL, retrieval date, size, SHA-256, HTTP content type, status (93 entries, all RETRIEVED) |
| `raw/fred/_metadata.json`, `raw/fred/_RETRIEVAL_NOTE.md` | FRED titles/units/vintages; why the FRED csv files were fetched with WebFetch |
| `processed/calibration_targets.csv` | 162 rows: `target_id, description, value, unit, year_or_period, geography, frequency, source, series_id, url, raw_file, retrieved, status, notes` |
| `processed/series/*.csv` | 92 tidy `date,value` series (every FRED series, Shiller, DFA, Treasury, Cleveland Fed, StatCan, BLS API) |
| `processed/bea_io_value_added_2023.csv` | 71 BEA summary industries: compensation, taxes, gross operating surplus, value added, intermediates, gross output, labor share, intermediate share |
| `processed/epoch_price_performance_fit.csv` | 60 accelerators used in the FLOP/s-per-dollar regression |

## Targets and transformations (US unless noted; “avg” = mean over 1990–latest full year of calendar-year means)
| Block | target_id prefix | Construction |
|---|---|---|
| 1 Labor share | `T01_labor_share_BLS_nonfarm_index_*` | BLS PRS85006173, **index 2017=100** (not a level). Latest 2026Q2 = 93.4; 2025 = 96.6; avg 103.9 |
| | `T01_labor_share_COE_GDI_*` | COE / GDI (both $bn SAAR). Latest 0.499 (2026Q2); 2025 0.508; avg 0.542 |
| | `T01_labor_share_COE_NDI_*` | COE / net domestic income W256RC1Q027SBEA. Latest 0.597; avg 0.642 |
| | `T01_labor_share_PWT_US_*`, `T12_labor_share_PWT_CA_*` | PWT `labsh` via FRED (PWT 11.0, to 2023): US 0.568 (2023), avg 0.604; Canada 0.659, avg 0.663. PWT 10.01 spreadsheet cross-check (`*_PWT1001_*`, to 2019): US 0.597, CA 0.655 |
| | `T01_labor_share_BEA_IO_2023_*` | 2023 summary use table: Σ compensation / Σ value added = 0.513 (all 71 industries incl. government); compensation/(compensation+GOS) = 0.548; intermediates / gross output = 0.427; selected industries (utilities 0.21, computer systems design 0.91, publishing incl. software 0.23, computer & electronic products 0.57, construction 0.61, professional services 0.74, ambulatory health 0.77) |
| 2 Profits | `T02_profits_after_tax_noIVA_GDP_*` | FRED **CP = after tax without IVA/CCAdj**: 0.131 (2026Q2), 2025 0.114, avg 0.086 |
| | `T02_profits_after_tax_IVA_CCAdj_GDP_*` | FRED **CPATAX = with IVA & CCAdj**: 0.119, 2025 0.110, avg 0.084 |
| | `T02_net_dividends_GDP_*` | DIVIDEND / GDP: 0.0675 (2026Q2), 2025 0.070, avg 0.051; annual B056RC1A027NBEA/GDPA 2025 0.070 |
| | `T02_GDP_*`, `T02_real_GDP_growth_avg` | GDPA 2025 = 30,861 $bn; GDP 2026Q2 = 32,563 $bn SAAR; GDPC1 2026Q2 = 24,408; real growth 1990–2025 = 2.48 %/yr (log) |
| 3 Household wealth | `T03_household_net_worth_GDP_*` | TNWBSHNO ($mn, end of quarter) / GDP ($bn SAAR) ×1e-3: **6.02** (2026Q2), 2025 5.72, avg 4.58 |
| | `T03_household_real_estate_GDP_*`, `T03_owners_equity_real_estate_GDP_*`, `T03_owners_equity_share_of_real_estate_*`, `T03_real_estate_share_of_net_worth_*` | HNOREMQ027S/GDP 1.66 (avg 1.47); OEHRENWBSHNO/GDP 1.10 (avg 0.81); HOEREPHRE 71.9 % (avg 60.7 %); real estate/net worth 0.276 (avg 0.322) |
| 4 Wealth concentration | `T04_wealth_share_*` | DFA `dfa-networth-shares.csv`, Net worth column, 2026Q2: top 0.1 % 15.0, **top 1 % 32.5**, **top 10 % 68.9**, p50–90 28.8, **bottom 50 % 2.3** (percent); 1990–2025 averages 11.3 / 28.0 / 65.3 / 32.4 / 2.3. FRED mirrors WFRBST01134 / WFRBSN09161 / WFRBSN40188 / WFRBSB50215 agree. Top 10 % hold 88.1 % of corporate equities & mutual fund shares |
| 5 Investment | `T05_*_GDP_*` | GPDI/GDP 0.179 (avg 0.175); PNFI/GDP 0.145 (avg 0.130); information-processing equipment Y034RC1Q027SBEA/GDP 0.0257 (avg 0.0214); software B985RC1Q027SBEA/GDP 0.0249 (avg 0.0154); IPE+software A679RC1Q027SBEA/GDP 0.0505; IPP Y001RC1Q027SBEA/GDP 0.0577 |
| 6 Capital | `T06_net_stock_private_fixed_assets_GDP_*` | current-cost net stock K1PTOTL1ES000 (end-2024, $mn)/GDPA ×1e-3 = **2.39** (avg 2.25); nonresidential K1NTOTL1ES000: 1.22 (avg 1.20) |
| | `T06_depreciation_rate_*` | current-cost depreciation / end-of-year current-cost net stock: private total M1PTOTL1ES000/K1PTOTL1ES000 = **0.0568** (2024; avg 0.0559); nonresidential 0.0890 (avg 0.0843); NIPA private CFC/net stock 0.0569 |
| | `T06_CFC_*` | COFC/GDP 0.165 (avg 0.155); private CFC A024RC1A027NBEA/GDPA 0.137 (2025) |
| 7 Real rate | `T07_TIPS10_*` | DFII10 daily: 2.66 % (2026-10-23), 2025 mean 1.96 %, 2003–2025 mean 0.97 %; Treasury.gov daily real curve 10 YR identical (1.963 / 0.973); Cleveland Fed REAINTRATREARAT10Y 2.47 % (2026-10), 2025 1.71 %, 1990–2025 avg 1.78 %; GS10 nominal 4.99 % (2026-09), 2025 4.29 % |
| 8 Equity valuation | `T08_*` | Shiller `ie_data.xls`: CAPE 30.8 (2023-09), 1990–2022 avg 26.4, 1881–2022 avg 17.3; D/P 0.0158 (2023-06), avg 0.0203; E/P 0.0417, avg 0.0461. **Vintage caveat: the Yale file ends 2023-09** |
| 9 Housing | `T09_*` | Shiller real HPI (1890=100) 209.3 (2023-01); real growth 1.54 %/yr 1990–2023, 0.56 %/yr 1890–2023. Case-Shiller national nominal growth 4.15 %/yr 1990–2025, real (CPI-U) 1.63 %/yr 1990–2024; CPI rent growth 3.27 %/yr. **Rent-price proxy** = (CPI rent / Case-Shiller), both rebased Jan-2000 = 100: 74.8 (2026-07), avg 91.1 — an index of the change in the rent-price ratio since 2000, not a level |
| 10 Energy | `T10_*` | LBNL 2024 report (PDF p. 7 [printed 5] and p. 53 [printed 52]): data centers **176 TWh in 2023 = 4.4 %** of US electricity consumption; 76 TWh = 1.9 % in 2018; 2028 scenarios 325–580 TWh = 6.7–12.0 % (p. 8 [6], p. 53 [52]). EIA Table 1.1: net generation 4,425 TWh (2025), 4,309 (2024); retail sales 4,080 TWh (2025), 3,975 (2024) |
| 11 Compute | `T11_*` | Epoch `ml_hardware.csv`, column `Price-performance` (= `Max performance` FLOP/s ÷ release price USD), 60 accelerators 2008–2026: OLS of log2 on release year gives a **doubling time of 2.02 years** (R² in notes), 2.02 years on 2015+. Caveat row: `Max performance` mixes precisions (FP32 for old GPUs, tensor FP16/FP8 for recent ones) |
| 12 Canada | `T12_*` | StatCan 36-10-0103-01: COE / GDP at market prices (SAAR) 0.492 (2026Q2), avg 0.506; 36-10-0580-01: households+NPISH net worth (market value) / GDP 5.65 (2026Q2), avg 4.25 |
| 13 Electricity price | `T13_*` | BLS APU000072610: 0.196 $/kWh (2026-09), 2025 mean 0.185, 1990–2025 avg 0.118 |
| cross-checks | `T00_crosscheck_BLS_API_*` | same three BLS series pulled through the BLS API v1: identical to the FRED copies |

Conventions: ratios of a quarterly stock (end of quarter) to quarterly GDP use SAAR GDP of the same quarter; annual Fixed-Assets stocks (end of year) are divided by annual GDP of the same year; depreciation rates divide depreciation during year t by the end-of-year-t stock (not the lagged stock). “Latest full calendar year” requires 4 quarterly / 11–12 monthly / ≥240 daily observations (11 because BLS published no October-2025 CPI/APU values).

## Explicit list of what could NOT be retrieved
* **No calibration target is NOT_RETRIEVED** (161 RETRIEVED, 1 INFO row for the Epoch caveat).
* Access paths that failed and were replaced (documented in `raw/README.md`, `raw/fred/_RETRIEVAL_NOTE.md`, `raw/manifest.json`):
  1. **FRED by direct HTTP** (`requests`/`urllib`/curl) — fred.stlouisfed.org timed out from this network on 2026-10-27 (also from the OVH server); the 43 csv files are verbatim copies obtained with the Claude Code WebFetch tool; `01_download_data.py --force` re-downloads them when FRED answers.
  2. **LBNL host** for the data-center report (HTTP 202 challenge page) — replaced by the eScholarship copy of the same report (DOI 10.71468/P1WC7Q).
  3. **BEA per-file xlsx URLs** (`apps.bea.gov/industry/xls/io-annual/...`) return “page not found” — replaced by BEA's own `AllTablesIO.zip`, which contains the same `IOUse_After_Redefinitions_PRO_1997-2023_Summary.xlsx`.
  4. **US Treasury “all years” csv** returns 403 — fetched year by year instead (2003–2026).
* Known limitations rather than failures: Shiller `ie_data.xls` and `Fig3-1.xls` on the Yale site end in 2023 (CAPE, D/P, E/P and the long real home-price index are therefore 2023 vintages; Case-Shiller via FRED runs to 2026-07); the rent-price ratio is a proxy index (no key-less level series); EIA direct consumption/losses are not in Table 1.1 (retail sales used); the Epoch doubling time is our own regression on Epoch's columns, not Epoch's published estimate; CSUSHPISA values are not redistributed in the targets (ratios only).