# Replication guide (WP47)

Computation runs on the MacLustr node **M3U96a** in `~/wp47_uqo` (FinData OS API on `http://127.0.0.1:8600`); the laptop folder
`~/Desktop/wp47-nelson-siegel-intraday` holds the code, the public inputs, the results and the LaTeX. Code goes node-ward with
`rsync -a --exclude data/cache --exclude .git --exclude paper . M3U96a:~/wp47_uqo/`; results come back with
`rsync -a M3U96a:~/wp47_uqo/results/ results/`.

## Environment (node)
```bash
cd ~/wp47_uqo && uv venv -p 3.12 .venv
uv pip install -p .venv/bin/python numpy pandas scipy statsmodels polars pyarrow matplotlib requests pyyaml arch
```
API key: `~/findata-secrets/api-key-wp47.txt` on the node (laptop copy `~/Desktop/Cluster/secrets/findata/api-key-wp47.txt`),
or `$FINDATA_API_KEY`; scopes read:market, query, export. On the laptop set `FINDATA_URL=https://findata.spboucher.ai`.
Never print or commit the key.

## Public inputs (`data/external/`, download timestamp in `DOWNLOADED_AT.txt` = 2026-11-09T19:08:56Z)
- `fred/DGS1MO … DGS30.csv`, `fred/DFF.csv` — FRED H.15 constant-maturity yields and effective fed funds (fredgraph CSV).
- `fed/feds200628.csv` — Gürkaynak–Sack–Wright zero-coupon curve and Svensson parameters (Federal Reserve Board).
- `fomc/fomchistorical2008–2020.html`, `fomc/fomccalendars.html` → `fomc_dates_parsed.json` (meeting end dates, statement dates).
- `bls/cpi_release_dates_alfred_rid10.txt`, `bls/empsit_release_dates_alfred_rid50.txt` — release-date lists read from the
  ALFRED release pages (rid 10 = CPI, rid 50 = Employment Situation) on 2026-11-09; revision releases are filtered in `events.py`.

## Pipeline (run in this order on the node; times measured on M3U96a, 2026-11-09)
| Script | Output | Time |
|---|---|---|
| `01_extract_futures.py` | `data/cache/futures_1m/{ROOT}_{a}_{b}.parquet` (vendor continuous series, kept for the EDA and SR3 roll check), `futures_1d/continuous_{raw,ratio}_*.parquet` | ~2 s per export block |
| `01b_extract_sr3.py` | `futures_1m/SR3_2018_2026.parquet`, `futures_1d/*_SR3_*`, `returns_1m_SR3.parquet`, `sr3_daily.parquet` | ~10 s |
| `01c_extract_etf.py` | `etf_1m/etf_{2010_2018,2019_2026}.parquet` (SHY IEI IEF TLH TLT, split/dividend adjusted) | ~5 s |
| `01d_extract_contracts.py` | `data/cache/contracts_1m/{ROOT}_{a}_{b}.parquet` — all individual contracts of the six roots (18 exports, 326 MB; dataset `market_bars.futures_contracts.1m.raw`) | ~5.6 min |
| `02_daily_factors.py` | `results/metrics/daily_factors_{cmt,cmt_lambdahat,gsw}.csv`, `daily_factors_summary.json` | ~5 s |
| `03_returns.py` | `data/cache/returns_1m.parquet` (most-traded contract per session, `act_*` flags), `returns_daily.parquet`, `results/metrics/returns_diagnostics.json` | ~15 s |
| `04_mimicking.py` | `mimicking_weights_quarterly.json`, `mimicking_fidelity.json`, `data/cache/mimicking_oos_daily.parquet` | ~2 s |
| `04b_fidelity_gsw_and_sofr.py` | `mimicking_fidelity_robustness.json`, `daily_factors_cmt_1y_plus.csv` | ~3 s |
| `04c_targets_and_spanning.py` | `targets_and_spanning.json`, `daily_factors_cmt_2y_plus.csv` | ~5 s |
| `04d_robustness_checks.py` | `robustness_checks.json` (expanding window, 15:00/16:00 boundaries, 5-min denominators, FOMC release-minute check) | ~2 s |
| `05_events_list.py` | `results/metrics/events.csv` (runs anywhere) | <1 s |
| `06_intraday_factors.py` | `data/cache/factors_1m.parquet`, `factors_daily_realized.parquet` (+ csv), `realized_correlations.json`, `intraday_periodicity.csv`, `fidelity_cumulated_intraday.json` | ~10 s |
| `06b_etf_crossval.py` | `data/cache/factors_1m_etf.parquet`, `etf_crossval.json` | ~60 s |
| `06c_noise.py` | `noise_benchmark.json` | <1 s |
| `07_events.py` | `event_windows.csv`, `event_windows_placebo.csv`, `event_summary.json`, `event_paths.json` | ~8 s |
| `08_har.py` | `har_forecasts.json`, `data/cache/har_preds_{L,S,C}_h{1,5,22}.parquet` | ~30 s |
| `13_irf.py` | `irf.json`, `data/cache/lp_events.parquet`, `results/figdata/{lp_irf,var_girf,vol_connectedness_rolling}.csv`, `paper/tables/tab_lp.tex`, `tab_connect.tex` (copy the two .tex files back to the laptop) | — (not timed) |
| `09a_export_figure_data.py` | `results/figdata/*.csv` (sample days, fidelity by year with bands, event paths with bootstrap bands, heatmap matrices, ETF pairs) | ~10 s |
| laptop `11_tables.py` (python3 -I) | `paper/tables/*.tex` (except tab_lp/tab_connect), `paper/numbers.tex`, `docs/NUMBERS_INDEX.md` | — |
| laptop `12_text_macros.py` (python3 -I) | `paper/numbers_text.tex` (text macros, `\providecommand`), `results/metrics/implied_yield_fidelity.json` | — |
| laptop `10_figures.py` (`--audit` renders colour + gray PNGs) and `14_irf_figures.py` | `paper/figures/*.pdf` (12 figures) | — |

Then `cd paper && latexmk -pdf main.tex` on the laptop (45 pages for v1.0-rc). Macro names in `numbers*.tex` must not contain digits.

`00_eda_futures.py` is an optional data-quality scan (`data/cache/eda_futures.json`). Then `cd paper && latexmk` on the laptop.

Determinism: the only random element is the bootstrap/placebo sampling in `07_events.py` (`numpy.random.default_rng(47)`).
Dataset versions are pinned by the FinData export (`version` field in `data/cache/extract_log.jsonl`: futures continuous 1m v132, daily v133; contracts 1m v19699 printed by `01d`). Random elements: bootstrap/placebo sampling in `07_events.py` (`default_rng(47)`), `13_irf.py` (`default_rng(13)`), `09a` bands.

## Licence constraint
FirstRate Data (FinData OS) is licensed for private use. `data/cache/` (raw and derived minute data, including `contracts_1m/`) stays on the node and is
git-ignored; only aggregates (daily realized measures, event-window sums, weights, figure data) leave the node and enter the paper.