# Operations — WP41

## Nodes
- **Compute: M3U96b** (`~/wp41_uqo`, venv `.venv` Python 3.12, torch 2.15 MPS). Cache `~/wp41-cache/{bars,instruments,grids,manifests}`.
- **Data: M3U96a** FinData OS API, reached over the LAN: `FINDATA_BASE_URL=http://192.168.2.80:8600/api/v1`.
- **Laptop**: mirror of the repository (`~/Desktop/UQO/Recherche/Recherche-UQO/finance/wp41_uqo`), tests, LaTeX.

## Environment on the node
```bash
export FINDATA_API_KEY_FILE=~/findata-secrets/api-key-wp41.txt      # mode 600, dedicated key (scopes read/query/export)
export FINDATA_BASE_URL=http://192.168.2.80:8600/api/v1
export WP41_RATE=8                                                  # client-side req/s (server limit 600/min)
export WP41_CACHE=~/wp41-cache
```

## Sync and run
```bash
# laptop -> node (code only; caches and results stay on the node, metrics are rsynced back)
rsync -a --exclude .venv --exclude .git --exclude results/logs ./ M3U96b:~/wp41_uqo/
ssh M3U96b 'cd ~/wp41_uqo && uv pip install -q -p .venv/bin/python -e . && .venv/bin/python -m pytest -q'
ssh M3U96b 'cd ~/wp41_uqo && nohup .venv/bin/python scripts/download_pilot.py --config configs/main.yaml --symbols-from all --workers 3 > results/logs/download_main.out 2>&1 &'
ssh M3U96b 'cd ~/wp41_uqo && nohup .venv/bin/python scripts/run_experiment.py --config configs/main.yaml > results/logs/main.out 2>&1 &'
# node -> laptop (aggregate results only)
rsync -a --exclude 'origins_*.csv' --exclude '*.npz' --exclude '*.pt' M3U96b:~/wp41_uqo/results/ results/
```

## Logs and provenance
- `results/logs/provenance_*.jsonl`: one line per API call (method, path, status, request id, dataset id/version, ms).
- `~/wp41-cache/manifests/bars_manifest.jsonl`: one line per export (dataset version, sha256, rows, request id, time).
- `results/metrics/<exp>/<model>/train_log.jsonl`: training curves, throughput, memory; `last.pt` resumes a crashed run.

## Safety
- Never print the key; the client redacts `fd_…` tokens in exceptions and logs.
- Downloads stop when free disk < 40 GB (`min_free_gb`). Exports are bounded (≤ 500 symbols, ≤ 12 years per request).
- No production app is touched; the node's PM2 apps (toveli, autoscholar, wp-web) keep running.

## Gotchas
- **Segfault when LightGBM is imported after torch** (two copies of libomp on macOS): every entry point imports `lightgbm` first (`scripts/run_experiment.py`, `chronos_x/experiment.py`). Verified on M3U96b with torch 2.15 / lightgbm 4.x.
- Detached runs: use `scripts/node_run.sh <name> <cmd>` (nohup + `< /dev/null` + disown); a plain `ssh host 'nohup … &'` was killed when the SSH channel closed.