1# PROGRESS.md — WP39 `wp39_uqo`23## Phase 0 — Discovery and feasibility (2026-11-02) — GATE PASSED4- Inspected the UQO Research workspace (`Recherche-UQO/`, 39 papers, harmonisation standard); no `wp39` existed; created5 `finance/invisible-balance-sheet/` from the template; git initialised on a dedicated repository (no other paper touched).6- Probed FinData: `/health` ok, 64 datasets, OpenAPI 3.1 (45 routes), bars schema v1, fundamentals (67 metrics, 5,4517 entities, 2012→2026), earnings events (Postgres), coverage endpoints, export throughput (≈ 1 s per symbol-decade;8 12-year cap per request). Created least-privilege key `wp39_uqo`.9- Verified external public sources: SEC `company_tickers.json` + `submissions` (CIK/SIC), Kenneth French daily factors.10- Verified 94 bibliography entries on Crossref; wrote `docs/NOVELTY_MEMO.md`, `literature/RELATED_WORK_MATRIX.csv`.11- Outputs: `DATA_AUDIT.md` (viability = YES with survivorship + PIT caveats), `configs/*.yaml`, `docs/API_INTEGRATION.md`,12 `docs/COMPUTE_BUDGET.md`, `docs/PIT_PROTOCOL.md`, `docs/RESEARCH_DESIGN.md` (frozen), `DECISIONS.md` D-001…D-010.13- Owner instructions received during the session: (1) the fundamentals API must be used — it is the accounting source;14 (2) minimise AI usage — no LLM features, no neural models (D-006).15- Blockers: none. Next safe step: Phase 1 gate memo, then code + tests (Phase 2).1617## Phase 1 — Research design and novelty — GATE (self-approved under the autonomous mandate, D-010)18- `docs/RESEARCH_DESIGN.md` frozen 2026-11-02 14:40 ET; hypotheses H1–H5 with null/alternative, primary endpoint (H1,19 h = 20, log loss, binary adverse operating-income surprise), decision rules, splits, inference, power plan.20- Owner review pending; nothing in Phases 2–5 depends on test-period data until the Phase 3 gate.2122## Phase 2 — Data acquisition and verification — IN PROGRESS23- 2026-11-02 13:31 — `01a_extract_node_tables.sh`: entities 5,451 · earnings_events 364,783 · corporate_actions 249,340 (raw format24 `firstrate:stock:<SYM>` / `<SYM>_DIVS`, loader adapted). `00_probe`: health ok, 67 datasets, schema v1 ok, no earnings route.25- 13:33–13:50 — `01_audit_universe`: 5,451 entities → 5,018 with ≥ 8 quarters → **4,568 eligible** (1m coverage ≥ 250 sessions);26 CIK matched 5,073, SIC 5,029 (SEC submissions fetched with a 6-thread, 8 req/s limiter after a sequential first attempt27 at ~1 req/s was restarted); 2,522 firms have former names (ticker-reuse audit input). `results/universe_audit.json`.28- 13:50–14:01 — `02_build_pit_panel`: first run hit the API rate limit (HTTP 429, 600 req/min) with 6 unthrottled threads →29 client now has a process-wide token bucket (`WP39_RATE`, default 6 req/s) and honours `Retry-After`; rerun succeeded:30 13,052,739 facts (all revision 1), **194,817 firm-quarters**, release match ok 172,328 (88.5 %), lag > 30 d 15,242,31 no release 6,020, filing before release 1,227; release timing after_close 76,752 / before_open 66,719 / unknown 51,346;32 base rates: adverse op-inc 31.2 %, adverse NI 32.7 %, ROA decline 29.4 %, OCF negative 25.9 %, liquidity 47.7 %,33 |S| ≥ 0.5 % 61.4 %; S_opinc quartiles −0.87 % / 0.00 % / +0.82 % (heavy tails, winsorised in continuous models).34 Design freeze hash written to `artifacts/manifests/design_freeze.json` before any model fit.35- 14:01 — `03_build_features --workers 8` (WP39_RATE=1 per process) running: streaming export → features → windows.36- 14:01–14:59 — `03_build_features`: 4,568/4,568 symbols ok, 0 errors (58 min, 8 workers, ≈1.3 symbols/s); raw exports37 deleted after use; `~/wp39-cache/features` 5.1 GB + `windows` 5.9 GB. `04_build_labels` (28 s): event sample38 **481,159** observations (3,796 firms, 102,522 firm-quarters; h = 20: 97,788, base rate 30.1 %), calendar sample 258,48039 firm-months; cascade in `results/table_sample_cascade_event.csv`. **Phase 2 gate passed**: zero known forward-looking40 features (tests + live as-of equivalence), coverage gaps explained (2013 start = 8 prior quarters required).4142## Phase 3 — Descriptives and baselines (2026-11-02 15:00–15:01) — GATE PASSED43- `05_train_baselines` (30 s): B0–B5 logistic (C tuned on validation only), ridge continuous target, firm hold-out,44 walk-forward OOF 2016–2026 (86,791 rows per set). Validation log loss h=20: B1 0.5938 → B2 0.5691 → B3 0.5695 /45 B4 0.5688 / B5 0.5706. Test set opened only after this gate (results in Phase 4/5).4647## Phase 4 — Advanced models (15:01–15:30) — GATE PASSED48- `06_train_advanced` (29 min): LightGBM, 24-trial equal budget per information set (LB1–LB5, LB5all), secondary targets,49 horizon ladder with a fixed configuration, firm hold-out, walk-forward OOF (event + calendar). Validation: LB1 0.5447 →50 LB2 0.5214 → LB3 0.5234 / LB4 0.5239 / LB5 0.5245 / all 0.5259.51- `07_evaluate`: first run failed on a dtype mismatch (Int32/Int64 `fold`), fixed, rerun ok. **Test period 2021–2026,52 53,667 obs**: Δ log loss B5−B2 = +0.0013 [−0.0013, +0.0041] (GW p = 0.52); LB5−LB2 = +0.0016 [+0.0010, +0.0023]53 (p = 0.0008, a deterioration); LB2−LB1 = −0.0068 [−0.0095, −0.0042]; AUC LB2 0.803, LB5 0.801. H1 and H2 rejected.54 H4: no horizon gradient (Spearman −0.5, p = 0.39/0.33). Secondary targets, firm hold-out, continuous target: same null.55 MDE (80 % power, 7 validation quarters): 0.0025.5657## Phase 5 — Robustness and interpretation (15:30–15:58)58- `08_run_robustness`: first run stuck 28 min in a row-wise pandas loop (CAR window) → vectorised in Polars, rerun in 25 s.59 Placebo (shuffled labels, 3 draws): AUC 0.50, Δ ≈ −0.0004 (noise floor). Subgroups: no gain anywhere; deterioration60 smallest for small/illiquid/loss firms. Panel: LFDS_B2 β = −0.0765 (t = −32.2); incremental d_lfds β = −0.012961 (t = −1.76); interactions opposite to H3 (less informative for small and low-leverage firms). H5: CAR on LFDS5 β ≈ 062 (t = 0.01); d_lfds +0.060 (t = 2.9, sign opposite to a deterioration signal); portfolios L/S alpha −0.59 %/mo (t = −1.85)63 gross, −0.62 % net for LB5; −0.49 % (t = −1.64) for LB2 → no economic value.6465## Phase 6 — Manuscript (15:58–16:10)66- `09_make_manuscript`: 15 LaTeX tables, 9 figures (validated palette), 206 macros in `paper/results_macros.tex`67 (fixes: NaN subgroup in figure, LaTeX escaping of math labels, Unicode in bib). `paper/main.pdf` compiled with68 `latexmk` on the laptop: 33 pages, 0 undefined references/citations.6970## Typography and figure pass (2026-11-02 16:30–17:00)71- Paper: `newtxtext` + `newtxmath` (Times math with proper metrics), `mathtools`, `bm`, `dsfont`; notation macros72 (`\E`, `\ind{}`, `\Info`, `\Acc`, `\Loss`, `\LFDS`, `\RV`, `\BPV`, `\RS`, `\TA`, `\dlog`); displayed definitions for the73 surprise and its seasonal expectation (eq. 1), the log loss (eq. 2), the information set (eq. 3), the realized-measure74 families (eq. 4), the panel regression and the Giacomini–White statistic; author names normalised in the bibliography.75- Figures regenerated in journal style (`src/wp39/figures.py`): Times/STIX typography, monochrome marks distinguished by76 shape and line style with a single restrained accent for the 1-minute series, Panel A/B labels, inward ticks, no grid,77 journal widths (3.25 / 6.5 in), redesigned timeline schematic, ordinal horizon axis. Tables: grouped rules by family.78- `paper/main.pdf`: 34 pages, 0 undefined references.79