SPB Git
spboucher.ai

spb/wp29_uqo

Public

Preregistration v1.0 (pilot-informed decisions), recovery drivers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 2c82656 Browse files

82 changed files +4,987 −2

82 changed files
Load all 82 files Unified Split
modified protocol/preregistration.md +9 −1
@@ -1,4 +1,6 @@
1−# Preregistration — WP29 (DRAFT v0.9, written 2026-10-30 before the pilot; to be frozen as v1.0 with hashes before the confirmatory stage) 1+# Preregistration — WP29 (v1.0, frozen 2026-10-30 after the pilot and before any confirmatory run; hashes in protocol/freeze_manifest_conf.json)
2+
3+> Version history: v0.9 drafted 2026-10-30 ≈ 04:00 UTC before the pilot (git 76de75e…2c82656); v1.0 adds §10 (pilot-informed decisions) only. No hypothesis, endpoint, contrast or analysis rule was changed after the pilot.
2 4
3 ## 1. Research question 5 ## 1. Research question
4 Under controlled information, tool access and resource conditions, does an explicit research protocol, combined with executable verification and persistent evidence records, improve Claude Code's ability to complete scientifically valid research tasks autonomously, and at what cost in compute, time and human effort? 6 Under controlled information, tool access and resource conditions, does an explicit research protocol, combined with executable verification and persistent evidence records, improve Claude Code's ability to complete scientifically valid research tasks autonomously, and at what cost in compute, time and human effort?
@@ -31,3 +33,9 @@ The confirmatory stage runs the frozen assignment once; no run is added or remov
31 33
32 ## 9. Freeze procedure 34 ## 9. Freeze procedure
33 Before any confirmatory run: hashes of every confirmatory bundle (`bundle.json`), every oracle (`oracle.json` + `evaluate.py`), the four arm texts, the common preamble, the reviewer prompt, the gate program, `configs/*.yaml`, the analysis scripts' interfaces, and this document are recorded in `protocol/freeze_manifest.json` and in `tasks/registry.csv` (`frozen_utc`). Any later change is an amendment (`protocol/amendments.csv`) with before/after-results status. 35 Before any confirmatory run: hashes of every confirmatory bundle (`bundle.json`), every oracle (`oracle.json` + `evaluate.py`), the four arm texts, the common preamble, the reviewer prompt, the gate program, `configs/*.yaml`, the analysis scripts' interfaces, and this document are recorded in `protocol/freeze_manifest.json` and in `tasks/registry.csv` (`frozen_utc`). Any later change is an amendment (`protocol/amendments.csv`) with before/after-results status.
36+
37+## 10. Pilot-informed decisions (recorded at freeze, before any confirmatory run)
38+- Pilot (24 runs, 6 development tasks × 4 arms, 2026-10-30 04:12–05:30 UTC): every run completed within the envelope and every run was scientifically successful in every arm; mean cost 0.38 USD per run, mean wall 110 s. The development tasks therefore show a ceiling; the confirmatory tasks (designed with planted defects, information traps, null truths and false premises before the pilot was run) are used unchanged. We state in advance that a ceiling in the confirmatory stage is a possible, legitimate outcome (no detectable protocol effect at this task difficulty with this model).
39+- Scale decision (rule §6): projected cost of 24 × 4 × 3 = 288 runs ≈ 1.15 × 0.38 × 288 ≈ 125 USD and ≈ 1.1 h at 8 concurrent runs, within the envelope; **the confirmatory design is 24 tasks × 4 arms × 3 repeats (288 runs)**, assignment seed 301, 4 nodes × 2 concurrent sessions.
40+- Harness amendment A-002 (evaluator workspace composition) was applied before the freeze; no oracle was modified after its validation except through that harness fix.
41+- One harness fact learned in the pilot: first-turn input tokens are identical across tasks within an arm (the task statement is read from the workspace, not injected), so arm-level context sizes are A 2,238 / B 3,717 / C 4,111 / D 4,319 tokens.
modified results/analysis_data/criteria.csv +28 −0
@@ -151,3 +151,31 @@ RP0020,criterion,budget_identity,1,Reported budget identity residual p1*x1 + p2*
151 RP0020,criterion,mrs_equals_price_ratio,1,MRS at the optimum within 1e-3 of p1/p2 and price_ratio exact 151 RP0020,criterion,mrs_equals_price_ratio,1,MRS at the optimum within 1e-3 of p1/p2 and price_ratio exact
152 RP0020,criterion,numerical_validation,1,Numerically maximised utility on the budget line within 1e-4 of the analytical indirect utility 152 RP0020,criterion,numerical_validation,1,Numerically maximised utility on the budget line within 1e-4 of the analytical indirect utility
153 RP0020,criterion,assumptions_stated,1,report.md states the assumptions/scope of the closed form (regex: 'assum' and one of interior|positive|concav|monoton|Cobb|log) 153 RP0020,criterion,assumptions_stated,1,report.md states the assumptions/scope of the closed form (regex: 'assum' and one of interior|positive|concav|monoton|Cobb|log)
154+RP0024,criterion,naive_h1,1,Naive RMSE and MAE at h=1 equal the exact reference (rel 1e-6); 35 evaluated pairs (target 2025-10 missing)
155+RP0024,criterion,naive_h3,1,Naive RMSE and MAE at h=3 equal the exact reference (rel 1e-6)
156+RP0024,criterion,counts,1,"n_origins = 36, n_evaluated_h1 = 35, n_evaluated_h3 = 35"
157+RP0024,criterion,forecasts_chronology,1,"forecasts.csv: 144 rows, target = origin + h, naive forecast = last observation <= origin, actual matches data"
158+RP0024,criterion,model_errors_consistent,0,model RMSE/MAE in results.json agree with forecasts.csv (rel 1e-5)
159+RP0024,criterion,rel_improvement_consistent,1,rel_improvement_rmse_h* = 1 - rmse_model/rmse_naive (abs 1e-6)
160+RP0024,criterion,data_issue_reported,1,data_issues mentions the missing 2025-10 observation
161+RP0021,criterion,naive_h1,1,Naive RMSE and MAE at h=1 equal the exact reference (rel 1e-6); 35 evaluated pairs (target 2025-10 missing)
162+RP0021,criterion,naive_h3,1,Naive RMSE and MAE at h=3 equal the exact reference (rel 1e-6)
163+RP0021,criterion,counts,1,"n_origins = 36, n_evaluated_h1 = 35, n_evaluated_h3 = 35"
164+RP0021,criterion,forecasts_chronology,1,"forecasts.csv: 144 rows, target = origin + h, naive forecast = last observation <= origin, actual matches data"
165+RP0021,criterion,model_errors_consistent,1,model RMSE/MAE in results.json agree with forecasts.csv (rel 1e-5)
166+RP0021,criterion,rel_improvement_consistent,1,rel_improvement_rmse_h* = 1 - rmse_model/rmse_naive (abs 1e-6)
167+RP0021,criterion,data_issue_reported,1,data_issues mentions the missing 2025-10 observation
168+RP0022,criterion,naive_h1,1,Naive RMSE and MAE at h=1 equal the exact reference (rel 1e-6); 35 evaluated pairs (target 2025-10 missing)
169+RP0022,criterion,naive_h3,1,Naive RMSE and MAE at h=3 equal the exact reference (rel 1e-6)
170+RP0022,criterion,counts,1,"n_origins = 36, n_evaluated_h1 = 35, n_evaluated_h3 = 35"
171+RP0022,criterion,forecasts_chronology,1,"forecasts.csv: 144 rows, target = origin + h, naive forecast = last observation <= origin, actual matches data"
172+RP0022,criterion,model_errors_consistent,1,model RMSE/MAE in results.json agree with forecasts.csv (rel 1e-5)
173+RP0022,criterion,rel_improvement_consistent,1,rel_improvement_rmse_h* = 1 - rmse_model/rmse_naive (abs 1e-6)
174+RP0022,criterion,data_issue_reported,1,data_issues mentions the missing 2025-10 observation
175+RP0023,criterion,naive_h1,1,Naive RMSE and MAE at h=1 equal the exact reference (rel 1e-6); 35 evaluated pairs (target 2025-10 missing)
176+RP0023,criterion,naive_h3,1,Naive RMSE and MAE at h=3 equal the exact reference (rel 1e-6)
177+RP0023,criterion,counts,1,"n_origins = 36, n_evaluated_h1 = 35, n_evaluated_h3 = 35"
178+RP0023,criterion,forecasts_chronology,1,"forecasts.csv: 144 rows, target = origin + h, naive forecast = last observation <= origin, actual matches data"
179+RP0023,criterion,model_errors_consistent,1,model RMSE/MAE in results.json agree with forecasts.csv (rel 1e-5)
180+RP0023,criterion,rel_improvement_consistent,1,rel_improvement_rmse_h* = 1 - rmse_model/rmse_naive (abs 1e-6)
181+RP0023,criterion,data_issue_reported,1,data_issues mentions the missing 2025-10 observation
modified results/analysis_data/gate_receipts.csv +6 −0
@@ -35,3 +35,9 @@ RP0018,0,final_common_contract_check,2026-10-30T03:52:52.904+00:00,1,0
35 RP0019,1,controller,2026-10-30T03:52:37.169+00:00,1,0 35 RP0019,1,controller,2026-10-30T03:52:37.169+00:00,1,0
36 RP0019,0,final_common_contract_check,2026-10-30T03:53:06.687+00:00,1,0 36 RP0019,0,final_common_contract_check,2026-10-30T03:53:06.687+00:00,1,0
37 RP0020,0,final_common_contract_check,2026-10-30T03:52:57.748+00:00,1,0 37 RP0020,0,final_common_contract_check,2026-10-30T03:52:57.748+00:00,1,0
38+RP0021,1,controller,2026-10-30T04:01:36.134+00:00,1,0
39+RP0021,0,final_common_contract_check,2026-10-30T04:01:38.436+00:00,1,0
40+RP0022,0,final_common_contract_check,2026-10-30T04:01:39.149+00:00,1,0
41+RP0023,1,controller,2026-10-30T04:01:45.291+00:00,1,0
42+RP0023,0,final_common_contract_check,2026-10-30T04:02:30.584+00:00,1,0
43+RP0024,0,final_common_contract_check,2026-10-30T04:00:33.195+00:00,1,0
modified results/analysis_data/reviews.csv +1 −0
@@ -5,3 +5,4 @@ RP0002,5aec3cd4-37d1-4c46-944a-1b2a916d9f46,2026-10-30T03:50:26.272+00:00,accept
5 RP0012,29963ab3-b996-45cd-a99c-907f85d31074,2026-10-30T03:51:08.765+00:00,accept,3 5 RP0012,29963ab3-b996-45cd-a99c-907f85d31074,2026-10-30T03:51:08.765+00:00,accept,3
6 RP0019,4063b667-f0d8-486c-8896-cb3a6a64b8e7,2026-10-30T03:53:05.663+00:00,accept,3 6 RP0019,4063b667-f0d8-486c-8896-cb3a6a64b8e7,2026-10-30T03:53:05.663+00:00,accept,3
7 RP0014,6fcb0c24-be55-47de-b0e0-caf598a26592,2026-10-30T03:54:17.869+00:00,accept,4 7 RP0014,6fcb0c24-be55-47de-b0e0-caf598a26592,2026-10-30T03:54:17.869+00:00,accept,4
8+RP0023,3c0817b8-c7e8-47f1-b003-1b4f1c3b06c0,2026-10-30T04:02:29.436+00:00,accept,3
modified results/analysis_data/runs.csv +4 −0
@@ -23,3 +23,7 @@ RP0017,F5-dev-01,F5,pilot,A,0,pilot-B05,0,M3U96a,evaluated,executor_finished,202
23 RP0018,F5-dev-01,F5,pilot,C,0,pilot-B05,1,M3U96b,evaluated,executor_finished,2026-10-30T03:51:14.878+00:00,2026-10-30T03:52:52.907+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,87a478b389d4bda09a1952a3a13fbb8cebdd597c97f4d2dbaf5e6de3fe9d4eac,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,f8afd6b133bcc262837261cc33b2b43a3a0a44e0f8be3041e1e53ca2352a57d6,1,0.347208,0.347208,0,96.5,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0 23 RP0018,F5-dev-01,F5,pilot,C,0,pilot-B05,1,M3U96b,evaluated,executor_finished,2026-10-30T03:51:14.878+00:00,2026-10-30T03:52:52.907+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,87a478b389d4bda09a1952a3a13fbb8cebdd597c97f4d2dbaf5e6de3fe9d4eac,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,f8afd6b133bcc262837261cc33b2b43a3a0a44e0f8be3041e1e53ca2352a57d6,1,0.347208,0.347208,0,96.5,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0
24 RP0019,F5-dev-01,F5,pilot,D,0,pilot-B05,2,M4M64a,evaluated,executor_finished,2026-10-30T03:50:58.117+00:00,2026-10-30T03:53:06.689+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,ea9b5ce324065b1ca86a06258c11f33a6be157ae307e72b5d424fc18cb5ff286,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,d1a7be8a99fcc552e7334c42050d9b5477ff9d55c5c4022a165cc782cac588b7,2,0.431164,0.326005,0.105159,127.0,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0 24 RP0019,F5-dev-01,F5,pilot,D,0,pilot-B05,2,M4M64a,evaluated,executor_finished,2026-10-30T03:50:58.117+00:00,2026-10-30T03:53:06.689+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,ea9b5ce324065b1ca86a06258c11f33a6be157ae307e72b5d424fc18cb5ff286,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,d1a7be8a99fcc552e7334c42050d9b5477ff9d55c5c4022a165cc782cac588b7,2,0.431164,0.326005,0.105159,127.0,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0
25 RP0020,F5-dev-01,F5,pilot,B,0,pilot-B05,3,M4M64b,evaluated,executor_finished,2026-10-30T03:51:10.342+00:00,2026-10-30T03:52:57.750+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,7b6ca1488fd600a1733707010cc91486b4ed88b09d8cb2d8ef7ad8c622c4ad29,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,bc05a0b4408205bc4b5152ec7e5c1b339ea07cd8c602cb06e684dfad775376ff,1,0.357086,0.357086,0,106.6,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0 25 RP0020,F5-dev-01,F5,pilot,B,0,pilot-B05,3,M4M64b,evaluated,executor_finished,2026-10-30T03:51:10.342+00:00,2026-10-30T03:52:57.750+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),76de75e34e7d7535b63ac807cf0f9207c542f4a1-dirty,7b6ca1488fd600a1733707010cc91486b4ed88b09d8cb2d8ef7ad8c622c4ad29,4f8a3ea40e38cbe69aa42f9fd49cc33ff2d52cb73815bc9ff52e1c341df8b4b3,bc05a0b4408205bc4b5152ec7e5c1b339ea07cd8c602cb06e684dfad775376ff,1,0.357086,0.357086,0,106.6,,1,1,6,6,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F5-dev-01/1.0
26+RP0021,F3-dev-01,F3,pilot,C,0,pilot-B01,0,M3U96a,evaluated,executor_finished,2026-10-30T03:59:03.923+00:00,2026-10-30T04:01:38.438+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),55b92f9bcc679fb8bcbc12d51e0b6519e7f8b07d-dirty,99e9ce54bbbd6985798deeed0f739b6df17736d928cbc831ab3cde917d87de62,47db3ab9ce0210fa4891fd00875f639ab089a3dc9d79145639ccd75e5ac72b99,f8afd6b133bcc262837261cc33b2b43a3a0a44e0f8be3041e1e53ca2352a57d6,1,0.496307,0.496307,0,150.4,,1,1,7,7,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F3-dev-01/1.0
27+RP0022,F3-dev-01,F3,pilot,B,0,pilot-B01,1,M3U96b,evaluated,executor_finished,2026-10-30T03:59:03.834+00:00,2026-10-30T04:01:39.150+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),55b92f9bcc679fb8bcbc12d51e0b6519e7f8b07d-dirty,0540f57f58e0f81cfca87ecf5a143fbe2880b4c27f21def38c0d37a0c4b3a71d,47db3ab9ce0210fa4891fd00875f639ab089a3dc9d79145639ccd75e5ac72b99,bc05a0b4408205bc4b5152ec7e5c1b339ea07cd8c602cb06e684dfad775376ff,1,0.488724,0.488724,0,154.2,,1,1,7,7,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F3-dev-01/1.0
28+RP0023,F3-dev-01,F3,pilot,D,0,pilot-B01,2,M4M64a,evaluated,executor_finished,2026-10-30T03:59:03.738+00:00,2026-10-30T04:02:30.586+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),55b92f9bcc679fb8bcbc12d51e0b6519e7f8b07d-dirty,0dab05f2358f782a51197392d521465a8e29f0c0643a59b75fae4f0f8d82a250,47db3ab9ce0210fa4891fd00875f639ab089a3dc9d79145639ccd75e5ac72b99,d1a7be8a99fcc552e7334c42050d9b5477ff9d55c5c4022a165cc782cac588b7,2,0.684951,0.517441,0.16751,205.0,,1,1,7,7,0,,1,1.0,1.0,1.0,1.0,1.0,1.0,F3-dev-01/1.0
29+RP0024,F3-dev-01,F3,pilot,A,0,pilot-B01,3,M4M64b,evaluated,executor_finished,2026-10-30T03:59:03.738+00:00,2026-10-30T04:00:33.197+00:00,claude-opus-5-5,claude-opus-5-5,2.1.314 (Claude Code),55b92f9bcc679fb8bcbc12d51e0b6519e7f8b07d-dirty,1fee789f298ebe2938f3720baa3a21507f5bddf43d2d44387249b59a31e0763a,47db3ab9ce0210fa4891fd00875f639ab089a3dc9d79145639ccd75e5ac72b99,1d6fe7d64c0b037d7429fd0d88e61f2e8f633cb68c9d020bfb6e61f3b1ce6d17,1,0.268502,0.268502,0,88.4,,1,0,7,6,0,,1,0.8571428571428571,1.0,1.0,1.0,1.0,0.9429,F3-dev-01/1.0
modified results/analysis_data/sessions.csv +5 −0
@@ -29,3 +29,8 @@ b431bbff-0aa5-4176-987c-fd7e5da0029c,RP0018,executor,initial,0,2026-10-30T03:51:
29 3eea0fc0-4407-4152-b5e0-05fb05f85f7a,RP0019,executor,initial,0,2026-10-30T03:50:58.117+00:00,2026-10-30T03:52:36.737+00:00,0,success,0,0.3260052,0.3260052,13,98456,4319,claude-opus-5-5,28,10575,135166,17472 29 3eea0fc0-4407-4152-b5e0-05fb05f85f7a,RP0019,executor,initial,0,2026-10-30T03:50:58.117+00:00,2026-10-30T03:52:36.737+00:00,0,success,0,0.3260052,0.3260052,13,98456,4319,claude-opus-5-5,28,10575,135166,17472
30 4063b667-f0d8-486c-8896-cb3a6a64b8e7,RP0019,reviewer,review,44,2026-10-30T03:52:37.209+00:00,2026-10-30T03:53:05.621+00:00,0,success,0,0.10515859999999999,0.10515859999999999,5,28249,2224,claude-opus-5-5,12,2477,28853,9960 30 4063b667-f0d8-486c-8896-cb3a6a64b8e7,RP0019,reviewer,review,44,2026-10-30T03:52:37.209+00:00,2026-10-30T03:53:05.621+00:00,0,success,0,0.10515859999999999,0.10515859999999999,5,28249,2224,claude-opus-5-5,12,2477,28853,9960
31 b3526ab7-883a-4530-802f-9781a18102c1,RP0020,executor,initial,0,2026-10-30T03:51:10.342+00:00,2026-10-30T03:52:56.893+00:00,0,success,0,0.3570858,0.3570858,11,106337,3717,claude-opus-5-5,22,12624,86939,17426 31 b3526ab7-883a-4530-802f-9781a18102c1,RP0020,executor,initial,0,2026-10-30T03:51:10.342+00:00,2026-10-30T03:52:56.893+00:00,0,success,0,0.3570858,0.3570858,11,106337,3717,claude-opus-5-5,22,12624,86939,17426
32+6c89c7fd-c6e2-42ae-af54-1f1d109ce3fd,RP0021,executor,initial,0,2026-10-30T03:59:03.923+00:00,2026-10-30T04:01:34.320+00:00,0,success,0,0.496307,0.496307,12,150159,4111,claude-opus-5-5,26,16703,167465,25730
33+d18d1f47-f7dc-49d7-b7b0-229ed6d72a98,RP0022,executor,initial,0,2026-10-30T03:59:03.835+00:00,2026-10-30T04:01:38.032+00:00,0,success,0,0.48872400000000005,0.48872400000000005,15,153986,3717,claude-opus-5-5,32,16007,235005,24291
34+fd081072-6055-4453-8558-18099d7c36dc,RP0023,executor,initial,0,2026-10-30T03:59:03.739+00:00,2026-10-30T04:01:44.633+00:00,0,success,0,0.5174408,0.5174408,16,160704,4319,claude-opus-5-5,32,16944,242214,25998
35+3c0817b8-c7e8-47f1-b003-1b4f1c3b06c0,RP0023,reviewer,review,74,2026-10-30T04:01:45.321+00:00,2026-10-30T04:02:29.403+00:00,0,success,0,0.16751,0.16751,6,43901,2224,claude-opus-5-5,14,3972,51770,15532
36+9b17892a-17dd-4863-9fe8-7564f240c508,RP0024,executor,initial,0,2026-10-30T03:59:03.739+00:00,2026-10-30T04:00:32.163+00:00,0,success,0,0.2685016,0.2685016,7,88258,2238,claude-opus-5-5,16,9288,50363,14521
added runs/pilot/RP0021/artifact_manifest.json +21 −0
@@ -0,0 +1,21 @@
1+{
2+ "root": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/methodes/autonomous-research-claude-code/runs/pilot/RP0021/out",
3+ "files": {
4+ "analysis.py": "906890deac2eec16189a1d9aee2b8391e37160966759e27eef6d972144112b6a",
5+ "claims.json": "3b6e90c1c6738bad5fc5f2a8df8aee2741c1f09c71d48cede6fa96245487fdef",
6+ "exclusions.csv": "ab9f0a51637d6b698d8c74bcbad7ed2050487f595b363ffce787aa1aa82aa534",
7+ "forecasts.csv": "8eb3a9e2a6b48dc56a8181784f28a72039e485c12f102a0d094d2a254c6bd72b",
8+ "report.md": "b0ab0e8f5f8976ed4ae28fedc57edb47b29767a7746159c2e0d50f391db8c9d7",
9+ "results.json": "02d04511b503984382a2667b8b515357f1b16f0cb5019aa696f39a0cc3f5d2af",
10+ "state/checks.md": "7d944f96cf90787f0c905a8a480eab64d783c1ddfc01e8270a4cdb9300afe129",
11+ "state/contract.md": "1cf7090b83a1ef21ff7e6a30bab6ab65a56f9974d428d441c4d7ed5cc7d87d83",
12+ "state/data_audit.md": "a913b65e00283c81d9b4d26bb1125a6bb41465a7803d6918c2b7dbaa5fdd8d30",
13+ "state/gate_receipts/20261030T040049.572960Z.json": "16861e10add5b7d34928603383df257ebf1707645d0cb09096c8114da52d0d95",
14+ "state/gate_receipts/20261030T040120.873744Z.json": "fda469d1bd495654676e169963fbf443f962cf4f69d4e8b50d794c8ea8f98e00",
15+ "state/gate_receipts/20261030T040125.359912Z.json": "29db10e76b2b42a2eff45eca42d85a250b172bd04c7c6713283406f114399156",
16+ "state/gate_receipts/20261030T040136.115952Z.json": "c6f343ee88eed8f0d97971e925c1f996815f64efce383bfa1796d596ff67cd2e",
17+ "state/log.md": "22a6ddf85fe3b6907dbd558876dc03e9fd94ce69443d8698854d4b1b645712e8",
18+ "state/phases.json": "cd23acf4c4c470d1658d0b762964ddbb0aa013fdab7799b4eb80857c1cefc76c"
19+ },
20+ "recorded_utc": "2026-10-30T04:01:36.554+00:00"
21+}
\ No newline at end of file
added runs/pilot/RP0021/behaviour_audit.json +5 −0
@@ -0,0 +1,5 @@
1+{
2+ "n_outside_workspace_refs": 0,
3+ "outside_refs": [],
4+ "n_gate_invocations": 3
5+}
\ No newline at end of file
added runs/pilot/RP0021/contract_receipt.json +217 −0
@@ -0,0 +1,217 @@
1+{
2+ "gate_version": "pgar_gate/1.0",
3+ "workspace": "/Users/simon-pierreboucher/wp29/runs/RP0021",
4+ "utc": "2026-10-30T04:01:38.418074+00:00",
5+ "passed": true,
6+ "n_checks": 41,
7+ "n_failed": 0,
8+ "checks": [
9+ {
10+ "id": "file:out/results.json",
11+ "passed": true,
12+ "detail": "present"
13+ },
14+ {
15+ "id": "file:out/report.md",
16+ "passed": true,
17+ "detail": "present"
18+ },
19+ {
20+ "id": "file:out/analysis.py",
21+ "passed": true,
22+ "detail": "present"
23+ },
24+ {
25+ "id": "file:out/claims.json",
26+ "passed": true,
27+ "detail": "present"
28+ },
29+ {
30+ "id": "file:out/forecasts.csv",
31+ "passed": true,
32+ "detail": "present"
33+ },
34+ {
35+ "id": "results_json",
36+ "passed": true,
37+ "detail": "parsed"
38+ },
39+ {
40+ "id": "key:task_id",
41+ "passed": true,
42+ "detail": "ok"
43+ },
44+ {
45+ "id": "key:model_name",
46+ "passed": true,
47+ "detail": "ok"
48+ },
49+ {
50+ "id": "key:model_description",
51+ "passed": true,
52+ "detail": "ok"
53+ },
54+ {
55+ "id": "key:n_origins",
56+ "passed": true,
57+ "detail": "ok"
58+ },
59+ {
60+ "id": "key:first_origin",
61+ "passed": true,
62+ "detail": "ok"
63+ },
64+ {
65+ "id": "key:last_origin",
66+ "passed": true,
67+ "detail": "ok"
68+ },
69+ {
70+ "id": "key:n_evaluated_h1",
71+ "passed": true,
72+ "detail": "ok"
73+ },
74+ {
75+ "id": "key:n_evaluated_h3",
76+ "passed": true,
77+ "detail": "ok"
78+ },
79+ {
80+ "id": "key:rmse_naive_h1",
81+ "passed": true,
82+ "detail": "ok"
83+ },
84+ {
85+ "id": "key:mae_naive_h1",
86+ "passed": true,
87+ "detail": "ok"
88+ },
89+ {
90+ "id": "key:rmse_naive_h3",
91+ "passed": true,
92+ "detail": "ok"
93+ },
94+ {
95+ "id": "key:mae_naive_h3",
96+ "passed": true,
97+ "detail": "ok"
98+ },
99+ {
100+ "id": "key:rmse_model_h1",
101+ "passed": true,
102+ "detail": "ok"
103+ },
104+ {
105+ "id": "key:mae_model_h1",
106+ "passed": true,
107+ "detail": "ok"
108+ },
109+ {
110+ "id": "key:rmse_model_h3",
111+ "passed": true,
112+ "detail": "ok"
113+ },
114+ {
115+ "id": "key:mae_model_h3",
116+ "passed": true,
117+ "detail": "ok"
118+ },
119+ {
120+ "id": "key:rel_improvement_rmse_h1",
121+ "passed": true,
122+ "detail": "ok"
123+ },
124+ {
125+ "id": "key:rel_improvement_rmse_h3",
126+ "passed": true,
127+ "detail": "ok"
128+ },
129+ {
130+ "id": "key:feasibility",
131+ "passed": true,
132+ "detail": "ok"
133+ },
134+ {
135+ "id": "key:feasibility_reason",
136+ "passed": true,
137+ "detail": "ok"
138+ },
139+ {
140+ "id": "key:data_issues",
141+ "passed": true,
142+ "detail": "ok"
143+ },
144+ {
145+ "id": "key:n_input_rows",
146+ "passed": true,
147+ "detail": "ok"
148+ },
149+ {
150+ "id": "key:n_analysed",
151+ "passed": true,
152+ "detail": "ok"
153+ },
154+ {
155+ "id": "key:random_seed",
156+ "passed": true,
157+ "detail": "ok"
158+ },
159+ {
160+ "id": "section:Question",
161+ "passed": true,
162+ "detail": "present"
163+ },
164+ {
165+ "id": "section:Data and exclusions",
166+ "passed": true,
167+ "detail": "present"
168+ },
169+ {
170+ "id": "section:Method",
171+ "passed": true,
172+ "detail": "present"
173+ },
174+ {
175+ "id": "section:Results",
176+ "passed": true,
177+ "detail": "present"
178+ },
179+ {
180+ "id": "section:Uncertainty",
181+ "passed": true,
182+ "detail": "present"
183+ },
184+ {
185+ "id": "section:Limitations and open issues",
186+ "passed": true,
187+ "detail": "present"
188+ },
189+ {
190+ "id": "claims_json",
191+ "passed": true,
192+ "detail": "5 claims, all keys resolve"
193+ },
194+ {
195+ "id": "phase_ledger",
196+ "passed": true,
197+ "detail": "consistent"
198+ },
199+ {
200+ "id": "exclusion_ledger",
201+ "passed": true,
202+ "detail": "2 steps consistent"
203+ },
204+ {
205+ "id": "chronology:first_origin<=last_origin",
206+ "passed": true,
207+ "detail": "2023-07-01 vs 2026-06-01"
208+ },
209+ {
210+ "id": "reproducibility",
211+ "passed": true,
212+ "detail": "regenerated in 1.8s; identical within tolerance"
213+ }
214+ ],
215+ "receipt_path": "/Users/simon-pierreboucher/wp29/runs/RP0021/out/state/gate_receipts/20261030T040138.418350Z.json",
216+ "receipt_sha256": "f87278d1fa9b7f07c5263b632edac9849a1e5abddde687ecbe16a95452c9d199"
217+}
\ No newline at end of file
added runs/pilot/RP0021/ctl/prompt.txt +1 −0
@@ -0,0 +1 @@
1+Begin now. The task is in `TASK.md` in the current directory. Work until the deliverables are complete, then stop.
added runs/pilot/RP0021/ctl/reviewer_prompt.txt +1 −0
@@ -0,0 +1 @@
1+Review the deliverables in this workspace now, as instructed, and write out/review/review.json.
added runs/pilot/RP0021/ctl/reviewer_system.md +10 −0
@@ -0,0 +1,10 @@
1+You are an independent scientific reviewer. In the current working directory, `TASK.md` describes a bounded scientific task and the directory `out/` contains the deliverables submitted by another assistant (you have not seen how they were produced and you must not look for their working transcript). There is no network access. Use `~/wp29/venv/bin/python` if you need to re-run or test code. Do not modify any file under `out/` except that you must create `out/review/review.json`.
2+
3+Review the deliverables against `TASK.md` only. Specifically check:
4+1. Does `out/results.json` contain every required key with legitimately computed values? Re-run `out/analysis.py` in a scratch copy if needed and compare.
5+2. Are the information-availability rules of `TASK.md` respected by the code (for example no use of future information, no use of forbidden inputs)? Read the code to verify; do not trust the report.
6+3. Are the central claims in `out/report.md` and `out/claims.json` supported by the numbers in `out/results.json`, with appropriate uncertainty and caveats? Flag any claim that is stronger than the evidence.
7+4. Are counts, exclusions, units, signs and magnitudes internally consistent? Are baseline comparisons present when the task requires them?
8+5. Are limitations stated honestly? Is anything declared complete that is not?
9+
10+You have no reference answers; do not guess what the "right" number is. Report only issues you can justify from the files. Write `out/review/review.json` as {"verdict": "accept"|"revise", "issues": [{"id": "R1", "severity": "critical"|"major"|"minor", "location": "<file or key>", "issue": "...", "suggested_fix": "..."}], "summary": "..."}. Use "revise" if there is at least one critical or major issue. Be concise and specific. End with a one-paragraph plain-text summary.
added runs/pilot/RP0021/ctl/system_append.md +56 −0
@@ -0,0 +1,56 @@
1+You are a research assistant executing one bounded scientific task inside the current working directory, which is your workspace.
2+
3+Environment facts (identical for every task):
4+- The task statement is the file `TASK.md` in the workspace. It is the complete and only authoritative description of what is required: the scientific question, the inputs under `data/`, the required output files and their exact keys, and the information-availability rules.
5+- There is no network access. Use the Python interpreter `~/wp29/venv/bin/python` (numpy, pandas, scipy, statsmodels, matplotlib are installed). Standard shell tools are available. Do not try to install packages.
6+- Write every deliverable under `./out/`. The required deliverables are listed in `TASK.md`; at minimum they always include `out/results.json`, `out/report.md`, and a replication entry point `out/analysis.py` that regenerates `out/results.json` from `data/` when run with `~/wp29/venv/bin/python out/analysis.py` from the workspace root.
7+- You are working alone and unattended; nobody will answer questions. Make reasonable, documented decisions.
8+
9+Integrity rules (identical for every task):
10+- Never fabricate numbers, data, results, or references. Every number in `out/results.json` and `out/report.md` must be produced by code you ran on the provided inputs.
11+- Respect the information-availability rules in `TASK.md` exactly (for example, never use information that would not have been available at the stated decision time).
12+- If part of the task cannot be completed as specified, say so explicitly in `out/report.md`, explain why, and still write every required file (use `null` for values you could not legitimately produce). An honest statement of a limitation is always preferred to a fabricated or guessed value.
13+- Do not modify anything under `data/`.
14+- When you are done, end with a brief plain-text summary of what you produced and any limitations.
15+
16+Complete the task described in `TASK.md` by following this research protocol. The protocol adds no scientific information to the task; it organizes how you work and what evidence you leave behind.
17+
18+PHASES (execute in order; a phase is eligible only when its dependencies are satisfied):
19+
20+P0 Scope and contract (depends on: nothing)
21+- Read `TASK.md` completely. In `out/state/contract.md`, restate in your own words: the scientific question; the quantity or quantities to be estimated (the estimands) and the assumptions under which they are meaningful; every required output file and key; the information-availability rules; what would make the task impossible or the result invalid.
22+- Write the initial phase ledger `out/state/phases.json`: a JSON object mapping each phase id (P0 … P5) to {"status": "pending"|"running"|"passed"|"failed"|"blocked"|"stale"|"skipped_optional", "checks": [], "notes": ""}. Keep it current as you work. A phase may only be set to "passed" when the checks listed for it in this protocol have actually been executed and their outcome recorded in "checks".
23+
24+P1 Data audit and provenance (depends on: P0)
25+- Inspect every input file: compute and record its SHA-256, row and column counts, schema, units, date ranges, missing values, duplicates and anomalies in `out/state/data_audit.md`.
26+- Keep an exclusion ledger in `out/exclusions.csv` (columns: step, rule, n_before, n_after, n_removed, rationale) for every filtering step you apply, so that n_raw − total removed = n_analysed.
27+- Identify any input that violates the information-availability rules and decide, in writing, how you will handle it.
28+
29+P2 Baseline (depends on: P1)
30+- Before any elaborate method, compute the simplest valid baseline that answers the question (for example a naïve or unconditional estimate), with the same evaluation rule used later. Record it in `out/results.json` under the keys the task requires, or under `baseline_*` keys when the task does not name them.
31+
32+P3 Method and computation (depends on: P2)
33+- Implement the analysis in `out/analysis.py` so that it reads only `data/`, writes `out/results.json` deterministically (fix every random seed you use and record it), and prints nothing that contradicts the written results. Any figure or table must be generated by this script.
34+
35+P4 Verification (depends on: P3)
36+- Re-run `out/analysis.py` from a clean state and confirm it regenerates `out/results.json` with identical values. Record the check in `out/state/phases.json` under P4.
37+- Check internal consistency: identities that must hold (for example counts that must add up, standard errors that must be positive, confidence intervals that must contain their estimates, date orderings that must respect the information rules), units, and magnitudes against the baseline. Record every check and its outcome in `out/state/checks.md`. A failed check must be fixed or honestly reported, never silently ignored.
38+
39+P5 Reporting (depends on: P4)
40+- Write `out/report.md` with the sections: Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues. Every quantitative statement must cite the key of `out/results.json` it comes from.
41+- Write `out/claims.json`: a list of the central claims of your report, each as {"id": "C1", "text": "...", "results_keys": ["..."], "status": "supported"|"unsupported"|"not_evaluated", "caveat": "..."}. Do not state a claim in the report that is not in this ledger.
42+- Set the final statuses in `out/state/phases.json`.
43+
44+STOPPING RULES
45+- Stop and declare the task complete only when every phase P0–P5 is "passed" with recorded checks. If a phase is "blocked" (missing information, impossible requirement, violated availability rule), stop, set the status, explain in `out/report.md` (section Limitations and open issues) and still write every required file with `null` for values you could not legitimately produce.
46+- Do not spend effort on extensions that the task does not require.
47+
48+RECOVERY RULES
49+- Append a short dated entry to `out/state/log.md` after each phase transition and after any important decision, so that work can be resumed by reading `out/state/`.
50+- If you discover that an upstream input or decision changed after a downstream phase passed, mark the downstream phases "stale" and redo them; do not reuse stale results.
51+- If you are interrupted and later resumed, first read `out/state/` and continue from the last consistent state; never re-run expensive steps that already have recorded, unchanged outputs.
52+
53+EXECUTABLE GATES (additional, mandatory)
54+- A read-only gate program is provided at `~/wp29/gates/pgar_gate.py`. Run it from the workspace root as `~/wp29/venv/bin/python ~/wp29/gates/pgar_gate.py .` It checks the output contract of `TASK.md` (required files, keys, types, finite values, report sections, the claims ledger, the phase ledger, the exclusion ledger) and reproducibility (it re-runs `out/analysis.py` in a temporary copy and compares the regenerated `out/results.json` with yours). It never checks scientific correctness and it contains no reference answers.
55+- You must run the gate at the end of P4 and again at the end of P5, and record each run's receipt (the gate writes `out/state/gate_receipts/<timestamp>.json`) in `out/state/phases.json`. You may not declare the task complete while the latest gate receipt reports `"passed": false`. Fix the reported problems and re-run the gate.
56+- The controller will run the same gate after your session ends. If it fails, you will be resumed with the gate output and asked to fix the problems; each such cycle consumes your remaining budget.
added runs/pilot/RP0021/evaluation_receipt.json +158 −0
@@ -0,0 +1,158 @@
1+{
2+ "critical_violations": [],
3+ "evaluated_utc": "2026-10-30T04:01:38.447226+00:00",
4+ "evaluator_version": "F3-dev-01/1.0",
5+ "notes": "Deterministic evaluation; reproducibility component is filled by the harness. Full-sample re-estimation leaks are NOT detectable on this series (documented in REFERENCE.md); the bound catches smoothing/foresight leaks.",
6+ "oracle_sha256": "3c7498c1f70346124c8233d46ab30a2eef957ec2e3437b34674287f93b43016a",
7+ "quality_components": {
8+ "baseline": 1.0,
9+ "correctness": 1.0,
10+ "integrity": 1.0,
11+ "reporting": 1.0,
12+ "reproducibility": 1.0
13+ },
14+ "quality_score": 1.0,
15+ "receipt_sha256": "316c1851986d82a485e328de56757793af04cdcf1cbf580282b8e10864edcab1",
16+ "required_criteria": [
17+ {
18+ "description": "Naive RMSE and MAE at h=1 equal the exact reference (rel 1e-6); 35 evaluated pairs (target 2025-10 missing)",
19+ "expected": {
20+ "mae_naive_h1": 0.09142857142857148,
21+ "rmse_naive_h1": 0.11710800875382403
22+ },
23+ "id": "naive_h1",
24+ "observed": {
25+ "mae_naive_h1": 0.09142857142857148,
26+ "rmse_naive_h1": 0.11710800875382403
27+ },
28+ "passed": true,
29+ "tolerance": {
30+ "mae_naive_h1": {
31+ "rel": 1e-06
32+ },
33+ "rmse_naive_h1": {
34+ "rel": 1e-06
35+ }
36+ }
37+ },
38+ {
39+ "description": "Naive RMSE and MAE at h=3 equal the exact reference (rel 1e-6)",
40+ "expected": {
41+ "mae_naive_h3": 0.12000000000000005,
42+ "rmse_naive_h3": 0.15856499343441843
43+ },
44+ "id": "naive_h3",
45+ "observed": {
46+ "mae_naive_h3": 0.12000000000000005,
47+ "rmse_naive_h3": 0.15856499343441843
48+ },
49+ "passed": true,
50+ "tolerance": {
51+ "mae_naive_h3": {
52+ "rel": 1e-06
53+ },
54+ "rmse_naive_h3": {
55+ "rel": 1e-06
56+ }
57+ }
58+ },
59+ {
60+ "description": "n_origins = 36, n_evaluated_h1 = 35, n_evaluated_h3 = 35",
61+ "expected": {
62+ "n_evaluated_h1": 35,
63+ "n_evaluated_h3": 35,
64+ "n_origins": 36
65+ },
66+ "id": "counts",
67+ "observed": {
68+ "n_evaluated_h1": 35,
69+ "n_evaluated_h3": 35,
70+ "n_origins": 36
71+ },
72+ "passed": true,
73+ "tolerance": {
74+ "n_evaluated_h1": {
75+ "abs": 0
76+ },
77+ "n_evaluated_h3": {
78+ "abs": 0
79+ },
80+ "n_origins": {
81+ "abs": 0
82+ }
83+ }
84+ },
85+ {
86+ "description": "forecasts.csv: 144 rows, target = origin + h, naive forecast = last observation <= origin, actual matches data",
87+ "expected": {
88+ "n_rows": 144
89+ },
90+ "id": "forecasts_chronology",
91+ "observed": {
92+ "actual_mismatches": 0,
93+ "model_rows": 72,
94+ "n_rows": 144,
95+ "naive_rows": 72
96+ },
97+ "passed": true,
98+ "tolerance": null
99+ },
100+ {
101+ "description": "model RMSE/MAE in results.json agree with forecasts.csv (rel 1e-5)",
102+ "expected": "recomputed from forecasts.csv",
103+ "id": "model_errors_consistent",
104+ "observed": {
105+ "h1": {
106+ "mae_from_csv": 0.09561928226000001,
107+ "mae_reported": 0.09561928226000001,
108+ "n": 35,
109+ "rmse_from_csv": 0.1197904141277979,
110+ "rmse_reported": 0.1197904141277979
111+ },
112+ "h3": {
113+ "mae_from_csv": 0.12090394601714284,
114+ "mae_reported": 0.12090394601714284,
115+ "n": 35,
116+ "rmse_from_csv": 0.15937419012351917,
117+ "rmse_reported": 0.15937419012351917
118+ }
119+ },
120+ "passed": true,
121+ "tolerance": {
122+ "rel": 1e-05
123+ }
124+ },
125+ {
126+ "description": "rel_improvement_rmse_h* = 1 - rmse_model/rmse_naive (abs 1e-6)",
127+ "expected": null,
128+ "id": "rel_improvement_consistent",
129+ "observed": {
130+ "h1": {
131+ "expected": -0.022905396501212927,
132+ "reported": -0.022905396501212927
133+ },
134+ "h3": {
135+ "expected": -0.005103249283300526,
136+ "reported": -0.005103249283300526
137+ }
138+ },
139+ "passed": true,
140+ "tolerance": {
141+ "abs": 1e-06
142+ }
143+ },
144+ {
145+ "description": "data_issues mentions the missing 2025-10 observation",
146+ "expected": "mention of 2025-10",
147+ "id": "data_issue_reported",
148+ "observed": "[{\"issue\": \"Missing observation(s) in UNRATE.csv: ['2025-10-01']\", \"handling\": \"Naive at origin 2025-10-01 uses the 2025-09-01 value; the ARIMA likelihood/filter skips the missing month; (origin,horizon) pairs targeting it are kept in forecasts.csv with empty actual and excluded from error statistic",
149+ "passed": true,
150+ "tolerance": null
151+ }
152+ ],
153+ "run_id": "RP0021",
154+ "schema_error": "",
155+ "schema_valid": true,
156+ "scientific_success": true,
157+ "task_id": "F3-dev-01"
158+}
\ No newline at end of file
added runs/pilot/RP0021/out/analysis.py +117 −0
@@ -0,0 +1,117 @@
1+"""F3-dev-01: rolling-origin forecasts of US UNRATE, naive vs ARIMA(2,1,0).
2+Run from workspace root: ~/wp29/venv/bin/python out/analysis.py
3+Reads only data/UNRATE.csv; writes out/results.json, out/forecasts.csv, out/exclusions.csv.
4+"""
5+import json, os, warnings
6+import numpy as np
7+import pandas as pd
8+from scipy import stats
9+from statsmodels.tsa.statespace.sarimax import SARIMAX
10+
11+warnings.filterwarnings("ignore")
12+OUT = "out"
13+os.makedirs(OUT, exist_ok=True)
14+MODEL = "ARIMA210_nc"
15+HORIZONS = [1, 3]
16+
17+raw = pd.read_csv("data/UNRATE.csv", dtype={"observation_date": str})
18+n_raw = len(raw)
19+s = pd.Series(pd.to_numeric(raw["UNRATE"], errors="coerce").values,
20+ index=pd.DatetimeIndex(pd.to_datetime(raw["observation_date"]), freq="MS"))
21+assert s.index.is_monotonic_increasing and not s.index.duplicated().any()
22+missing_dates = [d.strftime("%Y-%m-%d") for d in s.index[s.isna()]]
23+
24+origins = pd.date_range("2023-07-01", "2026-06-01", freq="MS")
25+rows = []
26+for t in origins:
27+ train = s.loc[:t] # only observations dated <= t
28+ # naive: latest available observation dated <= t
29+ naive = float(train.dropna().iloc[-1])
30+ # model: ARIMA(2,1,0), no constant, Gaussian MLE via Kalman filter (handles NaN)
31+ mod = SARIMAX(train, order=(2, 1, 0), trend="n")
32+ res = mod.fit(disp=False, method="lbfgs", maxiter=500)
33+ fc = res.forecast(steps=max(HORIZONS))
34+ for h in HORIZONS:
35+ tgt = t + pd.DateOffset(months=h)
36+ act = s.get(tgt, np.nan)
37+ for name, f in (("naive", naive), (MODEL, float(fc.iloc[h - 1]))):
38+ rows.append({"origin_date": t.strftime("%Y-%m-%d"), "target_date": tgt.strftime("%Y-%m-%d"),
39+ "horizon": h, "forecast": round(f, 10),
40+ "actual": (None if pd.isna(act) else float(act)), "model": name})
41+fc_df = pd.DataFrame(rows)
42+fc_df.to_csv(f"{OUT}/forecasts.csv", index=False, na_rep="")
43+
44+def err(model, h):
45+ d = fc_df[(fc_df.model == model) & (fc_df.horizon == h) & fc_df.actual.notna()].sort_values("origin_date")
46+ return (d.actual - d.forecast).to_numpy(dtype=float)
47+
48+def dm_test(e1, e2, h):
49+ """Diebold-Mariano (squared loss), HAC variance with h-1 rectangular lags, HLN small-sample correction."""
50+ d = e1 ** 2 - e2 ** 2
51+ n = len(d); dbar = d.mean(); dc = d - dbar
52+ g = [np.sum(dc[k:] * dc[:n - k]) / n for k in range(h)]
53+ v = (g[0] + 2 * sum(g[1:])) / n
54+ if v <= 0:
55+ return None, None
56+ dm = dbar / np.sqrt(v)
57+ hln = np.sqrt((n + 1 - 2 * h + h * (h - 1) / n) / n)
58+ stat = dm * hln
59+ p = 2 * stats.t.sf(abs(stat), df=n - 1)
60+ return float(stat), float(p)
61+
62+R = {"task_id": "F3-dev-01", "model_name": MODEL,
63+ "model_description": ("ARIMA(2,1,0) without constant, i.e. AR(2) on first differences of UNRATE, "
64+ "estimated by Gaussian maximum likelihood (Kalman filter, statsmodels SARIMAX, missing months skipped) "
65+ "on an expanding window starting 1948-01-01 and ending at the origin, re-estimated at each origin; "
66+ "multi-step forecasts are iterated."),
67+ "n_origins": int(len(origins)), "first_origin": origins[0].strftime("%Y-%m-%d"),
68+ "last_origin": origins[-1].strftime("%Y-%m-%d")}
69+for h in HORIZONS:
70+ en, em = err("naive", h), err(MODEL, h)
71+ assert len(en) == len(em)
72+ R[f"n_evaluated_h{h}"] = int(len(en))
73+for h in HORIZONS:
74+ en = err("naive", h)
75+ R[f"rmse_naive_h{h}"] = float(np.sqrt(np.mean(en ** 2)))
76+ R[f"mae_naive_h{h}"] = float(np.mean(np.abs(en)))
77+for h in HORIZONS:
78+ em = err(MODEL, h)
79+ R[f"rmse_model_h{h}"] = float(np.sqrt(np.mean(em ** 2)))
80+ R[f"mae_model_h{h}"] = float(np.mean(np.abs(em)))
81+for h in HORIZONS:
82+ R[f"rel_improvement_rmse_h{h}"] = 1.0 - R[f"rmse_model_h{h}"] / R[f"rmse_naive_h{h}"]
83+for h in HORIZONS:
84+ st, p = dm_test(err("naive", h), err(MODEL, h), h)
85+ R[f"dm_stat_h{h}"] = st # >0 means model has lower squared error
86+ R[f"dm_pvalue_h{h}"] = p
87+n_missing_targets = {h: int(((fc_df.model == "naive") & (fc_df.horizon == h) & fc_df.actual.isna()).sum()) for h in HORIZONS}
88+R["feasibility"] = "complete"
89+R["feasibility_reason"] = None
90+R["data_issues"] = [
91+ {"issue": f"Missing observation(s) in UNRATE.csv: {missing_dates}",
92+ "handling": ("Naive at origin 2025-10-01 uses the 2025-09-01 value; the ARIMA likelihood/filter skips the missing month; "
93+ f"(origin,horizon) pairs targeting it are kept in forecasts.csv with empty actual and excluded from error statistics "
94+ f"(h1: {n_missing_targets[1]}, h3: {n_missing_targets[3]}).")},
95+ {"issue": "Data are the final (revised) vintage retrieved 2026-10-30, not real-time vintages.",
96+ "handling": "Evaluation is pseudo-out-of-sample on revised data; no real-time validity is claimed."},
97+ {"issue": "COVID-19 spike (2020-04 = 14.8%) lies inside every training window.",
98+ "handling": "Retained (genuine observation); noted as a limitation for Gaussian MLE of the AR coefficients."},
99+]
100+R["n_input_rows"] = {"UNRATE.csv": int(n_raw)}
101+last_train = s.loc[:origins[-1]]
102+R["n_analysed"] = int(last_train.notna().sum())
103+R["random_seed"] = None
104+with open(f"{OUT}/results.json", "w") as f:
105+ json.dump(R, f, indent=2)
106+
107+# exclusion ledger: raw rows -> observations in the last training window
108+n1 = int(s.notna().sum())
109+excl = pd.DataFrame([
110+ {"step": 1, "rule": "drop rows with missing UNRATE value", "n_before": n_raw, "n_after": n1,
111+ "n_removed": n_raw - n1, "rationale": "empty value field (2025-10-01); no data to use"},
112+ {"step": 2, "rule": "drop observations dated after last origin (2026-06-01) from training",
113+ "n_before": n1, "n_after": R["n_analysed"], "n_removed": n1 - R["n_analysed"],
114+ "rationale": "information rule: training uses only obs <= origin; these serve only as evaluation actuals"},
115+])
116+excl.to_csv(f"{OUT}/exclusions.csv", index=False)
117+print(json.dumps({k: R[k] for k in R if k.startswith(("rmse", "mae", "rel", "n_eval", "dm"))}, indent=1))
added runs/pilot/RP0021/out/claims.json +7 −0
@@ -0,0 +1,7 @@
1+[
2+ {"id":"C1","text":"At h=1 the ARIMA(2,1,0) model does not beat the naive forecast: its RMSE and MAE are both slightly higher (relative RMSE improvement about -2.3%).","results_keys":["rmse_naive_h1","rmse_model_h1","mae_naive_h1","mae_model_h1","rel_improvement_rmse_h1"],"status":"supported","caveat":"35 evaluated pairs, final-vintage data, single 3-year window."},
3+ {"id":"C2","text":"At h=3 the model does not beat the naive forecast: its RMSE and MAE are marginally higher (relative RMSE improvement about -0.5%).","results_keys":["rmse_naive_h3","rmse_model_h3","mae_naive_h3","mae_model_h3","rel_improvement_rmse_h3"],"status":"supported","caveat":"Overlapping 3-month errors; small sample."},
4+ {"id":"C3","text":"The squared-loss difference is statistically detectable at h=1 (DM p about 0.02) but not at h=3 (p about 0.34).","results_keys":["dm_stat_h1","dm_pvalue_h1","dm_stat_h3","dm_pvalue_h3"],"status":"supported","caveat":"DM test ignores that the model nests the random walk; small sample; treat as approximate."},
5+ {"id":"C4","text":"Each horizon has 35 of 36 pairs evaluated because the 2025-10-01 observation is missing.","results_keys":["n_evaluated_h1","n_evaluated_h3","n_origins","data_issues"],"status":"supported","caveat":""},
6+ {"id":"C5","text":"The last training window contains 941 observations after 1 missing value and 3 post-origin months are excluded from the 945 input rows.","results_keys":["n_analysed","n_input_rows"],"status":"supported","caveat":""}
7+]
added runs/pilot/RP0021/out/exclusions.csv +3 −0
@@ -0,0 +1,3 @@
1+step,rule,n_before,n_after,n_removed,rationale
2+1,drop rows with missing UNRATE value,945,944,1,empty value field (2025-10-01); no data to use
3+2,drop observations dated after last origin (2026-06-01) from training,944,941,3,information rule: training uses only obs <= origin; these serve only as evaluation actuals
added runs/pilot/RP0021/out/forecasts.csv +145 −0
@@ -0,0 +1,145 @@
1+origin_date,target_date,horizon,forecast,actual,model
2+2023-07-01,2023-08-01,1,3.5,3.7,naive
3+2023-07-01,2023-08-01,1,3.4961602811,3.7,ARIMA210_nc
4+2023-07-01,2023-10-01,3,3.5,3.9,naive
5+2023-07-01,2023-10-01,3,3.5006303857,3.9,ARIMA210_nc
6+2023-08-01,2023-09-01,1,3.7,3.7,naive
7+2023-08-01,2023-09-01,1,3.7119469479,3.7,ARIMA210_nc
8+2023-08-01,2023-11-01,3,3.7,3.7,naive
9+2023-08-01,2023-11-01,3,3.7029941251,3.7,ARIMA210_nc
10+2023-09-01,2023-10-01,1,3.7,3.9,naive
11+2023-09-01,2023-10-01,1,3.6914155243,3.9,ARIMA210_nc
12+2023-09-01,2023-12-01,3,3.7,3.8,naive
13+2023-09-01,2023-12-01,3,3.6914430337,3.8,ARIMA210_nc
14+2023-10-01,2023-11-01,1,3.9,3.7,naive
15+2023-10-01,2023-11-01,1,3.9076489955,3.7,ARIMA210_nc
16+2023-10-01,2024-01-01,3,3.9,3.7,naive
17+2023-10-01,2024-01-01,3,3.8987677706,3.7,ARIMA210_nc
18+2023-11-01,2023-12-01,1,3.7,3.8,naive
19+2023-11-01,2023-12-01,1,3.6838725389,3.8,ARIMA210_nc
20+2023-11-01,2024-02-01,3,3.7,3.9,naive
21+2023-11-01,2024-02-01,3,3.6927790036,3.9,ARIMA210_nc
22+2023-12-01,2024-01-01,1,3.8,3.7,naive
23+2023-12-01,2024-01-01,1,3.8122838358,3.7,ARIMA210_nc
24+2023-12-01,2024-03-01,3,3.8,3.9,naive
25+2023-12-01,2024-03-01,3,3.8078330982,3.9,ARIMA210_nc
26+2024-01-01,2024-02-01,1,3.7,3.9,naive
27+2024-01-01,2024-02-01,1,3.6919881521,3.9,ARIMA210_nc
28+2024-01-01,2024-04-01,3,3.7,3.9,naive
29+2024-01-01,2024-04-01,3,3.6964094337,3.9,ARIMA210_nc
30+2024-02-01,2024-03-01,1,3.9,3.9,naive
31+2024-02-01,2024-03-01,1,3.9117473793,3.9,ARIMA210_nc
32+2024-02-01,2024-05-01,3,3.9,3.9,naive
33+2024-02-01,2024-05-01,3,3.9029473415,3.9,ARIMA210_nc
34+2024-03-01,2024-04-01,1,3.9,3.9,naive
35+2024-03-01,2024-04-01,1,3.8915566548,3.9,ARIMA210_nc
36+2024-03-01,2024-06-01,3,3.9,4.1,naive
37+2024-03-01,2024-06-01,3,3.89158361,4.1,ARIMA210_nc
38+2024-04-01,2024-05-01,1,3.9,3.9,naive
39+2024-04-01,2024-05-01,1,3.9,3.9,ARIMA210_nc
40+2024-04-01,2024-07-01,3,3.9,4.2,naive
41+2024-04-01,2024-07-01,3,3.9,4.2,ARIMA210_nc
42+2024-05-01,2024-06-01,1,3.9,4.1,naive
43+2024-05-01,2024-06-01,1,3.9,4.1,ARIMA210_nc
44+2024-05-01,2024-08-01,3,3.9,4.2,naive
45+2024-05-01,2024-08-01,3,3.9,4.2,ARIMA210_nc
46+2024-06-01,2024-07-01,1,4.1,4.2,naive
47+2024-06-01,2024-07-01,1,4.1075218973,4.2,ARIMA210_nc
48+2024-06-01,2024-09-01,3,4.1,4.1,naive
49+2024-06-01,2024-09-01,3,4.098739195,4.1,ARIMA210_nc
50+2024-07-01,2024-08-01,1,4.2,4.2,naive
51+2024-07-01,2024-08-01,1,4.195330379,4.2,ARIMA210_nc
52+2024-07-01,2024-10-01,3,4.2,4.1,naive
53+2024-07-01,2024-10-01,3,4.19096437,4.1,ARIMA210_nc
54+2024-08-01,2024-09-01,1,4.2,4.1,naive
55+2024-08-01,2024-09-01,1,4.1957794965,4.1,ARIMA210_nc
56+2024-08-01,2024-11-01,3,4.2,4.2,naive
57+2024-08-01,2024-11-01,3,4.1957923848,4.2,ARIMA210_nc
58+2024-09-01,2024-10-01,1,4.1,4.1,naive
59+2024-09-01,2024-10-01,1,4.0962269917,4.1,ARIMA210_nc
60+2024-09-01,2024-12-01,3,4.1,4.1,naive
61+2024-09-01,2024-12-01,3,4.1006246916,4.1,ARIMA210_nc
62+2024-10-01,2024-11-01,1,4.1,4.2,naive
63+2024-10-01,2024-11-01,1,4.1042264961,4.2,ARIMA210_nc
64+2024-10-01,2025-01-01,3,4.1,4.0,naive
65+2024-10-01,2025-01-01,3,4.1042133356,4.0,ARIMA210_nc
66+2024-11-01,2024-12-01,1,4.2,4.1,naive
67+2024-11-01,2024-12-01,1,4.2037729953,4.1,ARIMA210_nc
68+2024-11-01,2025-02-01,3,4.2,4.2,naive
69+2024-11-01,2025-02-01,3,4.1993688504,4.2,ARIMA210_nc
70+2024-12-01,2025-01-01,1,4.1,4.0,naive
71+2024-12-01,2025-01-01,1,4.0920012234,4.0,ARIMA210_nc
72+2024-12-01,2025-03-01,3,4.1,4.2,naive
73+2024-12-01,2025-03-01,3,4.0964188031,4.2,ARIMA210_nc
74+2025-01-01,2025-02-01,1,4.0,4.2,naive
75+2025-01-01,2025-02-01,1,4.0004657532,4.2,ARIMA210_nc
76+2025-01-01,2025-04-01,3,4.0,4.2,naive
77+2025-01-01,2025-04-01,3,4.0048623586,4.2,ARIMA210_nc
78+2025-02-01,2025-03-01,1,4.2,4.2,naive
79+2025-02-01,2025-03-01,1,4.2117711437,4.2,ARIMA210_nc
80+2025-02-01,2025-05-01,3,4.2,4.3,naive
81+2025-02-01,2025-05-01,3,4.2029099859,4.3,ARIMA210_nc
82+2025-03-01,2025-04-01,1,4.2,4.2,naive
83+2025-03-01,2025-04-01,1,4.1915010591,4.2,ARIMA210_nc
84+2025-03-01,2025-06-01,3,4.2,4.1,naive
85+2025-03-01,2025-06-01,3,4.1915307387,4.1,ARIMA210_nc
86+2025-04-01,2025-05-01,1,4.2,4.3,naive
87+2025-04-01,2025-05-01,1,4.2,4.3,ARIMA210_nc
88+2025-04-01,2025-07-01,3,4.2,4.3,naive
89+2025-04-01,2025-07-01,3,4.2,4.3,ARIMA210_nc
90+2025-05-01,2025-06-01,1,4.3,4.1,naive
91+2025-05-01,2025-06-01,1,4.3037589263,4.1,ARIMA210_nc
92+2025-05-01,2025-08-01,3,4.3,4.3,naive
93+2025-05-01,2025-08-01,3,4.2993377145,4.3,ARIMA210_nc
94+2025-06-01,2025-07-01,1,4.1,4.3,naive
95+2025-06-01,2025-07-01,1,4.0882596257,4.3,ARIMA210_nc
96+2025-06-01,2025-09-01,3,4.1,4.4,naive
97+2025-06-01,2025-09-01,3,4.0971162954,4.4,ARIMA210_nc
98+2025-07-01,2025-08-01,1,4.3,4.3,naive
99+2025-07-01,2025-08-01,1,4.3159062103,4.3,ARIMA210_nc
100+2025-07-01,2025-10-01,3,4.3,,naive
101+2025-07-01,2025-10-01,3,4.3070637746,,ARIMA210_nc
102+2025-08-01,2025-09-01,1,4.3,4.4,naive
103+2025-08-01,2025-09-01,1,4.2915364268,4.4,ARIMA210_nc
104+2025-08-01,2025-11-01,3,4.3,4.5,naive
105+2025-08-01,2025-11-01,3,4.2915682823,4.5,ARIMA210_nc
106+2025-09-01,2025-10-01,1,4.4,,naive
107+2025-09-01,2025-10-01,1,4.4037167521,,ARIMA210_nc
108+2025-09-01,2025-12-01,3,4.4,4.4,naive
109+2025-09-01,2025-12-01,3,4.3993282052,4.4,ARIMA210_nc
110+2025-10-01,2025-11-01,1,4.4,4.5,naive
111+2025-10-01,2025-11-01,1,4.3996366342,4.5,ARIMA210_nc
112+2025-10-01,2026-01-01,3,4.4,4.3,naive
113+2025-10-01,2026-01-01,3,4.3994888505,4.3,ARIMA210_nc
114+2025-11-01,2025-12-01,1,4.5,4.4,naive
115+2025-11-01,2025-12-01,1,4.4994218311,4.4,ARIMA210_nc
116+2025-11-01,2026-02-01,3,4.5,4.4,naive
117+2025-11-01,2026-02-01,3,4.4974048598,4.4,ARIMA210_nc
118+2025-12-01,2026-01-01,1,4.4,4.3,naive
119+2025-12-01,2026-01-01,1,4.3945466104,4.3,ARIMA210_nc
120+2025-12-01,2026-03-01,3,4.4,4.3,naive
121+2025-12-01,2026-03-01,3,4.3989442657,4.3,ARIMA210_nc
122+2026-01-01,2026-02-01,1,4.3,4.4,naive
123+2026-01-01,2026-02-01,1,4.3005540905,4.4,ARIMA210_nc
124+2026-01-01,2026-04-01,3,4.3,4.3,naive
125+2026-01-01,2026-04-01,3,4.3049312774,4.3,ARIMA210_nc
126+2026-02-01,2026-03-01,1,4.4,4.3,naive
127+2026-02-01,2026-03-01,1,4.4078948456,4.3,ARIMA210_nc
128+2026-02-01,2026-05-01,3,4.4,4.3,naive
129+2026-02-01,2026-05-01,3,4.4034752841,4.3,ARIMA210_nc
130+2026-03-01,2026-04-01,1,4.3,4.3,naive
131+2026-03-01,2026-04-01,1,4.2921190774,4.3,ARIMA210_nc
132+2026-03-01,2026-06-01,3,4.3,4.2,naive
133+2026-03-01,2026-06-01,3,4.2965311075,4.2,ARIMA210_nc
134+2026-04-01,2026-05-01,1,4.3,4.3,naive
135+2026-04-01,2026-05-01,1,4.3042229902,4.3,ARIMA210_nc
136+2026-04-01,2026-07-01,3,4.3,4.1,naive
137+2026-04-01,2026-07-01,3,4.3042047335,4.1,ARIMA210_nc
138+2026-05-01,2026-06-01,1,4.3,4.2,naive
139+2026-05-01,2026-06-01,1,4.3,4.2,ARIMA210_nc
140+2026-05-01,2026-08-01,3,4.3,4.1,naive
141+2026-05-01,2026-08-01,3,4.3,4.1,ARIMA210_nc
142+2026-06-01,2026-07-01,1,4.2,4.1,naive
143+2026-06-01,2026-07-01,1,4.19634303,4.1,ARIMA210_nc
144+2026-06-01,2026-09-01,3,4.2,4.2,naive
145+2026-06-01,2026-09-01,3,4.2007359991,4.2,ARIMA210_nc
added runs/pilot/RP0021/out/report.md +34 −0
@@ -0,0 +1,34 @@
1+# F3-dev-01 — Rolling-origin forecasts of the US unemployment rate
2+
3+## Question
4+Over 36 expanding-window forecast origins (`first_origin` = 2023-07-01 to `last_origin` = 2026-06-01, `n_origins` = 36), does a simple autoregressive-type model forecast the US civilian unemployment rate (FRED UNRATE, SA, %) better than the naive no-change forecast at horizons of 1 and 3 months? We measure this with out-of-sample RMSE and MAE, and with the relative improvement 1 − RMSE_model/RMSE_naive.
5+
6+## Data and exclusions
7+- Input: `data/UNRATE.csv`, final vintage retrieved 2026-10-30. It has 945 data rows covering 1948-01 to 2026-09 (`n_input_rows`), and its SHA-256 matches the README. There are no duplicate or missing dates.
8+- One missing value is listed in `data_issues`: 2025-10-01 has an empty field. Handling: at origin 2025-10-01 the naive forecast uses the 2025-09 value. The model's Kalman-filter likelihood skips the missing month. The two pairs whose target is 2025-10-01 (origin 2025-09 at h=1, origin 2025-07 at h=3) stay in `out/forecasts.csv` with an empty `actual` and are left out of the error statistics. This leaves 35 evaluated pairs per horizon (`n_evaluated_h1`, `n_evaluated_h3`).
9+- Exclusion ledger (`out/exclusions.csv`): 945 rows → drop 1 missing value → 944 → drop 3 observations dated after the last origin (used only as actuals) → 941 = `n_analysed`. This is the number of observations in the last training window.
10+
11+## Method
12+- **Naive:** every horizon is forecast with the latest non-missing observation dated ≤ origin.
13+- **Model (`model_name` = ARIMA210_nc):** see `model_description`. The model is AR(2) on first differences with no constant. It is estimated by Gaussian maximum likelihood (statsmodels SARIMAX, Kalman filter) on an expanding window from 1948-01 up to and including the origin, and re-estimated at each origin. Forecasts for 3 months are built by iterating the 1-month forecast. I fixed the specification before looking at any evaluation errors and did not try other models. There is no randomness, so `random_seed` is null.
14+- **Information rule:** at each origin the series is cut at the origin date (`s.loc[:t]`) before any estimation or transformation.
15+- **Evaluation:** errors are actual − forecast on the final-vintage data. A Diebold–Mariano test with squared-error loss, HAC variance using h−1 lags and the Harvey–Leybourne–Newbold small-sample correction is reported as a supplement.
16+
17+## Results
18+| horizon | RMSE naive | MAE naive | RMSE model | MAE model | 1 − RMSE_model/RMSE_naive |
19+|---|---|---|---|---|---|
20+| h=1 | 0.1171 (`rmse_naive_h1`) | 0.0914 (`mae_naive_h1`) | 0.1198 (`rmse_model_h1`) | 0.0956 (`mae_model_h1`) | −0.0229 (`rel_improvement_rmse_h1`) |
21+| h=3 | 0.1586 (`rmse_naive_h3`) | 0.1200 (`mae_naive_h3`) | 0.1594 (`rmse_model_h3`) | 0.1209 (`mae_model_h3`) | −0.0051 (`rel_improvement_rmse_h3`) |
22+
23+All errors are in percentage points. The AR model does **not** beat the naive forecast at either horizon. Its RMSE is about 2.3% higher at h=1 and about 0.5% higher at h=3, and its MAE is also slightly higher at both horizons. The model's forecasts stay very close to the no-change forecast, always within 0.2 pp (see `out/state/checks.md`).
24+
25+## Uncertainty
26+- **h=1:** DM statistic = −2.46 (`dm_stat_h1`), p = 0.019 (`dm_pvalue_h1`). The model's squared-error loss is higher than the naive loss by a statistically detectable amount, but the difference in RMSE is small (`rel_improvement_rmse_h1`).
27+- **h=3:** DM statistic = −0.96 (`dm_stat_h3`), p = 0.34 (`dm_pvalue_h3`). There is no detectable difference.
28+- With only 35 evaluated pairs, overlapping errors at h=3, and a single 3-year window, these results are fragile. The DM test also does not account for the fact that the model nests the random walk. That issue matters most at h=1, where the DM p-value should be treated as approximate.
29+
30+## Limitations and open issues
31+- Final-vintage data are used, so this is a pseudo-out-of-sample exercise. No real-time validity is claimed, because revisions to seasonal factors were not known at the origins.
32+- Every training window contains the 2020 COVID spike (e.g. 14.8% in 2020-04). This may distort the Gaussian-MLE AR coefficients. I kept the full sample on purpose to avoid choosing the sample after seeing the evaluation results. A model fitted on a shorter or robust sample might do differently, but that was not evaluated.
33+- Only one model was assessed, so the results say nothing about AR-type models in general or about other specifications.
34+- The 2025-10 missing value lowers the evaluated count to 35 per horizon. The task is otherwise complete (`feasibility` = "complete").
added runs/pilot/RP0021/out/results.json +45 −0
@@ -0,0 +1,45 @@
1+{
2+ "task_id": "F3-dev-01",
3+ "model_name": "ARIMA210_nc",
4+ "model_description": "ARIMA(2,1,0) without constant, i.e. AR(2) on first differences of UNRATE, estimated by Gaussian maximum likelihood (Kalman filter, statsmodels SARIMAX, missing months skipped) on an expanding window starting 1948-01-01 and ending at the origin, re-estimated at each origin; multi-step forecasts are iterated.",
5+ "n_origins": 36,
6+ "first_origin": "2023-07-01",
7+ "last_origin": "2026-06-01",
8+ "n_evaluated_h1": 35,
9+ "n_evaluated_h3": 35,
10+ "rmse_naive_h1": 0.11710800875382403,
11+ "mae_naive_h1": 0.09142857142857148,
12+ "rmse_naive_h3": 0.15856499343441843,
13+ "mae_naive_h3": 0.12000000000000005,
14+ "rmse_model_h1": 0.1197904141277979,
15+ "mae_model_h1": 0.09561928226000001,
16+ "rmse_model_h3": 0.15937419012351917,
17+ "mae_model_h3": 0.12090394601714284,
18+ "rel_improvement_rmse_h1": -0.022905396501212927,
19+ "rel_improvement_rmse_h3": -0.005103249283300526,
20+ "dm_stat_h1": -2.4559640134162657,
21+ "dm_pvalue_h1": 0.019315858251851482,
22+ "dm_stat_h3": -0.9643792812517766,
23+ "dm_pvalue_h3": 0.3416653293625982,
24+ "feasibility": "complete",
25+ "feasibility_reason": null,
26+ "data_issues": [
27+ {
28+ "issue": "Missing observation(s) in UNRATE.csv: ['2025-10-01']",
29+ "handling": "Naive at origin 2025-10-01 uses the 2025-09-01 value; the ARIMA likelihood/filter skips the missing month; (origin,horizon) pairs targeting it are kept in forecasts.csv with empty actual and excluded from error statistics (h1: 1, h3: 1)."
30+ },
31+ {
32+ "issue": "Data are the final (revised) vintage retrieved 2026-10-30, not real-time vintages.",
33+ "handling": "Evaluation is pseudo-out-of-sample on revised data; no real-time validity is claimed."
34+ },
35+ {
36+ "issue": "COVID-19 spike (2020-04 = 14.8%) lies inside every training window.",
37+ "handling": "Retained (genuine observation); noted as a limitation for Gaussian MLE of the AR coefficients."
38+ }
39+ ],
40+ "n_input_rows": {
41+ "UNRATE.csv": 945
42+ },
43+ "n_analysed": 941,
44+ "random_seed": null
45+}
\ No newline at end of file
added runs/pilot/RP0021/out/state/checks.md +14 −0
@@ -0,0 +1,14 @@
1+# Verification checks (P4)
2+| check | outcome |
3+|---|---|
4+| Clean re-run (deleted results/forecasts/exclusions, re-ran analysis.py): byte-identical results.json and forecasts.csv | PASS |
5+| forecasts.csv has 144 rows, 36 unique origins | PASS |
6+| target_date = origin_date + h months for every row | PASS |
7+| n_evaluated_h1 = n_evaluated_h3 = 35; 36 − 35 = 1 missing target per horizon (2025-10-01) | PASS |
8+| RMSE ≥ MAE > 0 for all model/horizon pairs | PASS |
9+| rel_improvement = 1 − rmse_model/rmse_naive (to 1e-12) | PASS |
10+| first_origin < last_origin | PASS |
11+| Exclusion ledger: 945 − 1 − 3 = 941 = n_analysed | PASS |
12+| Naive forecast equals latest non-missing obs ≤ origin (incl. origin 2025-10-01 → 2025-09 value 4.4) | PASS |
13+| Magnitude: model forecasts within 0.2 pp of naive; model RMSE within 3% of naive RMSE | PASS |
14+| Information rule: training series truncated with s.loc[:t] at each origin (code inspection); model spec fixed a priori, not tuned on evaluation errors | PASS |
added runs/pilot/RP0021/out/state/contract.md +6 −0
@@ -0,0 +1,6 @@
1+# Contract (P0) — F3-dev-01
2+**Question:** Over 36 rolling expanding-window origins (2023-07-01 … 2026-06-01), does a simple autoregressive-type model forecast US UNRATE better than the naive no-change forecast at h=1 and h=3 months?
3+**Estimands:** out-of-sample RMSE and MAE (percentage points) of naive and model per horizon over origins with non-missing actuals; relative improvement 1 − RMSE_model/RMSE_naive per horizon. Meaningful assuming the final-vintage series is the evaluation truth (not real-time) and that the 36-origin sample is a fair (small) sample of recent forecasting conditions.
4+**Outputs:** out/forecasts.csv (144 rows: origin_date,target_date,horizon,forecast,actual,model), out/results.json (task_id, model_name, model_description, n_origins, first_origin, last_origin, n_evaluated_h1/h3, rmse/mae naive & model h1/h3, rel_improvement_rmse_h1/h3, feasibility, feasibility_reason, data_issues, n_input_rows, n_analysed, random_seed), out/report.md (Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues), out/analysis.py, out/claims.json, out/exclusions.csv.
5+**Information rules:** forecast at origin t uses only obs dated ≤ t (estimation, model choice, transforms). Naive = latest available obs ≤ t. Missing actual → ledger row with empty actual, excluded from stats. No real-time claims.
6+**Invalidity conditions:** use of data after t in fitting; model chosen by looking at evaluation-period errors; fewer origins than specified; fabricated values.
added runs/pilot/RP0021/out/state/data_audit.md +7 −0
@@ -0,0 +1,7 @@
1+# Data audit (P1)
2+File: data/UNRATE.csv — SHA-256 621ff6fadf50910cb7a21ee20ba610c247cb043b29b4e05c4c1a72fadca2a06c (matches README and bundle.json).
3+- 945 data rows + header; 2 columns: observation_date (YYYY-MM-01), UNRATE (percent, SA, float).
4+- Date range 1948-01-01 … 2026-09-01; monthly grid complete (no missing dates), no duplicate dates.
5+- Missing values: 1 (2025-10-01, empty field). Non-missing: 944. Range 2.5–14.8; max is 2020-04 (COVID spike, genuine, retained).
6+- Implications of the missing month: origin 2025-10-01 has no own observation (naive uses 2025-09-01 value); target 2025-10-01 is missing for (origin 2025-09-01, h=1) and (origin 2025-07-01, h=3) → those pairs kept in ledger with empty actual, excluded from stats. Model handles the gap via the Kalman filter (state-space likelihood skips the missing observation).
7+- Availability: the file is the final vintage (retrieved 2026-10-30), so values may include revisions unavailable at origin dates. This cannot be avoided with the given input; handled by stating results are pseudo-out-of-sample on revised data, no real-time claim. Data-dating rule (≤ t) is enforced in code by truncating the series at each origin.
added runs/pilot/RP0021/out/state/gate_receipts/20261030T040049.572960Z.json +215 −0
@@ -0,0 +1,215 @@
1+{
2+ "gate_version": "pgar_gate/1.0",
3+ "workspace": "/Users/simon-pierreboucher/wp29/runs/RP0021",
4+ "utc": "2026-10-30T04:00:49.572652+00:00",
5+ "passed": false,
6+ "n_checks": 41,
7+ "n_failed": 10,
8+ "checks": [
9+ {
10+ "id": "file:out/results.json",
11+ "passed": true,
12+ "detail": "present"
13+ },
14+ {
15+ "id": "file:out/report.md",
16+ "passed": false,
17+ "detail": "missing or empty"
18+ },
19+ {
20+ "id": "file:out/analysis.py",
21+ "passed": true,
22+ "detail": "present"
23+ },
24+ {
25+ "id": "file:out/claims.json",
26+ "passed": false,
27+ "detail": "missing or empty"
28+ },
29+ {
30+ "id": "file:out/forecasts.csv",
31+ "passed": true,
32+ "detail": "present"
33+ },
34+ {
35+ "id": "results_json",
36+ "passed": true,
37+ "detail": "parsed"
38+ },
39+ {
40+ "id": "key:task_id",
41+ "passed": true,
42+ "detail": "ok"
43+ },
44+ {
45+ "id": "key:model_name",
46+ "passed": true,
47+ "detail": "ok"
48+ },
49+ {
50+ "id": "key:model_description",
51+ "passed": true,
52+ "detail": "ok"
53+ },
54+ {
55+ "id": "key:n_origins",
56+ "passed": true,
57+ "detail": "ok"
58+ },
59+ {
60+ "id": "key:first_origin",
61+ "passed": true,
62+ "detail": "ok"
63+ },
64+ {
65+ "id": "key:last_origin",
66+ "passed": true,
67+ "detail": "ok"
68+ },
69+ {
70+ "id": "key:n_evaluated_h1",
71+ "passed": true,
72+ "detail": "ok"
73+ },
74+ {
75+ "id": "key:n_evaluated_h3",
76+ "passed": true,
77+ "detail": "ok"
78+ },
79+ {
80+ "id": "key:rmse_naive_h1",
81+ "passed": true,
82+ "detail": "ok"
83+ },
84+ {
85+ "id": "key:mae_naive_h1",
86+ "passed": true,
87+ "detail": "ok"
88+ },
89+ {
90+ "id": "key:rmse_naive_h3",
91+ "passed": true,
92+ "detail": "ok"
93+ },
94+ {
95+ "id": "key:mae_naive_h3",
96+ "passed": true,
97+ "detail": "ok"
98+ },
99+ {
100+ "id": "key:rmse_model_h1",
101+ "passed": true,
102+ "detail": "ok"
103+ },
104+ {
105+ "id": "key:mae_model_h1",
106+ "passed": true,
107+ "detail": "ok"
108+ },
109+ {
110+ "id": "key:rmse_model_h3",
111+ "passed": true,
112+ "detail": "ok"
113+ },
114+ {
115+ "id": "key:mae_model_h3",
116+ "passed": true,
117+ "detail": "ok"
118+ },
119+ {
120+ "id": "key:rel_improvement_rmse_h1",
121+ "passed": true,
122+ "detail": "ok"
123+ },
124+ {
125+ "id": "key:rel_improvement_rmse_h3",
126+ "passed": true,
127+ "detail": "ok"
128+ },
129+ {
130+ "id": "key:feasibility",
131+ "passed": true,
132+ "detail": "ok"
133+ },
134+ {
135+ "id": "key:feasibility_reason",
136+ "passed": true,
137+ "detail": "ok"
138+ },
139+ {
140+ "id": "key:data_issues",
141+ "passed": true,
142+ "detail": "ok"
143+ },
144+ {
145+ "id": "key:n_input_rows",
146+ "passed": true,
147+ "detail": "ok"
148+ },
149+ {
150+ "id": "key:n_analysed",
151+ "passed": true,
152+ "detail": "ok"
153+ },
154+ {
155+ "id": "key:random_seed",
156+ "passed": true,
157+ "detail": "ok"
158+ },
159+ {
160+ "id": "section:Question",
161+ "passed": false,
162+ "detail": "heading not found"
163+ },
164+ {
165+ "id": "section:Data and exclusions",
166+ "passed": false,
167+ "detail": "heading not found"
168+ },
169+ {
170+ "id": "section:Method",
171+ "passed": false,
172+ "detail": "heading not found"
173+ },
174+ {
175+ "id": "section:Results",
176+ "passed": false,
177+ "detail": "heading not found"
178+ },
179+ {
180+ "id": "section:Uncertainty",
181+ "passed": false,
182+ "detail": "heading not found"
183+ },
184+ {
185+ "id": "section:Limitations and open issues",
186+ "passed": false,
187+ "detail": "heading not found"
188+ },
189+ {
190+ "id": "claims_json",
191+ "passed": false,
192+ "detail": "out/claims.json missing"
193+ },
194+ {
195+ "id": "phase_ledger",
196+ "passed": false,
197+ "detail": "phases not terminal: ['P1', 'P2', 'P3', 'P4', 'P5']"
198+ },
199+ {
200+ "id": "exclusion_ledger",
201+ "passed": true,
202+ "detail": "2 steps consistent"
203+ },
204+ {
205+ "id": "chronology:first_origin<=last_origin",
206+ "passed": true,
207+ "detail": "2023-07-01 vs 2026-06-01"
208+ },
209+ {
210+ "id": "reproducibility",
211+ "passed": true,
212+ "detail": "regenerated in 1.7s; identical within tolerance"
213+ }
214+ ]
215+}
\ No newline at end of file
added runs/pilot/RP0021/out/state/gate_receipts/20261030T040120.873744Z.json +215 −0
@@ -0,0 +1,215 @@
1+{
2+ "gate_version": "pgar_gate/1.0",
3+ "workspace": "/Users/simon-pierreboucher/wp29/runs/RP0021",
4+ "utc": "2026-10-30T04:01:20.873481+00:00",
5+ "passed": true,
6+ "n_checks": 41,
7+ "n_failed": 0,
8+ "checks": [
9+ {
10+ "id": "file:out/results.json",
11+ "passed": true,
12+ "detail": "present"
13+ },
14+ {
15+ "id": "file:out/report.md",
16+ "passed": true,
17+ "detail": "present"
18+ },
19+ {
20+ "id": "file:out/analysis.py",
21+ "passed": true,
22+ "detail": "present"
23+ },
24+ {
25+ "id": "file:out/claims.json",
26+ "passed": true,
27+ "detail": "present"
28+ },
29+ {
30+ "id": "file:out/forecasts.csv",
31+ "passed": true,
32+ "detail": "present"
33+ },
34+ {
35+ "id": "results_json",
36+ "passed": true,
37+ "detail": "parsed"
38+ },
39+ {
40+ "id": "key:task_id",
41+ "passed": true,
42+ "detail": "ok"
43+ },
44+ {
45+ "id": "key:model_name",
46+ "passed": true,
47+ "detail": "ok"
48+ },
49+ {
50+ "id": "key:model_description",
51+ "passed": true,
52+ "detail": "ok"
53+ },
54+ {
55+ "id": "key:n_origins",
56+ "passed": true,
57+ "detail": "ok"
58+ },
59+ {
60+ "id": "key:first_origin",
61+ "passed": true,
62+ "detail": "ok"
63+ },
64+ {
65+ "id": "key:last_origin",
66+ "passed": true,
67+ "detail": "ok"
68+ },
69+ {
70+ "id": "key:n_evaluated_h1",
71+ "passed": true,
72+ "detail": "ok"
73+ },
74+ {
75+ "id": "key:n_evaluated_h3",
76+ "passed": true,
77+ "detail": "ok"
78+ },
79+ {
80+ "id": "key:rmse_naive_h1",
81+ "passed": true,
82+ "detail": "ok"
83+ },
84+ {
85+ "id": "key:mae_naive_h1",
86+ "passed": true,
87+ "detail": "ok"
88+ },
89+ {
90+ "id": "key:rmse_naive_h3",
91+ "passed": true,
92+ "detail": "ok"
93+ },
94+ {
95+ "id": "key:mae_naive_h3",
96+ "passed": true,
97+ "detail": "ok"
98+ },
99+ {
100+ "id": "key:rmse_model_h1",
101+ "passed": true,
102+ "detail": "ok"
103+ },
104+ {
105+ "id": "key:mae_model_h1",
106+ "passed": true,
107+ "detail": "ok"
108+ },
109+ {
110+ "id": "key:rmse_model_h3",
111+ "passed": true,
112+ "detail": "ok"
113+ },
114+ {
115+ "id": "key:mae_model_h3",
116+ "passed": true,
117+ "detail": "ok"
118+ },
119+ {
120+ "id": "key:rel_improvement_rmse_h1",
121+ "passed": true,
122+ "detail": "ok"
123+ },
124+ {
125+ "id": "key:rel_improvement_rmse_h3",
126+ "passed": true,
127+ "detail": "ok"
128+ },
129+ {
130+ "id": "key:feasibility",
131+ "passed": true,
132+ "detail": "ok"
133+ },
134+ {
135+ "id": "key:feasibility_reason",
136+ "passed": true,
137+ "detail": "ok"
138+ },
139+ {
140+ "id": "key:data_issues",
141+ "passed": true,
142+ "detail": "ok"
143+ },
144+ {
145+ "id": "key:n_input_rows",
146+ "passed": true,
147+ "detail": "ok"
148+ },
149+ {
150+ "id": "key:n_analysed",
151+ "passed": true,
152+ "detail": "ok"
153+ },
154+ {
155+ "id": "key:random_seed",
156+ "passed": true,
157+ "detail": "ok"
158+ },
159+ {
160+ "id": "section:Question",
161+ "passed": true,
162+ "detail": "present"
163+ },
164+ {
165+ "id": "section:Data and exclusions",
166+ "passed": true,
167+ "detail": "present"
168+ },
169+ {
170+ "id": "section:Method",
171+ "passed": true,
172+ "detail": "present"
173+ },
174+ {
175+ "id": "section:Results",
176+ "passed": true,
177+ "detail": "present"
178+ },
179+ {
180+ "id": "section:Uncertainty",
181+ "passed": true,
182+ "detail": "present"
183+ },
184+ {
185+ "id": "section:Limitations and open issues",
186+ "passed": true,
187+ "detail": "present"
188+ },
189+ {
190+ "id": "claims_json",
191+ "passed": true,
192+ "detail": "5 claims, all keys resolve"
193+ },
194+ {
195+ "id": "phase_ledger",
196+ "passed": true,
197+ "detail": "consistent"
198+ },
199+ {
200+ "id": "exclusion_ledger",
201+ "passed": true,
202+ "detail": "2 steps consistent"
203+ },
204+ {
205+ "id": "chronology:first_origin<=last_origin",
206+ "passed": true,
207+ "detail": "2023-07-01 vs 2026-06-01"
208+ },
209+ {
210+ "id": "reproducibility",
211+ "passed": true,
212+ "detail": "regenerated in 1.7s; identical within tolerance"
213+ }
214+ ]
215+}
\ No newline at end of file
added runs/pilot/RP0021/out/state/gate_receipts/20261030T040125.359912Z.json +215 −0
@@ -0,0 +1,215 @@
1+{
2+ "gate_version": "pgar_gate/1.0",
3+ "workspace": "/Users/simon-pierreboucher/wp29/runs/RP0021",
4+ "utc": "2026-10-30T04:01:25.359644+00:00",
5+ "passed": true,
6+ "n_checks": 41,
7+ "n_failed": 0,
8+ "checks": [
9+ {
10+ "id": "file:out/results.json",
11+ "passed": true,
12+ "detail": "present"
13+ },
14+ {
15+ "id": "file:out/report.md",
16+ "passed": true,
17+ "detail": "present"
18+ },
19+ {
20+ "id": "file:out/analysis.py",
21+ "passed": true,
22+ "detail": "present"
23+ },
24+ {
25+ "id": "file:out/claims.json",
26+ "passed": true,
27+ "detail": "present"
28+ },
29+ {
30+ "id": "file:out/forecasts.csv",
31+ "passed": true,
32+ "detail": "present"
33+ },
34+ {
35+ "id": "results_json",
36+ "passed": true,
37+ "detail": "parsed"
38+ },
39+ {
40+ "id": "key:task_id",
41+ "passed": true,
42+ "detail": "ok"
43+ },
44+ {
45+ "id": "key:model_name",
46+ "passed": true,
47+ "detail": "ok"
48+ },
49+ {
50+ "id": "key:model_description",
51+ "passed": true,
52+ "detail": "ok"
53+ },
54+ {
55+ "id": "key:n_origins",
56+ "passed": true,
57+ "detail": "ok"
58+ },
59+ {
60+ "id": "key:first_origin",
61+ "passed": true,
62+ "detail": "ok"
63+ },
64+ {
65+ "id": "key:last_origin",
66+ "passed": true,
67+ "detail": "ok"
68+ },
69+ {
70+ "id": "key:n_evaluated_h1",
71+ "passed": true,
72+ "detail": "ok"
73+ },
74+ {
75+ "id": "key:n_evaluated_h3",
76+ "passed": true,
77+ "detail": "ok"
78+ },
79+ {
80+ "id": "key:rmse_naive_h1",
81+ "passed": true,
82+ "detail": "ok"
83+ },
84+ {
85+ "id": "key:mae_naive_h1",
86+ "passed": true,
87+ "detail": "ok"
88+ },
89+ {
90+ "id": "key:rmse_naive_h3",
91+ "passed": true,
92+ "detail": "ok"
93+ },
94+ {
95+ "id": "key:mae_naive_h3",
96+ "passed": true,
97+ "detail": "ok"
98+ },
99+ {
100+ "id": "key:rmse_model_h1",
101+ "passed": true,
102+ "detail": "ok"
103+ },
104+ {
105+ "id": "key:mae_model_h1",
106+ "passed": true,
107+ "detail": "ok"
108+ },
109+ {
110+ "id": "key:rmse_model_h3",
111+ "passed": true,
112+ "detail": "ok"
113+ },
114+ {
115+ "id": "key:mae_model_h3",
116+ "passed": true,
117+ "detail": "ok"
118+ },
119+ {
120+ "id": "key:rel_improvement_rmse_h1",
121+ "passed": true,
122+ "detail": "ok"
123+ },
124+ {
125+ "id": "key:rel_improvement_rmse_h3",
126+ "passed": true,
127+ "detail": "ok"
128+ },
129+ {
130+ "id": "key:feasibility",
131+ "passed": true,
132+ "detail": "ok"
133+ },
134+ {
135+ "id": "key:feasibility_reason",
136+ "passed": true,
137+ "detail": "ok"
138+ },
139+ {
140+ "id": "key:data_issues",
141+ "passed": true,
142+ "detail": "ok"
143+ },
144+ {
145+ "id": "key:n_input_rows",
146+ "passed": true,
147+ "detail": "ok"
148+ },
149+ {
150+ "id": "key:n_analysed",
151+ "passed": true,
152+ "detail": "ok"
153+ },
154+ {
155+ "id": "key:random_seed",
156+ "passed": true,
157+ "detail": "ok"
158+ },
159+ {
160+ "id": "section:Question",
161+ "passed": true,
162+ "detail": "present"
163+ },
164+ {
165+ "id": "section:Data and exclusions",
166+ "passed": true,
167+ "detail": "present"
168+ },
169+ {
170+ "id": "section:Method",
171+ "passed": true,
172+ "detail": "present"
173+ },
174+ {
175+ "id": "section:Results",
176+ "passed": true,
177+ "detail": "present"
178+ },
179+ {
180+ "id": "section:Uncertainty",
181+ "passed": true,
182+ "detail": "present"
183+ },
184+ {
185+ "id": "section:Limitations and open issues",
186+ "passed": true,
187+ "detail": "present"
188+ },
189+ {
190+ "id": "claims_json",
191+ "passed": true,
192+ "detail": "5 claims, all keys resolve"
193+ },
194+ {
195+ "id": "phase_ledger",
196+ "passed": true,
197+ "detail": "consistent"
198+ },
199+ {
200+ "id": "exclusion_ledger",
201+ "passed": true,
202+ "detail": "2 steps consistent"
203+ },
204+ {
205+ "id": "chronology:first_origin<=last_origin",
206+ "passed": true,
207+ "detail": "2023-07-01 vs 2026-06-01"
208+ },
209+ {
210+ "id": "reproducibility",
211+ "passed": true,
212+ "detail": "regenerated in 1.7s; identical within tolerance"
213+ }
214+ ]
215+}
\ No newline at end of file
added runs/pilot/RP0021/out/state/gate_receipts/20261030T040136.115952Z.json +215 −0
Large diff (216 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0021/out/state/log.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0021/out/state/phases.json +60 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0021/sessions/6c89c7fd-c6e2-42ae-af54-1f1d109ce3fd.jsonl +80 −0
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/artifact_manifest.json +17 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/contract_receipt.json +217 −0
Large diff (218 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/ctl/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/evaluation_receipt.json +158 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/analysis.py +153 −0
Large diff (153 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/claims.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/exclusions.csv +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/forecasts.csv +145 −0
Large diff (145 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/report.md +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/results.json +47 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/state/checks.md +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/state/contract.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/state/data_audit.md +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/state/log.md +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/out/state/phases.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0022/sessions/d18d1f47-f7dc-49d7-b7b0-229ed6d72a98.jsonl +93 −0
Large diff (93 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/artifact_manifest.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/behaviour_audit.json +38 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/contract_receipt.json +217 −0
Large diff (218 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/ctl/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/evaluation_receipt.json +158 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/analysis.py +115 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/exclusions.csv +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/forecasts.csv +145 −0
Large diff (145 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/report.md +36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/results.json +48 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/review/review.json +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/checks.md +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/data_audit.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/gate_receipts/20261030T040119.136024Z.json +215 −0
Large diff (216 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/gate_receipts/20261030T040131.012947Z.json +215 −0
Large diff (216 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/gate_receipts/20261030T040134.939723Z.json +215 −0
Large diff (216 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/gate_receipts/20261030T040144.940110Z.json +215 −0
Large diff (216 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/log.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/out/state/phases.json +54 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/sessions/3c0817b8-c7e8-47f1-b003-1b4f1c3b06c0.jsonl +45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0023/sessions/fd081072-6055-4453-8558-18099d7c36dc.jsonl +95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
modified runs/pilot/pilot_batch2_controller.log +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
modified runs/state.sqlite +0 −0

Binary file not shown.

deleted runs/state.sqlite-shm +0 −0

Binary file not shown.

deleted runs/state.sqlite-wal +0 −0

Binary file not shown.

added src/pgar/recovery_experiment.py +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
modified src/pgar/scheduler.py +11 −1
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere