SPB Git

spb/wp29_uqo

Public
2.0 KB · 21 lines json
1{2  "verdict": "accept",3  "issues": [4    {5      "id": "R1",6      "severity": "minor",7      "location": "out/report.md (Uncertainty); claims.json C3",8      "issue": "The DM/HLN test treats the forecast errors as given and ignores the uncertainty from estimating the ridge parameters on expanding windows. For model-based forecasts, a Giacomini-White-style interpretation (conditional on the estimation scheme) is the more defensible reading. The p-values are approximate for this reason as well as because of the small sample.",9      "suggested_fix": "Add a sentence to the C3 caveat and the Uncertainty section that the test is read as equal predictive accuracy of the forecasting methods, conditional on the estimation scheme."10    },11    {12      "id": "R2",13      "severity": "minor",14      "location": "out/results.json (dm_* keys)",15      "issue": "Extra keys that the task did not ask for (dm_stat_*, dm_pvalue_*) were added to results.json. They are harmless and the report labels them as additional, but they go beyond the required schema.",16      "suggested_fix": "None required; optionally move them to a separate file or sub-object."17    }18  ],19  "summary": "Re-running analysis.py in a scratch copy reproduces results.json and forecasts.csv byte-for-byte. An independent re-implementation matches all six RMSE/MAE values to within floating-point precision, and n_analysed = 413. The data SHA-256 matches the README. The code follows the information rules: the training rows require s+1 <= t, mu, sigma (ddof=1) and ybar are recomputed at every origin, the forecast uses only features dated t, and the expanding-mean window and ridge formula match the specification. forecasts.csv has 180 rows. The exclusion ledger (417 -> 415 -> 413) is consistent. The claims are worded in proportion to the evidence: ridge beats naive (36.5% lower RMSE, DM p=0.0014) but does not beat the expanding mean (-3.1%, p=0.59), and the report correctly calls that comparison inconclusive. The limitations (current vintage, a single lambda, COVID outliers) are stated honestly."20}21
Files · main
Menu

Theme