SPB Git

spb/wp29_uqo

Public
2.8 KB · 11 lines json
1{2  "verdict": "accept",3  "issues": [4    {"id": "R1", "severity": "minor", "location": "claims.json C4; report.md Results item 3", "issue": "C4 says WCB is closer to nominal than CR1 at every G=30 cell. At G30/ICC0 the two are 0.946 and 0.944, a gap of 0.002, which is about 0.4 MC SE. The caveat says the G=30 differences are '1-3 MC SE', which is wrong for this cell. In that cell the ordering cannot be told apart from noise.", "suggested_fix": "Narrow C4 to G=10 with ICC>0 and G30/ICC0.2 and 0.5, or say WCB and CR1 cannot be distinguished at G30/ICC0. Compute paired SEs of the coverage differences, which are cheap because both use the same replications."},5    {"id": "R2", "severity": "minor", "location": "report.md Results item 4", "issue": "The report explains WCB's lower coverage at G=5 despite wider mean intervals by a variable, discrete bootstrap critical value. This is stated as fact, but the distribution of q was never examined. C6 does carry an 'interpretive' caveat; the report text does not.", "suggested_fix": "Report the distribution of q across replications (e.g. quantiles) at G=5, or mark the explanation as a hypothesis in the report."},6    {"id": "R3", "severity": "minor", "location": "report.md Data and exclusions", "issue": "'The SHA-256 hashes match the bundle manifest' cannot be checked from the deliverables. No hash or manifest is reproduced.", "suggested_fix": "Give the hash values or remove the statement."},7    {"id": "R4", "severity": "minor", "location": "figure1.png", "issue": "The naive i.i.d. baseline and the 0.90 failure line are both drawn in red. At ICC=0 the baseline curve sits near the reference lines, which makes the panel harder to read.", "suggested_fix": "Use a different colour for the baseline."}8  ],9  "summary": "The deliverables are complete, reproducible and correctly implemented. A re-run of analysis.py in a scratch copy reproduced results.json and table1.csv exactly (about 2 s). The DGP, CR0 (literal 1.96), CR1 (t with G−1 df), WCB (unrestricted Rademacher bootstrap-t, B=399, 380th order statistic) and the seeds all follow TASK.md. The vectorised WCB uses the identity that the cluster sums of the bootstrap residuals equal w_g·S_g − n_g·D, which is correct. A separate brute-force loop implementation (different RNG) of cell G5/ICC0.2 gave coverages of 0.725/0.883/0.853 against 0.720/0.882/0.847, consistent within MC error. Every required key is present. The failure set, nominal set, best procedure and counts are consistent with the coverage and MC SE values. The Monte Carlo relative bias agrees with analytic expectations (|z| ≤ 2). A naive i.i.d. baseline is included and the limitations are stated honestly. The only issues are minor: one over-stated comparison at G30/ICC0, an untested causal explanation, a hash claim that cannot be verified, and the figure colours."10}11
Files · main
Menu

Theme