SPB Git

spb/wp29_uqo

Public

# Contract — T3-synthesis (written P0, 2026-10-30)

Question. Does access to a generative-AI assistant shorten the time workers need to complete their tasks? Synthesis of five pre-extracted primary sources (data/sources/*.txt), using only what is printed in the extracts.

Estimands.

  • For each of S1..S5: the printed value named in data/studies.json (extract), verbatim, with its printed uncertainty, sample and locator.
  • θ = ln(T_AI / T_noAI) (ratio of mean task-completion time with vs without AI, same tasks, randomised comparison; negative = faster with AI), and SE(θ), for every study where obtainable under the TASK conventions (log-time coefficient → θ=b; % change c → θ=ln(1+c); CI → SE=(U−L)/3.92; p-only → SE≈|θ|/z_{1−p/2} (approximate); stars / figure-only CI → harmonisable but not poolable; non-time or non-AI-vs-no-AI → not harmonisable).
  • Inverse-variance FE and DerSimonian–Laird RE pooled θ over poolable studies, 95% CIs with 1.96, Q (df=k−1), I², τ², heterogeneity label (<25 low, 25–50 moderate, ≥50 substantial), pooled_significant_5pct (RE CI excludes 0), pooled_pct_change_re = 100(exp(re)−1).
  • Meaningful only if: the harmonised quantities target the same contrast (AI vs no AI, time on same tasks, randomised); normal approx on log scale; approximate SE from p-values is taken as valid; with k_pooled small, τ² is poorly estimated.

Outputs. out/table1.csv (12 columns, 5 rows), out/figure1.png (forest plot, harmonisable studies, no-SE as bare points, RE diamond), out/results.json (keys listed in TASK.md: task_id, target_definition, items{S1..S5 with 11 fields}, k_extracted, k_harmonised, k_pooled, fe_, re_, q_stat, i2, tau2, heterogeneity_interpretation, pooled_significant_5pct, pooled_pct_change_re, feasibility, feasibility_reason, data_issues, n_input_rows, n_analysed, random_seed), out/report.md (Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues), out/analysis.py (regenerates json/csv/png from data/ only, prints source lines used), out/claims.json; plus protocol files (state/, exclusions.csv).

Information rules. Only numbers printed in the extracts are admissible; no remembered values (e.g., METR CI in Figure 1 is not available). Printed values reported exactly as printed.

Invalidity / impossibility. Using non-printed numbers; pooling studies without legitimate SE; pooling non-time outcomes; <1 poolable study would make pooling impossible (not the case if ≥2 have SE).

Files · main
Menu

Theme