SPB Git

spb/wp29_uqo

Public

# Contract (P0) — T3-synthesis

Question. Does access to a generative-AI assistant shorten task-completion time for workers (developers, IT admins, security analysts)? Synthesise five primary studies given only as text extracts.

Estimand. θ = ln(T_AI / T_noAI): log ratio of mean completion time with vs without AI on the same tasks, from a randomised comparison (negative = faster with AI). Meaningful only if the printed effect is an AI-vs-no-AI comparison of completion time on the same tasks; pooling assumes the studies estimate (fixed effect) the same θ or (random effects) draws from a distribution of θ — with k=2 τ² is barely identified.

Per study: the printed value verbatim (estimate, uncertainty, sample, definition, locator), harmonised θ and SE (per TASK conventions: log-time coefficient → θ=b, SE=SE(b); % change c → ln(1+c); p-value only → SE≈|θ|/z_{1-p/2} (approximate); stars only / figure-only CI → not poolable; not time-on-same-task AI-vs-no-AI → not harmonisable). Pooled: IV fixed effect, DerSimonian–Laird RE, 95% using 1.96, Q (df=k−1), I², τ², interpretation thresholds 25/50, significance = RE CI excludes 0, pooled % change = 100(exp(re)−1).

Outputs. out/table1.csv (5 rows; study, estimate_as_printed, uncertainty_as_printed, sample_as_printed, definition, locator, harmonised_estimate, harmonised_se, harmonisable, poolable, transformation, reason_if_not); out/figure1.png forest plot (harmonisable studies, no-SE as points, RE diamond); out/results.json keys: task_id, target_definition, items{S1..S5 with the fields above}, k_extracted, k_harmonised, k_pooled, fe_estimate, fe_se, fe_ci_low, fe_ci_high, re_estimate, re_se, re_ci_low, re_ci_high, q_stat, i2, tau2, heterogeneity_interpretation, pooled_significant_5pct, pooled_pct_change_re, feasibility, feasibility_reason, data_issues, n_input_rows, n_analysed, random_seed; out/report.md (Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues); out/analysis.py (regenerates results/table/figure, prints relied-on source lines); out/claims.json; plus protocol files (state/, exclusions.csv).

Information rules. Only numbers printed in the extracts are admissible; no remembered values (e.g. METR CI shown only in Figure 1, Bono & Xu bootstrap SEs not printed). Printed numbers reported verbatim.

Invalidity / impossibility. Fewer than 1 poolable study would make pooled keys impossible (they are non-nullable). Using non-printed values, mis-transcribing printed values, or pooling non-poolable studies would invalidate results.

Files · main
Menu

Theme