1# T3-synthesis — Does generative-AI assistance shorten task-completion time?23## Question4Does access to a generative-AI assistant change the time workers need to finish their tasks? Target quantity (`target_definition`): θ = ln(T_AI / T_noAI), the log ratio of mean completion time with vs without AI, same tasks, randomised comparison (negative = faster with AI). Five primary studies were assessed (`n_analysed` = 5); only the provided extracts were used as evidence.56## Data and exclusions7Inputs: `data/studies.json` (5 items) and five arXiv text extracts (line counts in `n_input_rows`); all SHA-256 hashes match `data/README.md` (see `out/state/data_audit.md`). Table 1 (`out/table1.csv`, mirrored in `items`) gives the printed values verbatim with locators:89| study | printed estimate | printed uncertainty | θ | SE(θ) | poolable |10|---|---|---|---|---|---|11| S1 Paradis et al. 2024 (Google RCT) | ExpCon Model 2 = −0.24 (log time) | SE (0.14); 95% CI [−0.51, 0.03]; p=0.086 | −0.240 | 0.140 | yes |12| S2 Becker et al. 2025 (METR RCT) | 19% more time | none printed (CI only in a figure) | 0.174 | — | no |13| S3 Bono & Xu 2024 (Copilot RCT) | 44.29 vs 63.10 min, −29.79%*** | stars only | −0.354 | — | no |14| S4 Bono 2025 (phishing agent RCT) | 50.5 vs 56.9, 11.1% time savings | p = 0.471 | −0.118 | 0.163 (approx.) | yes |15| S5 Kreitmeir & Raschky 2023 (DiD) | −0.278 on Output | SE (0.295), N=18332 | — | — | not harmonisable |1617(Values from `items.S1`…`items.S5`.) Exclusion ledger (`out/exclusions.csv`): 5 extracted (`k_extracted` = 5) → 4 harmonisable (`k_harmonised` = 4; S5 removed: its outcome is GitHub output counts around the ChatGPT ban in a non-randomised DiD, not completion time) → 2 poolable (`k_pooled` = 2; S2 removed because its uncertainty appears only in a figure that is not reproduced; S3 removed because only significance stars are printed).1819Data issues detected in the sources (`data_issues`) include: S1 sample stated as n = 96 in Table III but 47 + 46 = 93 in the text, and log-time means that look swapped relative to the raw means; S3's displayed minutes give −29.81% vs the printed −29.79%; S4's printed 11.1% saving does not equal 1 − 50.5/56.9 = 11.25%, and the "(s) per email" units of its Full Corpus column cannot be reconciled with the other columns; S5's table note prints "** p<0.5". None of these changes the printed headline quantities used.2021## Method22Harmonisation followed the TASK conventions (formula and inputs in each `items.*.transformation`):23- S1: coefficient on log time → θ = b = −0.24, SE = printed 0.14.24- S2: +19% → θ = ln(1.19). This equals the regression β, because the paper defines S = exp(β) − 1.25- S3: −29.79% → θ = ln(1 − 0.2979).26- S4: 11.1% time saving → c = −0.111, θ = ln(0.889). The SE is **approximate**, recovered from the two-sided p = 0.471 as |θ|/z₀.₇₆₄₅.27- S5: not harmonisable.2829Pooling over S1 and S4 used inverse-variance fixed effect and DerSimonian–Laird random effects on the θ scale, with 1.96-based 95% intervals, Q on k−1 = 1 df, and I² and τ² as defined in TASK.md. As a baseline (P2), I computed unweighted means of θ. `out/analysis.py` regenerates everything deterministically (`random_seed` = null) and prints the source lines it parses.3031## Results32- Harmonised θ per study: S1 −0.240, S2 +0.174, S3 −0.354, S4 −0.118 (`items.*.harmonised_estimate`). Three point estimates indicate less time with AI, and the METR study indicates more (+19%).33- Fixed effect (k = 2): θ = −0.188, SE 0.106, 95% CI [−0.396, 0.020] (`fe_estimate`, `fe_se`, `fe_ci_low`, `fe_ci_high`).34- Random effects (DL): θ = −0.188, SE 0.106, 95% CI [−0.396, 0.020] (`re_estimate`, `re_se`, `re_ci_low`, `re_ci_high`). This corresponds to a 17.2% shorter completion time (`pooled_pct_change_re` = −17.15), with an interval from −32.7% to +2.0% (`pooled_pct_change_re_ci_low`, `pooled_pct_change_re_ci_high`).35- Heterogeneity: Q = 0.324 (`q_stat`), I² = 0% (`i2`), τ² = 0 (`tau2`), labelled "low" (`heterogeneity_interpretation`).36- The pooled estimate is **not** significant at 5%: the RE interval includes 0 (`pooled_significant_5pct` = false).37- Baselines: the unweighted mean of the two pooled θ is −0.179 (`baseline_unweighted_mean_theta_pooled`), and the unweighted mean of all four harmonised θ is −0.134 (`baseline_unweighted_mean_theta_harmonised`).38- Forest plot: `out/figure1.png`.3940**Synthesis.** The two poolable RCTs point toward shorter completion times with AI (about −17%), but the pooled interval includes no effect. The evidence therefore does not establish that AI assistance shortens task time. The data are also compatible with no change, or with a small slowdown.4142## Uncertainty43- The pooled interval rests on only two studies. One of them (S4) has an SE back-calculated from a p-value, which assumes normality on the θ scale even though the paper's test was on its own scale.44- With k = 2, Q has 1 df. The finding τ² = 0 / I² = 0 reflects very low power to detect heterogeneity, not evidence of homogeneity.45- RE = FE here only because τ² was truncated at 0.46- Sensitivity checks:47 - Using S1's CI-derived SE (0.1378) instead of the printed 0.14 gives RE −0.189 (SE 0.105) (`sensitivity_ci_se_s1_re_estimate`, `sensitivity_ci_se_s1_re_se`).48 - Using S4's ratio of displayed means (θ = −0.119, `sensitivity_ratio_s4_theta`) gives RE −0.190 (SE 0.107) (`sensitivity_ratio_s4_re_estimate`, `sensitivity_ratio_s4_re_se`).49 - Neither change alters the conclusion.5051## Limitations and open issues52- **The pooled estimate is not a summary of the evidence base.** It leaves out S2 (+19%, the only study of experienced developers on their own real issues) and S3 (−35%), solely because their uncertainty is not printed in the extracts. Adding S2 would pull the pooled estimate toward zero, and adding S3 would pull it away from zero; neither direction can be quantified legitimately. The four harmonised point estimates range from −0.354 to +0.174, which points to real heterogeneity that k = 2 cannot detect.53- **The studies are not comparable.** They differ in population (Google engineers; open-source maintainers; Upwork IT administrators; security analysts), task (one lab coding task; real issues; admin scenarios; email triage), and the AI arm itself:54 - S4's Aware arm also has an agent-prioritised queue.55 - S3's control group partly used other AI tools.56 - S3 holds accuracy constant via a model-based adjustment.57 - S1 and S2 condition on covariates.58- **The SEs are uncertain.** The S4 SE is approximate. For S1, the reported sample size is internally inconsistent (93 vs 96).59- **S5 is excluded by design.** It measures output counts after AI withdrawal, a different construct. Its estimate (−0.278, SE 0.295) is reported but not harmonised.60- **Feasibility.** I set `feasibility` to "complete" (`feasibility_reason` = null): every item has harmonised values or a justified statement of what cannot be obtained, and the pooled estimates were computed over the poolable items, as TASK.md defines it. "Complete" refers to the project, not to the strength of the evidence, which is weak.61