# T3-synthesis — Does generative-AI assistance shorten task-completion time? A five-study evidence synthesis

# Question

Does having a generative-AI assistant change how long workers (software developers, IT administrators, security analysts) take to finish their tasks? Every study is mapped onto one target, θ = ln(T_AI / T_noAI). This is the natural log of the ratio of mean completion time with AI to mean completion time without AI, for the same tasks, in a randomised comparison. A negative θ means people were faster with AI (target_definition).

# Data and exclusions

The inputs are five text extracts of arXiv e-prints and data/studies.json, which says which printed quantity to take from each. All SHA-256 values match data/README.md (see out/state/data_audit.md). Line counts are in n_input_rows. All five studies were assessed (n_analysed = 5), and a printed value was extracted for each (k_extracted = 5). Exclusion ledger (out/exclusions.csv), studies only:

So k_harmonised = 4 and k_pooled = 2 (S1, S4; pooled_studies). The full extraction table is out/table1.csv (same content as items). For every study it gives the values exactly as printed, a locator (extract line numbers), the transformation used, and the reason for any exclusion.

data_issues lists 13 problems found in the inputs. The most important:

• S1: the printed mean log-times (4.6 with AI, 4.3 without) contradict the raw means (96 vs 114 min). These means are not used.
• S1: Table III gives n = 96, but N = 47 + 46 = 93 in Section VI-B.
• S3 and S4: the printed percentages do not exactly match the ratio of the printed means (−29.79% vs −29.81%; 11.1% vs 11.25%).
• S4: the seconds-per-email values in the Malicious and Non-Malicious columns are implausible next to the Full Corpus column, so the units are ambiguous. Only the Full Corpus column is used.

# Method

Harmonisation follows the conventions in TASK.md:

• S1, Paradis et al. 2024. The coefficient on the AI indicator in Model 2 of Table III, which regresses log time on task, is used directly: θ = b = −0.24 and SE = 0.14. The SE implied by the printed CI [−0.51, 0.03] is 0.1378; this is kept only as a sensitivity check.
• S2, Becker et al. 2025. θ = ln(1 + 0.19). There is no SE.
• S3, Bono & Xu 2024. θ = ln(1 − 0.2979). There is no SE, because only stars are printed.
• S4, Bono 2025. θ = ln(1 − 0.111), comparing the Aware arm with Control. The SE is approximate, recovered from the two-sided p = 0.471 as |θ| / z₀.₇₆₄₅.
• S5, Kreitmeir & Raschky 2023. Not harmonised.

The two poolable studies were combined with an inverse-variance fixed-effect model and with DerSimonian–Laird random effects. Both use 95% intervals of ±1.96·SE, Cochran's Q (df = 1), I² and τ² as defined in TASK.md. As a baseline, I also computed the unweighted mean θ of the pooled studies (baseline_unweighted_mean_poolable). out/analysis.py checks every source line it relies on, prints those lines, and regenerates results.json, table1.csv and figure1.png. It involves no randomness (random_seed = null).

# Results

Harmonised θ for each study (items.*.harmonised_estimate, items.*.harmonised_se):

• S1: −0.240 (SE 0.140)
• S2: +0.174 (no SE; 19% slower)
• S3: −0.354 (no SE)
• S4: −0.118 (approximate SE 0.163)

Pooled over S1 and S4:

• Fixed effect: θ = −0.188 (fe_estimate), SE 0.106 (fe_se), 95% CI [−0.396, 0.020] (fe_ci_low, fe_ci_high).
• Random effects: identical to the fixed effect, because τ² = 0 (tau2). θ = −0.188 (re_estimate), SE 0.106 (re_se), 95% CI [−0.396, 0.020] (re_ci_low, re_ci_high). This corresponds to a 17.2% shorter completion time (pooled_pct_change_re = −17.15).
• The random-effects interval includes 0 (pooled_significant_5pct = false).
• Heterogeneity between the two pooled studies: Q = 0.324 (q_stat), I² = 0% (i2), rated "low" (heterogeneity_interpretation).
• The baseline unweighted mean of the two is −0.179 (baseline_unweighted_mean_poolable).

Figure: out/figure1.png is a forest plot. S2 and S3 are drawn as points without intervals, and the random-effects result as a diamond.

# Uncertainty

• The pooled interval rests on only two studies. One of them (S4) has an SE back-calculated from a p-value, which assumes normality and that the p-value comes from a two-sided z-type test.
• The heterogeneity statistics carry almost no information. With k = 2, Q has one degree of freedom, so I² = 0 and τ² = 0 do not show that the studies agree; τ² is essentially unidentified.
• Sensitivity checks barely move the result. Using the S1 SE implied by its CI gives a fixed-effect estimate of −0.189 (sensitivity_s1_se_from_ci). Using the S4 ratio of printed means instead of the printed percentage gives −0.190 (sensitivity_s4_ratio_of_means).
• The excluded studies point in different directions. S3, from an RCT reported only with stars (p ≤ 0.01), suggests a large speed-up of θ = −0.354. S2, a field RCT of experienced developers, suggests a slowdown of θ = +0.174. Neither can be weighted.

# Limitations and open issues

• What the numbers can and cannot support. The two poolable studies give a point estimate of about 17% shorter task time with AI. That estimate is not statistically distinguishable from no effect at the 5% level (pooled_significant_5pct = false), and it should not be read as the overall effect of generative AI on completion time:
  • only 2 of the 5 studies enter the pool (k_pooled), and one of those has an approximate SE;
  • the two harmonisable studies that are left out disagree in sign (S2 slower, S3 faster), so the true spread of effects is probably larger than the pooled I² suggests;
  • the settings differ a lot: one lab coding task (S1), real open-source issues (S2), IT-admin tasks with accuracy held constant (S3), and phishing triage where the agent also reorders the queue (S4).
• Not every θ means exactly the same thing. S1 and S2 come from log-time regressions with covariate adjustment. Strictly, that gives a difference in mean log time, which equals the log ratio of means only under the assumptions spelled out in Becker et al. Appendix D.3. S3 is adjusted for accuracy, and S4's Aware arm bundles AI verdicts with queue prioritisation.
• The extracts limit what can be obtained. SEs for S2 and S3 probably exist in the full papers or figures, but the extracts do not contain them, and under the information rules they were not substituted. S5 measures a different construct (output volume after access was withdrawn) and cannot be put on the θ scale.
• Feasibility. The project is feasibility = "complete" under TASK.md's definition: every item is either harmonised or has a justified statement of what cannot be obtained, and the pooled estimates use all poolable items. The substantive evidence, however, is thin.