# T3-synthesis — Does generative-AI assistance shorten task-completion time? A small evidence synthesis

# Question

Does access to a generative-AI assistant shorten the time workers need to complete their tasks? Five primary studies were assessed (n_analysed). Each was mapped onto one target scale, θ = ln(T_AI / T_noAI): the log ratio of mean completion time with AI to mean completion time without it, on the same tasks, from a randomised comparison. Negative θ means faster with AI (target_definition).

# Data and exclusions

The only evidence used is the five plain-text extracts under data/sources/ plus data/studies.json. Their SHA-256 values match data/README.md. Line counts are in n_input_rows, and the full audit is in out/state/data_audit.md. The printed values for each study are in items.*.estimate_as_printed, items.*.uncertainty_as_printed and items.*.locator, and in out/table1.csv.

All 5 items had a printed value (k_extracted = 5). The exclusion ledger (out/exclusions.csv) runs 5 assessed → 4 harmonisable → 2 poolable:

• S5 (Kreitmeir & Raschky) is not harmonisable (items.S5.harmonisable = false). It reports a difference-in-differences coefficient on GitHub output counts around Italy's ChatGPT ban (−0.278, SE 0.295, N = 18332). That is an activity-volume effect from an observational design, not a comparison of completion times on the same tasks (items.S5.reason_if_not).
• S2 (Becker et al., +19%) is harmonisable but not poolable. Its confidence interval appears only in figures, which are not reproduced in the extract (items.S2.reason_if_not).
• S3 (Bono & Xu, −29.79%)* is harmonisable but not poolable. Its uncertainty is given by significance stars only (items.S3.reason_if_not).

Problems found in the inputs are listed in data_issues:

• S1: Section VI-B gives a Mean LogToT of 4.6 (AI) vs 4.3 (no-AI). That contradicts the AI group being faster (96 vs 114 min). VI-B also gives N = 93, while Table III says n = 96.
• S3: the printed durations imply −29.81%, not the printed −29.79%.
• S4: the printed times imply 11.25% savings, not the printed 11.1%. The Malicious/Non-Malicious columns have ambiguous units.
• S5: the significance legend contains a typo.

None of these issues changes the values used.

# Method

Harmonisation followed the conventions in TASK.md. Each formula, with its printed inputs, is in items.*.transformation.

• S1 (Paradis et al., Google RCT): the ExpCon coefficient on log time on task in Table III, Model 2, is taken directly: θ = −0.24, with the printed SE of 0.14 (items.S1.harmonised_estimate, items.S1.harmonised_se).
• S2: θ = ln(1.19) (items.S2.harmonised_estimate).
• S3: θ = ln(1 − 0.2979) (items.S3.harmonised_estimate).
• S4 (Bono, phishing-triage RCT, Aware vs Control, Full Corpus): θ = ln(1 − 0.111). Only a p-value is printed, so the SE is approximate: SE ≈ |θ| / z₁₋ₚ/₂ with p = 0.471 (items.S4.harmonised_estimate, items.S4.harmonised_se).

Pooling uses only the poolable studies, S1 and S4 (k_pooled = 2). Methods: inverse-variance fixed effect and DerSimonian–Laird random effects on the θ scale, with 95% intervals of ±1.96·SE. Heterogeneity is summarised by Cochran's Q (df = 1), I² and τ², using the formulas in TASK.md.

As a simple baseline, the script also reports the unweighted mean θ (baseline_unweighted_mean_theta_harmonised, baseline_unweighted_mean_theta_poolable) and a vote count of studies with negative θ (baseline_vote_count_faster_with_ai).

Everything is produced by out/analysis.py, which is deterministic (random_seed = null). It also generates out/table1.csv and out/figure1.png.

# Results

Harmonised θ by study:

Keys: items.S1.harmonised_estimate, items.S2.harmonised_estimate, items.S3.harmonised_estimate, items.S4.harmonised_estimate, items.S4.harmonised_se. Four studies are harmonised (k_harmonised = 4), and 3 of the 4 point the "faster with AI" way (baseline_vote_count_faster_with_ai).

Pooled over S1 and S4:

• Fixed effect: θ = −0.188, SE 0.106, 95% CI [−0.396, 0.020] (fe_estimate, fe_se, fe_ci_low, fe_ci_high).
• Random effects: identical, because τ² = 0: θ = −0.188, SE 0.106, 95% CI [−0.396, 0.020] (re_estimate, re_se, re_ci_low, re_ci_high, tau2).
• Heterogeneity: Q = 0.324 on 1 df, so I² = 0% and the label is "low" (q_stat, i2, heterogeneity_interpretation).
• As a time change: the pooled random-effects estimate is −17.2% (pooled_pct_change_re), with a 95% interval of about −32.7% to +2.0% (pooled_pct_change_re_ci).
• Significance: the random-effects interval includes 0, so the pooled effect is not significant at the 5% level (pooled_significant_5pct = false).

The baselines agree in size: the unweighted mean of the two poolable θ is −0.179 (baseline_unweighted_mean_theta_poolable). The unweighted mean over all four harmonised studies is −0.134 (baseline_unweighted_mean_theta_harmonised).

Figure 1 (out/figure1.png) is a forest plot. It shows all four harmonised studies (S2 and S3 as points without intervals) and the pooled random-effects estimate.

Synthesis conclusion. The two poolable randomised studies point the same way: completion time is about 17% shorter with AI. However, the pooled 95% interval runs from about a one-third reduction to a slight increase, so the data cannot establish a time reduction at the 5% level. The near-zero heterogeneity estimate means little with only two studies. The unpooled evidence goes in both directions. S3 reports a large, starred reduction for IT administrators. S2, the only field study on real open-source issues, reports a 19% increase in completion time. The evidence therefore does not support a general claim that generative-AI access shortens task time. Effects likely depend on the task and setting.

# Uncertainty

• The pooled interval rests on only k = 2 studies (k_pooled). With 1 df, Q has almost no power, so I² = 0 and τ² = 0 (i2, tau2) do not show that the true effects are homogeneous. With k = 2, the DerSimonian–Laird τ² is especially unreliable, and the random-effects interval probably understates between-study uncertainty.
• S4's SE (items.S4.harmonised_se) is approximate. It was recovered from a p-value under a normal approximation on the log-ratio scale, but the paper's test may have been run on another scale. S4 carries about 42% of the pooled weight (weights_fe).
• Sensitivity check: using S1's CI-implied SE (0.1378) instead of the printed 0.14 gives a random-effects θ of −0.189 with 95% CI [−0.395, 0.017] (sens_re_estimate_s1_ci_se, sens_re_ci_low_s1_ci_se, sens_re_ci_high_s1_ci_se). The conclusion does not change.
• The S1 and S2 estimates are covariate-adjusted coefficients on log time. They equal a log ratio of means only under the distributional assumptions those papers state, such as Becker et al. D.3.

# Limitations and open issues

• Two of the four harmonised studies cannot be pooled. S2 and S3 are left out because the extracts print no usable uncertainty. They are the largest positive and the largest negative effects (+0.174 and −0.354), so the pooled estimate leaves out most of the visible spread between studies. If their intervals had been available, the pooled result and its heterogeneity could look very different.
• The studies are not like-for-like. S1 is one lab coding task. S3 uses a model-predicted control duration "holding accuracy constant". S4's Aware arm bundles AI verdicts with an agent-ordered queue, and its time is per email over the whole queue. S2 is a field setting with self-reported times. Treating these as estimates of one θ is a strong assumption.
• S5 is excluded by design. It measures output volume after access was withdrawn, not task time, so it says nothing about θ.
• Source inconsistencies (data_issues) could not be resolved from the extracts. Each one was handled by using the directly cited printed quantity.
• Feasibility. The project is complete in the sense TASK.md defines (feasibility = "complete"): every item has either harmonised values or an explicit, justified reason why they cannot be obtained, and pooling was done over the poolable items. This does not make the pooled estimate a strong answer to the question. It is a limited, two-study summary.