# T3-synthesis — Does generative-AI assistance shorten task-completion time?

# Question

The question is whether access to a generative-AI assistant changes how long workers take to finish their tasks. Every study is harmonised onto θ = ln(T_AI / T_noAI). This is the log of the ratio of mean completion time with AI to mean completion time without AI, for the same tasks, from a randomised comparison. Negative values mean workers were faster with AI (target_definition). Five extracts are assessed (n_analysed = 5, k_extracted = 5).

# Data and exclusions

The only inputs are the five arXiv extracts under data/sources/ and data/studies.json. Their SHA-256 hashes match data/README.md, and line counts are in n_input_rows. The full audit is in out/state/data_audit.md, and the extraction table is in out/table1.csv. The exclusion ledger (out/exclusions.csv) runs as follows:

• 5 items extracted.
• S5 is excluded as not harmonisable, leaving 4 (k_harmonised = 4).
• S2 and S3 are excluded as not poolable, leaving 2 (k_pooled = 2, pooled_studies = S1, S4).

Data issues found in the sources are listed with their handling in data_issues. The most important ones are:

• S1: the Mean LogToT values in the text (AI 4.6 vs no-AI 4.3) contradict the minute means and the stated direction. The text also gives n = 96 next to N = 47 + 46 = 93.
• S3: the printed −29.79% differs slightly from the ratio of the printed times.
• S4: the "(s)" unit labels are implausible. The per-email and full-corpus columns cannot both be seconds per email. Because θ is a unit-free ratio, this does not affect the result.

# Method

The conventions from TASK.md are applied as follows (each formula is in items.*.transformation):

• S1: a coefficient on log time, so θ = b and SE = the printed SE(b). The SE implied by the printed CI, 0.137755 (s1_se_from_ci), is used only as a cross-check.
• S2 and S3: printed percentage changes, so θ = ln(1 + c). S2's 19% is exp(β) − 1 from a log-linear regression, so this gives back β.
• S4: θ = ln(1 − 0.111). The approximate SE is |θ| / z_{1−p/2} with p = 0.471, assumed two-sided.
• S2 and S3 are not poolable: S2 prints no SE, CI or p-value (the CI is only in a figure that is not reproduced), and S3 prints only significance stars.
• S5 is not harmonisable: it is a non-randomised difference-in-differences on GitHub output counts around a ChatGPT ban, not a comparison of completion times on the same tasks.

Pooling uses inverse-variance fixed effects and DerSimonian–Laird random effects on the θ scale, with 95% intervals of ±1.96·SE. Q has k − 1 degrees of freedom, and I² and τ² follow the TASK formulas. As a baseline, the unweighted mean θ is baseline_unweighted_mean_theta_poolable = −0.178829 over S1 and S4, and baseline_unweighted_mean_theta_harmonised = −0.134346 over all four harmonised studies. The script (out/analysis.py) is deterministic (random_seed = null), prints every source line it relies on, and checks each number it uses against the source.

# Results

• Fixed effect: fe_estimate = −0.188143, fe_se = 0.106265, 95% CI [fe_ci_low = −0.396422, fe_ci_high = 0.020136].
• Random effects: re_estimate = −0.188143, re_se = 0.106265, 95% CI [re_ci_low = −0.396422, re_ci_high = 0.020136]. These are identical to the fixed-effect values because τ² = 0.
• Heterogeneity: q_stat = 0.323686 on q_df = 1 (q_pvalue = 0.569401), i2 = 0, tau2 = 0, so heterogeneity_interpretation = "low".
• As a percentage change in time: pooled_pct_change_re = −17.15%, with a CI of −32.7% to +2.0% (pooled_pct_change_re_ci).
• Significance: pooled_significant_5pct = false. The random-effects interval includes 0.
• Forest plot: out/figure1.png shows the four harmonised estimates. S2 and S3 appear as points without intervals.

Synthesis conclusion. The two poolable studies (one enterprise coding task, one phishing-triage task) point to roughly 17% shorter completion time with AI. However, the 95% interval includes no effect, so the pooled evidence does not establish a time saving at the 5% level. The two harmonised studies that could not be pooled disagree in direction:

• S3 (IT administrators) shows about 30% less time, reported as highly significant by stars only (θ = −0.354).
• S2 (experienced open-source developers on their own repositories) shows 19% more time (θ = +0.174).

The available evidence is therefore mixed and depends on the setting. It does not support a general claim that generative AI shortens task time.

# Uncertainty

• Number of studies: with k = 2, the DerSimonian–Laird τ² is essentially unidentified. Q = 0.32 on 1 df cannot detect heterogeneity, so "low" heterogeneity is a statement about these two estimates only. The between-study disagreement visible among the four harmonised studies (signs differ) is not captured by I².
• S4's SE: it is back-calculated from a p-value of 0.471 and is approximate. The sidedness of the test is not stated, and the p-value may come from a test on raw time rather than log time.
• S1: its interval is wide on its own (p = 0.086).
• Sensitivities:
  • Using the CI-implied SE for S1 gives FE −0.189107 with CI [−0.395442, 0.017228] (sensitivity_fe_s1_se_from_ci).
  • Computing S4's θ from the ratio of printed times instead of the printed % gives −0.189675 with CI [−0.399187, 0.019838] (sensitivity_fe_s4_theta_from_time_ratio).
  • Neither changes the conclusion.

# Limitations and open issues

• What was pooled: only 2 of 5 studies could be pooled. Leaving out S3 (a large, stars-only effect in favour of AI) and S2 (a slowdown) is driven by how the results were reported, not by study quality. The pooled estimate is therefore not a summary of the whole evidence base, and adding either study could move it in opposite directions.
• Comparability: the estimands are not fully the same.
  • S1 and S2 are regression-adjusted log-time contrasts.
  • S3 holds accuracy constant (the control's time is predicted at the treatment group's accuracy), and 25% of its control subjects used other AI tools.
  • S4's Aware arm also changes the order of the queue, and its time is total queue time.
  • The tasks, populations and tools differ a lot (enterprise coding, open-source maintenance, IT administration, phishing triage).
• S5: this study is about output after access was withdrawn (a ban), not task time. It is reported but excluded.
• No missing values substituted: no value outside the extracts was used. The S2 confidence interval and S3's exact uncertainty exist in the full papers' figures or bootstrap results, but they are not admissible here. Those SEs are left null rather than guessed.
• Feasibility: the project is feasibility = "complete" in the TASK sense. Every item has either harmonised values or a justified reason, and pooling was carried out over the poolable items. The substantive conclusion is limited, as described above.