# T3-synthesis — Does generative-AI assistance shorten task-completion time? A small evidence synthesis

# Question

Does access to a generative-AI assistant change the time workers need to complete their tasks? Target quantity (target_definition): θ = ln(T_AI / T_noAI), the log of the ratio of mean completion time with AI to mean completion time without AI, same tasks, randomised comparison (negative = faster with AI). Five studies (S1–S5) are assessed (n_analysed = 5), using only the provided arXiv extracts.

# Data and exclusions

Inputs: data/studies.json (5 items) and five text extracts (line counts in n_input_rows); every SHA-256 matches data/README.md (see out/state/data_audit.md). All five requested quantities were found as printed (k_extracted = 5). The full extraction table, with printed values, locators and transformations, is out/table1.csv (= items in results.json).

Exclusion ledger (out/exclusions.csv): 5 studies → 4 harmonisable (S5 dropped) → 2 poolable (S2 and S3 dropped), so k_harmonised = 4 and k_pooled = 2 (pooled_studies = S1, S4).

• S5 (Kreitmeir & Raschky, ChatGPT ban): Treat×Post on Output = -0.278 (0.295), N 18332 (items.S5). Not harmonisable: the outcome is counts of GitHub output events, the design is a non-randomised difference-in-differences around the withdrawal of access, and no task-completion time is measured.
• S2 (Becker et al., METR): +19% completion time (items.S2). Harmonisable, but not poolable, because the confidence interval appears only in Figure 1, which is not reproduced in the extract. No remembered value was used in its place.
• S3 (Bono & Xu, Security Copilot): -29.79%*** (items.S3). Harmonisable, but not poolable, because significance stars are the only uncertainty information given.

Problems found in the inputs are listed in data_issues. Examples: S1 prints mean log time-on-task values that look swapped (4.6 for AI vs 4.3 for no-AI, yet AI is described as faster), and its sample is given as n = 96 in Table III but 47 + 46 = 93 in the text. In S3 and S4 the printed percentages differ slightly from the ratios of the printed times. In S4 the per-class time columns cannot be reconciled with the full-corpus times. S5's significance key contains a typo.

# Method

How each study was put on the θ scale (the formula used is in each items.*.transformation):

• S1: log-time regression coefficient, θ = b, SE = SE(b) as printed.
• S2 and S3: printed percentage change, θ = ln(1 + c).
• S4: θ = ln(1 − 0.111). Its SE is approximate, recovered from the printed two-sided p = 0.471 as SE = |θ| / z₍₁₋ₚ/₂₎.

When a printed percentage and a ratio of printed times were both available (S3, S4), I used the printed percentage, following the TASK convention.

Pooling covers the poolable studies only and is done on the θ scale:

• inverse-variance fixed effect;
• DerSimonian–Laird random effects;
• 95% intervals = estimate ± 1.96·SE;
• Cochran's Q with k − 1 = 1 df, I² and τ² as defined in TASK.md.

As a baseline, I also computed the unweighted mean θ. The analysis has no randomness (random_seed = null). out/analysis.py prints every source line it relies on.

# Results

Harmonised study estimates:

Pooled results over S1 and S4:

• Fixed effect: θ = -0.1881, SE 0.1063, 95% CI [-0.3964, 0.0201] (fe_estimate, fe_se, fe_ci_low, fe_ci_high). Weights are 57.6% S1 and 42.4% S4 (fe_weights_pct).
• Heterogeneity: Q = 0.324 (q_stat), I² = 0% (i2), τ² = 0 (tau2), labelled "low" (heterogeneity_interpretation).
• Random effects: because τ² = 0, the random-effects result equals the fixed-effect one: θ = -0.1881, SE 0.1063, 95% CI [-0.3964, 0.0201] (re_estimate, re_se, re_ci_low, re_ci_high).
• In percentage terms this is a -17.2% change in mean completion time (pooled_pct_change_re). The interval includes 0, so the result is not statistically significant at 5% (pooled_significant_5pct = false).

Baseline unweighted mean θ: -0.1788 over the two poolable studies (baseline_unweighted_mean_theta_poolable) and -0.1343 over the four harmonisable studies (baseline_unweighted_mean_theta_harmonised).

The forest plot is out/figure1.png. The project is feasibility = "complete": every item has either harmonised values or a justified statement of what cannot be obtained.

Synthesis. The two poolable RCTs point towards shorter task times with AI (a pooled change of about -17%). The pooled 95% interval nevertheless runs from about a 33% reduction to a 2% increase (exp of re_ci_low and re_ci_high), so the data do not establish an effect at the 5% level. The studies that cannot be pooled point in opposite directions:

• S3 shows a large reduction (θ = -0.354), marked p ≤ 0.01 by stars.
• S2 shows a slowdown (θ = +0.174) for experienced open-source developers on real issues.

The evidence is therefore inconclusive and context-dependent. It does not support a general claim that generative AI shortens task time.

# Uncertainty

• Sampling uncertainty: the pooled SE is 0.1063 (re_se). The S4 SE is approximate (derived from a p-value, assuming a Wald-type test on the log-ratio scale).
• Heterogeneity cannot be estimated meaningfully from k = 2 (Q has 1 df). I² = 0 and τ² = 0 reflect low power, not demonstrated homogeneity. With S2 and S3 included, heterogeneity across all harmonisable studies is plainly large (θ ranges from -0.354 to +0.174), but it cannot be quantified without their SEs.
• Sensitivity checks barely move the pooled estimate. Using the CI-derived SE for S1 gives a pooled θ of -0.1891 (sensitivity_fe_with_S1_ci_se). Using the S4 ratio of printed times instead of the printed percentage gives -0.1897 (sensitivity_fe_with_S4_from_times).

# Limitations and open issues

• Only 2 of 5 studies could be pooled (k_pooled). The pooled number rests on one coding task at Google and one phishing-triage exercise. Neither is representative, and the random-effects model is essentially unidentified at k = 2.
• Leaving out S2 and S3 is a matter of missing uncertainty information, not study quality, so the pooled estimate is selected by reporting format. Adding S3 (large speed-up) or S2 (slowdown) would move it in opposite directions.
• The constructs are not identical:
  • S1 is adjusted for covariates.
  • S2 is a ratio conditional on forecasts.
  • S3 holds accuracy constant using a model-predicted control duration.
  • S4's AI arm also includes agent-prioritised queue ordering, and its p-value-based SE is approximate.
  • The populations (developers, IT administrators, security analysts) and AI tools differ.
• Rendering and internal inconsistencies are documented in data_issues. None of them changes the harmonised values beyond rounding.
• Process note: during the work a shell working-directory slip briefly created a scratch out/ folder inside data/. It was moved out immediately, and all data/ hashes were re-verified as unchanged (out/state/log.md).