# T3-synthesis: Does generative-AI assistance shorten the time workers need to finish their tasks?

# Question

Five primary studies report the effect of access to a generative-AI assistant on worker performance. Each uses a different effect definition. The target is θ = ln(T_AI / T_noAI): the log ratio of mean task-completion time with AI to mean time without AI, on the same tasks, from a randomised comparison. Negative θ means workers were faster with AI. Steps: extract each printed value with a locator, harmonise it to θ, pool the poolable studies (inverse-variance fixed effect and DerSimonian–Laird random effects), and state what the result can support.

# Data and exclusions

The only evidence is the five extracts in data/sources/ plus data/studies.json. Extracted values, exactly as printed (full detail in out/table1.csv):

Exclusions:

• S5 is not harmonisable. It is a difference-in-differences effect of a national ban on counts of GitHub events. It does not compare completion times on the same tasks with and without AI, and it is not randomised.
• S2 and S3 are harmonisable but not poolable. S2 prints no SE, CI or p-value for its 19% estimate; the CI appears only in a figure that is not reproduced. S3 prints only significance stars.

Every file read is counted in n_input_rows.

# Method

Harmonisation followed the conventions in TASK.md (the exact formulas with printed inputs are in the transformation field):

• S1: θ = b = −0.24 and SE = 0.14, as printed in Table III.
• S2 and S3: θ = ln(1 + c), with c = 0.19 and c = −0.2979.
• S4: c = −0.111, because a positive "time saving" means less time with AI. θ = ln(0.889) = −0.118. The SE is approximate: SE = |θ| / z₁₋ₚ/₂ with p = 0.471, which gives 0.163.

Pooling: inverse-variance fixed effect and DerSimonian–Laird random effects over S1 and S4, with 95% intervals (±1.96 SE). I computed Q (df = 1), I² and τ² as defined in the task. out/analysis.py regenerates every output and prints the source lines it uses. It also checks each value against the text of the extract.

# Results

• Fixed effect: θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020].
• Random effects: identical, because τ² = 0. θ = −0.188, 95% CI [−0.396, 0.020]. This corresponds to about −17.2% completion time with AI.
• Heterogeneity: Q = 0.324 (df = 1), I² = 0%, τ² = 0, classed as "low".
• The random-effects interval includes 0, so the pooled effect is not statistically significant at 5%.
• Figure: out/figure1.png.

# Uncertainty

• Very few studies. The pooled estimate rests on k = 2. With df = 1, Q has almost no power to detect heterogeneity, so "I² = 0, low" is not evidence that the studies agree. The DL τ² cannot be estimated meaningfully.
• S4's SE is approximate. It is recovered from a p-value whose underlying test is not described in the extract (probably a test on raw seconds, not logs).
• S1's SE is rounded. The printed SE (0.14) is rounded to two decimals. Using the SE implied by the CI (0.1378) changes the fixed-effect estimate only slightly, to −0.189.
• Both pooled studies have wide intervals that individually include 0.

# Limitations and open issues

• The two pooled studies are not representative of the evidence. The two harmonisable studies left out point in opposite directions: S2 shows a slowdown (θ = +0.174) and S3 a large speed-up (θ = −0.354). Without SEs they cannot be weighted, so the pooled number leaves out half the harmonisable evidence. It also ignores what is probably real heterogeneity between settings: experienced open-source developers on their own repositories versus lab or crowd-sourced tasks.
• The study designs differ. Estimands differ in subtle ways:
  • S1 and S2 adjust for covariates.
  • S3 holds accuracy constant through a regression on control-group accuracy.
  • S4's Aware arm bundles agent verdicts with queue reordering.
  • Populations and tasks also differ (developers, IT administrators, security analysts).
• What the numbers support: a point estimate of roughly a 17% reduction in completion time across the two poolable trials, with an interval running from about a 33% reduction to a 2% increase.
• What they do not support: a firm conclusion that generative AI shortens completion time, a precise effect size, or any claim about how consistent the effect is across settings.
• Data issues (full list in results.json → data_issues):
  • S1's mean LogToT values contradict its own raw means.
  • S1 reports n = 96 in Table III but 93 in the text.
  • S3's printed −29.79% differs slightly from the ratio of its printed means (−29.81%).
  • S4's per-class column units look wrong.
  • S5 prints its significance key as "p<0.5".
• Feasibility: complete. Every item has either harmonised values or an explicit, justified reason why they cannot be obtained, and pooling covers all poolable items.