# T3-synthesis: Does generative-AI assistance shorten the time workers need to complete their tasks?

# Question

Does access to a generative-AI assistant change how long workers take to complete their tasks? The target effect is θ = ln(T_AI / T_noAI): the natural log of the ratio of mean task-completion time with AI to mean time without it, on the same tasks, from a randomised comparison. Negative θ means faster with AI. The evidence comes from five study extracts (data/studies.json, data/sources/*.txt). Only numbers printed in those extracts are used.

# Data and exclusions

I extracted all five items (k_extracted = 5); out/table1.csv has the full extraction table with locators.

Exclusions:

• S5 is not harmonisable. Its outcome is a count of GitHub output events per user-day, not the time needed to complete the same tasks. The design is a country-level difference-in-differences around the withdrawal of access, not a randomised AI-vs-no-AI comparison.
• S2 is harmonisable but not poolable. The extract prints no SE, CI or p-value for the 19% estimate; the CI appears only in Figure 1, which is not reproduced.
• S3 is harmonisable but not poolable. Its only uncertainty information is significance stars.

# Method

Harmonisation followed the TASK conventions:

• S1: θ = b and SE = SE(b), using the printed SE of 0.14. The SE implied by the printed CI, 0.54/3.92 = 0.138, is used as a sensitivity check.
• S2: θ = ln(1 + 0.19).
• S3: θ = ln(1 − 0.2979). The cross-check ln(44.29/63.10) = −0.354 agrees.
• S4: the printed 11.1% saving means a time change of c = −0.111, so θ = ln(0.889). The SE is approximate, recovered from the two-sided p-value as |θ| / z₀.₇₆₄₅ = 0.1177 / 0.7206 = 0.163.

Pooling covers only the poolable studies (S1 and S4, so k = 2):

• inverse-variance fixed effect;
• DerSimonian–Laird random effects;
• Cochran's Q with df = 1;
• I² = max(0, (Q − df)/Q) × 100;
• τ² = max(0, (Q − df)/C);
• 95% intervals using 1.96.

out/analysis.py regenerates every output from data/ and prints the source lines it uses. No randomness is involved.

# Results

• Fixed effect: θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020].
• Random effects (DL): θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020]. This equals the fixed-effect result because τ² = 0.
  • As a percentage change in time: 100 × (exp(θ) − 1) = −17.2%.
  • The 95% interval includes 0, so the pooled effect is not statistically significant at 5% (pooled_significant_5pct = false).
• Heterogeneity: Q = 0.324 (df = 1), I² = 0%, τ² = 0, which counts as "low" under the stated rule.
• Sensitivity: using the CI-derived SE for S1 (0.138) gives θ = −0.189 (SE 0.105) and Q = 0.328. The conclusions do not change.

Studies that could not be pooled (Figure 1 shows them as points without intervals):

• S3 points to a large speed-up: θ = −0.354, i.e. −29.8% time.
• S2 points to a slowdown: θ = +0.174, i.e. +19% time.

What the synthesis supports: across the two poolable RCTs, the point estimate is that AI access shortens task time by about 17%. This is compatible with no effect, and the 95% interval runs from about a 33% reduction to a 2% increase (exp of the CI bounds).

What it cannot support:

• A general claim that generative AI shortens task time. The pool contains only two small studies, of a single coding task and of email triage. One of them (S4) has only an approximate SE, back-calculated from a non-significant p = 0.471.
• Reliable heterogeneity statistics. "Low" heterogeneity (I² = 0, τ² = 0) is uninformative with k = 2. Q has one degree of freedom, and DL τ² is essentially unidentifiable.
• Agreement among all studies. The printed point estimates of the studies left out span opposite directions (S3: −30%; S2: +19%; S2 is the setting with experienced developers on their own repositories). Excluding them is a matter of missing uncertainty information, not irrelevance. A pool that could include them might differ in sign of conclusion and would very likely show substantial heterogeneity, though this cannot be quantified from the extracts.
• Comparability of outcomes. The pooled studies measure different outcomes and tasks: S1 is a covariate-adjusted log-time regression; S4 is the difference in mean time per email between arms that also differ in queue ordering.

# Uncertainty

• The pooled 95% interval is [−0.396, 0.020] on the θ scale.
• S4's SE is approximate. It assumes a normal test statistic and that the printed p-value refers to the same contrast as the 11.1% figure.
• The p-value is rounded to 3 decimals and the % saving to 1 decimal. Back-calculation from an estimate near zero is imprecise.
• The S1 SE is rounded to 2 decimals; the sensitivity check above covers this.
• The interval does not reflect between-study uncertainty beyond DL τ² = 0, so with k = 2 it understates the true uncertainty about a population-average effect.

# Limitations and open issues

• Too few studies. Only 2 of 5 studies can be pooled. The pooled estimate is not a summary of the evidence base. It is a summary of the two studies that report usable uncertainty in the provided extracts.

• Values not used. The extracts' rules forbid substituting values from figures or the full papers. For example, S2's Figure 1 CI was not used.

• Data issues (detailed in results.json → data_issues):

  • S1 prints mean LogToT 4.6 (AI) vs 4.3 (no-AI), which contradicts the printed mean times of 96 vs 114 min. These means were not used.
  • S1 has n = 96 in Table III vs 47 + 46 = 93 in the text.
  • S1's Table III has collapsed columns with significance markers fused to the numbers.
  • S3's "Improvement" column is a signed % change, and its star convention is non-standard.
  • S4's Table 3 units are internally inconsistent: Malicious and Non-Malicious "seconds" around 2 vs Full Corpus around 50. Only the unit-free Full Corpus contrast was used.
  • S5's footnote has a typo ("** p<0.5").

• Different estimands. The studies differ in estimand details:

  • S1: covariate-adjusted, so the target is effectively the conditional ratio of geometric means.
  • S2: regression-adjusted ratio of means.
  • S3: durations holding accuracy constant, which is model-based.
  • S4: per-email time across arms that differ in more than AI verdicts.

  θ is therefore only approximately the same quantity across studies.

• Feasibility. The project is complete in the TASK's sense: every item has either harmonised values or a justified statement of what cannot be obtained, and pooling was done over the poolable items.