# T3-synthesis: Does generative-AI assistance shorten the time workers need to finish their tasks?

## Question
Five primary studies report the effect of access to a generative-AI assistant on worker performance. Each uses a different effect definition. The target is θ = ln(T_AI / T_noAI): the log ratio of mean task-completion time with AI to mean time without AI, on the same tasks, from a randomised comparison. Negative θ means workers were faster with AI. Steps: extract each printed value with a locator, harmonise it to θ, pool the poolable studies (inverse-variance fixed effect and DerSimonian–Laird random effects), and state what the result can support.

## Data and exclusions
The only evidence is the five extracts in `data/sources/` plus `data/studies.json`. Extracted values, exactly as printed (full detail in `out/table1.csv`):

| | Study | Printed estimate | Printed uncertainty | θ | SE(θ) | Pooled? |
|---|---|---|---|---|---|---|
| S1 | Paradis et al. 2024 (Google, n = 96) | ExpCon, Table III Model 2: −0.24 | (0.14); 95% CI [−0.51, 0.03]; p = 0.086 | −0.240 | 0.140 | yes |
| S2 | Becker et al. 2025 (METR, 246 issues) | +19% completion time | none printed (CI only in Figure 1, not reproduced) | +0.174 | null | no |
| S3 | Bono & Xu 2024 (Security Copilot, 181 subjects) | Table 3 All Content: 44.29 vs 63.10 min, −29.79%*** | stars only (***) | −0.354 | null | no |
| S4 | Bono 2025 (phishing triage, Aware 56 / Control 52) | Table 3 Full Corpus: 50.5 vs 56.9 s, 11.1% time savings | p-value 0.471 | −0.118 | 0.163 (approx.) | yes |
| S5 | Kreitmeir & Raschky 2023 (ChatGPT ban, N = 18332) | Treat×Post, Output, col. 3, panel A: −0.278 | (0.295) | null | null | no (not harmonisable) |

Exclusions:
- **S5 is not harmonisable.** It is a difference-in-differences effect of a national ban on counts of GitHub events. It does not compare completion times on the same tasks with and without AI, and it is not randomised.
- **S2 and S3 are harmonisable but not poolable.** S2 prints no SE, CI or p-value for its 19% estimate; the CI appears only in a figure that is not reproduced. S3 prints only significance stars.

Every file read is counted in `n_input_rows`.

## Method
Harmonisation followed the conventions in TASK.md (the exact formulas with printed inputs are in the `transformation` field):
- **S1:** θ = b = −0.24 and SE = 0.14, as printed in Table III.
- **S2 and S3:** θ = ln(1 + c), with c = 0.19 and c = −0.2979.
- **S4:** c = −0.111, because a positive "time saving" means less time with AI. θ = ln(0.889) = −0.118. The SE is approximate: SE = |θ| / z₁₋ₚ/₂ with p = 0.471, which gives 0.163.

Pooling: inverse-variance fixed effect and DerSimonian–Laird random effects over S1 and S4, with 95% intervals (±1.96 SE). I computed Q (df = 1), I² and τ² as defined in the task. `out/analysis.py` regenerates every output and prints the source lines it uses. It also checks each value against the text of the extract.

## Results
- **Fixed effect:** θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020].
- **Random effects:** identical, because τ² = 0. θ = −0.188, 95% CI [−0.396, 0.020]. This corresponds to about **−17.2%** completion time with AI.
- **Heterogeneity:** Q = 0.324 (df = 1), I² = 0%, τ² = 0, classed as "low".
- The random-effects interval includes 0, so **the pooled effect is not statistically significant at 5%**.
- Figure: `out/figure1.png`.

## Uncertainty
- **Very few studies.** The pooled estimate rests on k = 2. With df = 1, Q has almost no power to detect heterogeneity, so "I² = 0, low" is not evidence that the studies agree. The DL τ² cannot be estimated meaningfully.
- **S4's SE is approximate.** It is recovered from a p-value whose underlying test is not described in the extract (probably a test on raw seconds, not logs).
- **S1's SE is rounded.** The printed SE (0.14) is rounded to two decimals. Using the SE implied by the CI (0.1378) changes the fixed-effect estimate only slightly, to −0.189.
- Both pooled studies have wide intervals that individually include 0.

## Limitations and open issues
- **The two pooled studies are not representative of the evidence.** The two harmonisable studies left out point in opposite directions: S2 shows a slowdown (θ = +0.174) and S3 a large speed-up (θ = −0.354). Without SEs they cannot be weighted, so the pooled number leaves out half the harmonisable evidence. It also ignores what is probably real heterogeneity between settings: experienced open-source developers on their own repositories versus lab or crowd-sourced tasks.
- **The study designs differ.** Estimands differ in subtle ways:
  - S1 and S2 adjust for covariates.
  - S3 holds accuracy constant through a regression on control-group accuracy.
  - S4's Aware arm bundles agent verdicts with queue reordering.
  - Populations and tasks also differ (developers, IT administrators, security analysts).
- **What the numbers support:** a point estimate of roughly a 17% reduction in completion time across the two poolable trials, with an interval running from about a 33% reduction to a 2% increase.
- **What they do not support:** a firm conclusion that generative AI shortens completion time, a precise effect size, or any claim about how consistent the effect is across settings.
- **Data issues** (full list in `results.json` → `data_issues`):
  - S1's mean LogToT values contradict its own raw means.
  - S1 reports n = 96 in Table III but 93 in the text.
  - S3's printed −29.79% differs slightly from the ratio of its printed means (−29.81%).
  - S4's per-class column units look wrong.
  - S5 prints its significance key as "p<0.5".
- **Feasibility: complete.** Every item has either harmonised values or an explicit, justified reason why they cannot be obtained, and pooling covers all poolable items.