# T3-synthesis: Does generative-AI assistance shorten the time workers need to complete their tasks?

## Question
Does access to a generative-AI assistant change how long workers take to complete their tasks? The target effect is θ = ln(T_AI / T_noAI): the natural log of the ratio of mean task-completion time with AI to mean time without it, on the same tasks, from a randomised comparison. Negative θ means faster with AI. The evidence comes from five study extracts (`data/studies.json`, `data/sources/*.txt`). Only numbers printed in those extracts are used.

## Data and exclusions
I extracted all five items (k_extracted = 5); `out/table1.csv` has the full extraction table with locators.

| Study | Printed value (as printed) | Printed uncertainty | θ | SE(θ) | Pooled? |
|---|---|---|---|---|---|
| S1 Paradis et al. 2024 (Google RCT, 96 devs) | ExpCon, Model 2, Table III: -0.24 (on log time on task) | SE (0.14); 95%CI = [-0.51,0.03], p=0.086 | -0.240 | 0.140 | yes |
| S2 Becker et al. 2025 (METR RCT, 16 devs, 246 issues) | "19% more time" (regression-based) | none printed (CI shown only in Figure 1) | 0.174 | – | **no**: no SE |
| S3 Bono & Xu 2024 (Security Copilot RCT, IT admins, 181 subjects) | All Content: Treatment 44.29, Control 63.10 min; -29.79%*** | stars only (*** p≤0.01) | -0.354 | – | **no**: stars only |
| S4 Bono 2025 (phishing-triage agent RCT; Aware N=56 vs Control N=52) | Full Corpus: 50.5 vs 56.9 s per email; % Time Savings 11.1 | p-value 0.471 | -0.118 | 0.163 (approx.) | yes, with approximate SE |
| S5 Kreitmeir & Raschky 2023 (ChatGPT ban DiD, GitHub) | Treat×Post, Output, col. 3, panel A: -0.278 | (0.295), N = 18332 | – | – | **no**: not harmonisable |

Exclusions:
- **S5 is not harmonisable.** Its outcome is a count of GitHub output events per user-day, not the time needed to complete the same tasks. The design is a country-level difference-in-differences around the withdrawal of access, not a randomised AI-vs-no-AI comparison.
- **S2 is harmonisable but not poolable.** The extract prints no SE, CI or p-value for the 19% estimate; the CI appears only in Figure 1, which is not reproduced.
- **S3 is harmonisable but not poolable.** Its only uncertainty information is significance stars.

## Method
Harmonisation followed the TASK conventions:
- **S1:** θ = b and SE = SE(b), using the printed SE of 0.14. The SE implied by the printed CI, 0.54/3.92 = 0.138, is used as a sensitivity check.
- **S2:** θ = ln(1 + 0.19).
- **S3:** θ = ln(1 − 0.2979). The cross-check ln(44.29/63.10) = −0.354 agrees.
- **S4:** the printed 11.1% saving means a time change of c = −0.111, so θ = ln(0.889). The SE is approximate, recovered from the two-sided p-value as |θ| / z₀.₇₆₄₅ = 0.1177 / 0.7206 = 0.163.

Pooling covers only the poolable studies (S1 and S4, so k = 2):
- inverse-variance fixed effect;
- DerSimonian–Laird random effects;
- Cochran's Q with df = 1;
- I² = max(0, (Q − df)/Q) × 100;
- τ² = max(0, (Q − df)/C);
- 95% intervals using 1.96.

`out/analysis.py` regenerates every output from `data/` and prints the source lines it uses. No randomness is involved.

## Results
- **Fixed effect:** θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020].
- **Random effects (DL):** θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020]. This equals the fixed-effect result because τ² = 0.
  - As a percentage change in time: 100 × (exp(θ) − 1) = **−17.2%**.
  - The 95% interval includes 0, so the pooled effect is **not statistically significant at 5%** (`pooled_significant_5pct` = false).
- **Heterogeneity:** Q = 0.324 (df = 1), I² = 0%, τ² = 0, which counts as "low" under the stated rule.
- **Sensitivity:** using the CI-derived SE for S1 (0.138) gives θ = −0.189 (SE 0.105) and Q = 0.328. The conclusions do not change.

Studies that could not be pooled (Figure 1 shows them as points without intervals):
- S3 points to a large speed-up: θ = −0.354, i.e. −29.8% time.
- S2 points to a slowdown: θ = +0.174, i.e. +19% time.

**What the synthesis supports:** across the two poolable RCTs, the point estimate is that AI access shortens task time by about 17%. This is compatible with no effect, and the 95% interval runs from about a 33% reduction to a 2% increase (exp of the CI bounds).

**What it cannot support:**
- **A general claim that generative AI shortens task time.** The pool contains only two small studies, of a single coding task and of email triage. One of them (S4) has only an approximate SE, back-calculated from a non-significant p = 0.471.
- **Reliable heterogeneity statistics.** "Low" heterogeneity (I² = 0, τ² = 0) is uninformative with k = 2. Q has one degree of freedom, and DL τ² is essentially unidentifiable.
- **Agreement among all studies.** The printed point estimates of the studies left out span opposite directions (S3: −30%; S2: +19%; S2 is the setting with experienced developers on their own repositories). Excluding them is a matter of missing uncertainty information, not irrelevance. A pool that could include them might differ in sign of conclusion and would very likely show substantial heterogeneity, though this cannot be quantified from the extracts.
- **Comparability of outcomes.** The pooled studies measure different outcomes and tasks: S1 is a covariate-adjusted log-time regression; S4 is the difference in mean time per email between arms that also differ in queue ordering.

## Uncertainty
- The pooled 95% interval is [−0.396, 0.020] on the θ scale.
- S4's SE is approximate. It assumes a normal test statistic and that the printed p-value refers to the same contrast as the 11.1% figure.
- The p-value is rounded to 3 decimals and the % saving to 1 decimal. Back-calculation from an estimate near zero is imprecise.
- The S1 SE is rounded to 2 decimals; the sensitivity check above covers this.
- The interval does not reflect between-study uncertainty beyond DL τ² = 0, so with k = 2 it understates the true uncertainty about a population-average effect.

## Limitations and open issues
- **Too few studies.** Only 2 of 5 studies can be pooled. The pooled estimate is not a summary of the evidence base. It is a summary of the two studies that report usable uncertainty in the provided extracts.
- **Values not used.** The extracts' rules forbid substituting values from figures or the full papers. For example, S2's Figure 1 CI was not used.
- **Data issues** (detailed in `results.json` → `data_issues`):
  - S1 prints mean LogToT 4.6 (AI) vs 4.3 (no-AI), which contradicts the printed mean times of 96 vs 114 min. These means were not used.
  - S1 has n = 96 in Table III vs 47 + 46 = 93 in the text.
  - S1's Table III has collapsed columns with significance markers fused to the numbers.
  - S3's "Improvement" column is a signed % change, and its star convention is non-standard.
  - S4's Table 3 units are internally inconsistent: Malicious and Non-Malicious "seconds" around 2 vs Full Corpus around 50. Only the unit-free Full Corpus contrast was used.
  - S5's footnote has a typo ("** p<0.5").
- **Different estimands.** The studies differ in estimand details:
  - S1: covariate-adjusted, so the target is effectively the conditional ratio of geometric means.
  - S2: regression-adjusted ratio of means.
  - S3: durations holding accuracy constant, which is model-based.
  - S4: per-email time across arms that differ in more than AI verdicts.

  θ is therefore only approximately the same quantity across studies.
- **Feasibility.** The project is complete in the TASK's sense: every item has either harmonised values or a justified statement of what cannot be obtained, and pooling was done over the poolable items.