1# T3-synthesis: Does generative-AI assistance shorten the time workers need to complete their tasks?23## Question4Does access to a generative-AI assistant change how long workers take to complete their tasks? The target effect is θ = ln(T_AI / T_noAI): the natural log of the ratio of mean task-completion time with AI to mean time without it, on the same tasks, from a randomised comparison. Negative θ means faster with AI. The evidence comes from five study extracts (`data/studies.json`, `data/sources/*.txt`). Only numbers printed in those extracts are used.56## Data and exclusions7I extracted all five items (k_extracted = 5); `out/table1.csv` has the full extraction table with locators.89| Study | Printed value (as printed) | Printed uncertainty | θ | SE(θ) | Pooled? |10|---|---|---|---|---|---|11| S1 Paradis et al. 2024 (Google RCT, 96 devs) | ExpCon, Model 2, Table III: -0.24 (on log time on task) | SE (0.14); 95%CI = [-0.51,0.03], p=0.086 | -0.240 | 0.140 | yes |12| S2 Becker et al. 2025 (METR RCT, 16 devs, 246 issues) | "19% more time" (regression-based) | none printed (CI shown only in Figure 1) | 0.174 | – | **no**: no SE |13| S3 Bono & Xu 2024 (Security Copilot RCT, IT admins, 181 subjects) | All Content: Treatment 44.29, Control 63.10 min; -29.79%*** | stars only (*** p≤0.01) | -0.354 | – | **no**: stars only |14| S4 Bono 2025 (phishing-triage agent RCT; Aware N=56 vs Control N=52) | Full Corpus: 50.5 vs 56.9 s per email; % Time Savings 11.1 | p-value 0.471 | -0.118 | 0.163 (approx.) | yes, with approximate SE |15| S5 Kreitmeir & Raschky 2023 (ChatGPT ban DiD, GitHub) | Treat×Post, Output, col. 3, panel A: -0.278 | (0.295), N = 18332 | – | – | **no**: not harmonisable |1617Exclusions:18- **S5 is not harmonisable.** Its outcome is a count of GitHub output events per user-day, not the time needed to complete the same tasks. The design is a country-level difference-in-differences around the withdrawal of access, not a randomised AI-vs-no-AI comparison.19- **S2 is harmonisable but not poolable.** The extract prints no SE, CI or p-value for the 19% estimate; the CI appears only in Figure 1, which is not reproduced.20- **S3 is harmonisable but not poolable.** Its only uncertainty information is significance stars.2122## Method23Harmonisation followed the TASK conventions:24- **S1:** θ = b and SE = SE(b), using the printed SE of 0.14. The SE implied by the printed CI, 0.54/3.92 = 0.138, is used as a sensitivity check.25- **S2:** θ = ln(1 + 0.19).26- **S3:** θ = ln(1 − 0.2979). The cross-check ln(44.29/63.10) = −0.354 agrees.27- **S4:** the printed 11.1% saving means a time change of c = −0.111, so θ = ln(0.889). The SE is approximate, recovered from the two-sided p-value as |θ| / z₀.₇₆₄₅ = 0.1177 / 0.7206 = 0.163.2829Pooling covers only the poolable studies (S1 and S4, so k = 2):30- inverse-variance fixed effect;31- DerSimonian–Laird random effects;32- Cochran's Q with df = 1;33- I² = max(0, (Q − df)/Q) × 100;34- τ² = max(0, (Q − df)/C);35- 95% intervals using 1.96.3637`out/analysis.py` regenerates every output from `data/` and prints the source lines it uses. No randomness is involved.3839## Results40- **Fixed effect:** θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020].41- **Random effects (DL):** θ = −0.188 (SE 0.106), 95% CI [−0.396, 0.020]. This equals the fixed-effect result because τ² = 0.42 - As a percentage change in time: 100 × (exp(θ) − 1) = **−17.2%**.43 - The 95% interval includes 0, so the pooled effect is **not statistically significant at 5%** (`pooled_significant_5pct` = false).44- **Heterogeneity:** Q = 0.324 (df = 1), I² = 0%, τ² = 0, which counts as "low" under the stated rule.45- **Sensitivity:** using the CI-derived SE for S1 (0.138) gives θ = −0.189 (SE 0.105) and Q = 0.328. The conclusions do not change.4647Studies that could not be pooled (Figure 1 shows them as points without intervals):48- S3 points to a large speed-up: θ = −0.354, i.e. −29.8% time.49- S2 points to a slowdown: θ = +0.174, i.e. +19% time.5051**What the synthesis supports:** across the two poolable RCTs, the point estimate is that AI access shortens task time by about 17%. This is compatible with no effect, and the 95% interval runs from about a 33% reduction to a 2% increase (exp of the CI bounds).5253**What it cannot support:**54- **A general claim that generative AI shortens task time.** The pool contains only two small studies, of a single coding task and of email triage. One of them (S4) has only an approximate SE, back-calculated from a non-significant p = 0.471.55- **Reliable heterogeneity statistics.** "Low" heterogeneity (I² = 0, τ² = 0) is uninformative with k = 2. Q has one degree of freedom, and DL τ² is essentially unidentifiable.56- **Agreement among all studies.** The printed point estimates of the studies left out span opposite directions (S3: −30%; S2: +19%; S2 is the setting with experienced developers on their own repositories). Excluding them is a matter of missing uncertainty information, not irrelevance. A pool that could include them might differ in sign of conclusion and would very likely show substantial heterogeneity, though this cannot be quantified from the extracts.57- **Comparability of outcomes.** The pooled studies measure different outcomes and tasks: S1 is a covariate-adjusted log-time regression; S4 is the difference in mean time per email between arms that also differ in queue ordering.5859## Uncertainty60- The pooled 95% interval is [−0.396, 0.020] on the θ scale.61- S4's SE is approximate. It assumes a normal test statistic and that the printed p-value refers to the same contrast as the 11.1% figure.62- The p-value is rounded to 3 decimals and the % saving to 1 decimal. Back-calculation from an estimate near zero is imprecise.63- The S1 SE is rounded to 2 decimals; the sensitivity check above covers this.64- The interval does not reflect between-study uncertainty beyond DL τ² = 0, so with k = 2 it understates the true uncertainty about a population-average effect.6566## Limitations and open issues67- **Too few studies.** Only 2 of 5 studies can be pooled. The pooled estimate is not a summary of the evidence base. It is a summary of the two studies that report usable uncertainty in the provided extracts.68- **Values not used.** The extracts' rules forbid substituting values from figures or the full papers. For example, S2's Figure 1 CI was not used.69- **Data issues** (detailed in `results.json` → `data_issues`):70 - S1 prints mean LogToT 4.6 (AI) vs 4.3 (no-AI), which contradicts the printed mean times of 96 vs 114 min. These means were not used.71 - S1 has n = 96 in Table III vs 47 + 46 = 93 in the text.72 - S1's Table III has collapsed columns with significance markers fused to the numbers.73 - S3's "Improvement" column is a signed % change, and its star convention is non-standard.74 - S4's Table 3 units are internally inconsistent: Malicious and Non-Malicious "seconds" around 2 vs Full Corpus around 50. Only the unit-free Full Corpus contrast was used.75 - S5's footnote has a typo ("** p<0.5").76- **Different estimands.** The studies differ in estimand details:77 - S1: covariate-adjusted, so the target is effectively the conditional ratio of geometric means.78 - S2: regression-adjusted ratio of means.79 - S3: durations holding accuracy constant, which is model-based.80 - S4: per-email time across arms that differ in more than AI verdicts.8182 θ is therefore only approximately the same quantity across studies.83- **Feasibility.** The project is complete in the TASK's sense: every item has either harmonised values or a justified statement of what cannot be obtained, and pooling was done over the poolable items.84