1# T3-synthesis — Does generative-AI assistance shorten task-completion time? A small evidence synthesis23## Question4Does access to a generative-AI assistant change the time workers need to complete their tasks? Target quantity (`target_definition`): θ = ln(T_AI / T_noAI), the log of the ratio of mean completion time with AI to mean completion time without AI, same tasks, randomised comparison (negative = faster with AI). Five studies (S1–S5) are assessed (`n_analysed` = 5), using only the provided arXiv extracts.56## Data and exclusions7Inputs: `data/studies.json` (5 items) and five text extracts (line counts in `n_input_rows`); every SHA-256 matches `data/README.md` (see `out/state/data_audit.md`). All five requested quantities were found as printed (`k_extracted` = 5). The full extraction table, with printed values, locators and transformations, is `out/table1.csv` (= `items` in `results.json`).89Exclusion ledger (`out/exclusions.csv`): 5 studies → 4 harmonisable (S5 dropped) → 2 poolable (S2 and S3 dropped), so `k_harmonised` = 4 and `k_pooled` = 2 (`pooled_studies` = S1, S4).10- **S5** (Kreitmeir & Raschky, ChatGPT ban): Treat×Post on Output = -0.278 (0.295), N 18332 (`items.S5`). **Not harmonisable**: the outcome is counts of GitHub output events, the design is a non-randomised difference-in-differences around the withdrawal of access, and no task-completion time is measured.11- **S2** (Becker et al., METR): +19% completion time (`items.S2`). Harmonisable, but **not poolable**, because the confidence interval appears only in Figure 1, which is not reproduced in the extract. No remembered value was used in its place.12- **S3** (Bono & Xu, Security Copilot): -29.79%*** (`items.S3`). Harmonisable, but **not poolable**, because significance stars are the only uncertainty information given.1314Problems found in the inputs are listed in `data_issues`. Examples: S1 prints mean log time-on-task values that look swapped (4.6 for AI vs 4.3 for no-AI, yet AI is described as faster), and its sample is given as n = 96 in Table III but 47 + 46 = 93 in the text. In S3 and S4 the printed percentages differ slightly from the ratios of the printed times. In S4 the per-class time columns cannot be reconciled with the full-corpus times. S5's significance key contains a typo.1516## Method17How each study was put on the θ scale (the formula used is in each `items.*.transformation`):18- S1: log-time regression coefficient, θ = b, SE = SE(b) as printed.19- S2 and S3: printed percentage change, θ = ln(1 + c).20- S4: θ = ln(1 − 0.111). Its SE is **approximate**, recovered from the printed two-sided p = 0.471 as SE = |θ| / z₍₁₋ₚ/₂₎.2122When a printed percentage and a ratio of printed times were both available (S3, S4), I used the printed percentage, following the TASK convention.2324Pooling covers the poolable studies only and is done on the θ scale:25- inverse-variance fixed effect;26- DerSimonian–Laird random effects;27- 95% intervals = estimate ± 1.96·SE;28- Cochran's Q with k − 1 = 1 df, I² and τ² as defined in TASK.md.2930As a baseline, I also computed the unweighted mean θ. The analysis has no randomness (`random_seed` = null). `out/analysis.py` prints every source line it relies on.3132## Results33Harmonised study estimates:34| study | θ | SE | poolable |35|---|---|---|---|36| S1 Paradis et al. (Google RCT, coding task) | -0.24 (`items.S1.harmonised_estimate`) | 0.14 (`items.S1.harmonised_se`) | yes |37| S2 Becker et al. (METR RCT, OSS issues) | 0.1740 (`items.S2.harmonised_estimate`) | — | no |38| S3 Bono & Xu (Copilot RCT, IT admin) | -0.3537 (`items.S3.harmonised_estimate`) | — | no |39| S4 Bono (phishing-triage RCT) | -0.1177 (`items.S4.harmonised_estimate`) | 0.1632, approximate (`items.S4.harmonised_se`) | yes |40| S5 Kreitmeir & Raschky (DiD) | not harmonisable | — | no |4142Pooled results over S1 and S4:43- Fixed effect: θ = -0.1881, SE 0.1063, 95% CI [-0.3964, 0.0201] (`fe_estimate`, `fe_se`, `fe_ci_low`, `fe_ci_high`). Weights are 57.6% S1 and 42.4% S4 (`fe_weights_pct`).44- Heterogeneity: Q = 0.324 (`q_stat`), I² = 0% (`i2`), τ² = 0 (`tau2`), labelled "low" (`heterogeneity_interpretation`).45- Random effects: because τ² = 0, the random-effects result equals the fixed-effect one: θ = -0.1881, SE 0.1063, 95% CI [-0.3964, 0.0201] (`re_estimate`, `re_se`, `re_ci_low`, `re_ci_high`).46- In percentage terms this is a -17.2% change in mean completion time (`pooled_pct_change_re`). The interval includes 0, so the result is **not statistically significant at 5%** (`pooled_significant_5pct` = false).4748Baseline unweighted mean θ: -0.1788 over the two poolable studies (`baseline_unweighted_mean_theta_poolable`) and -0.1343 over the four harmonisable studies (`baseline_unweighted_mean_theta_harmonised`).4950The forest plot is `out/figure1.png`. The project is `feasibility` = "complete": every item has either harmonised values or a justified statement of what cannot be obtained.5152**Synthesis.** The two poolable RCTs point towards shorter task times with AI (a pooled change of about -17%). The pooled 95% interval nevertheless runs from about a 33% reduction to a 2% increase (exp of `re_ci_low` and `re_ci_high`), so the data do not establish an effect at the 5% level. The studies that cannot be pooled point in opposite directions:53- S3 shows a large reduction (θ = -0.354), marked p ≤ 0.01 by stars.54- S2 shows a slowdown (θ = +0.174) for experienced open-source developers on real issues.5556The evidence is therefore inconclusive and context-dependent. It does not support a general claim that generative AI shortens task time.5758## Uncertainty59- Sampling uncertainty: the pooled SE is 0.1063 (`re_se`). The S4 SE is approximate (derived from a p-value, assuming a Wald-type test on the log-ratio scale).60- Heterogeneity cannot be estimated meaningfully from k = 2 (Q has 1 df). I² = 0 and τ² = 0 reflect low power, not demonstrated homogeneity. With S2 and S3 included, heterogeneity across all harmonisable studies is plainly large (θ ranges from -0.354 to +0.174), but it cannot be quantified without their SEs.61- Sensitivity checks barely move the pooled estimate. Using the CI-derived SE for S1 gives a pooled θ of -0.1891 (`sensitivity_fe_with_S1_ci_se`). Using the S4 ratio of printed times instead of the printed percentage gives -0.1897 (`sensitivity_fe_with_S4_from_times`).6263## Limitations and open issues64- Only 2 of 5 studies could be pooled (`k_pooled`). The pooled number rests on one coding task at Google and one phishing-triage exercise. Neither is representative, and the random-effects model is essentially unidentified at k = 2.65- Leaving out S2 and S3 is a matter of missing uncertainty information, not study quality, so the pooled estimate is selected by reporting format. Adding S3 (large speed-up) or S2 (slowdown) would move it in opposite directions.66- The constructs are not identical:67 - S1 is adjusted for covariates.68 - S2 is a ratio conditional on forecasts.69 - S3 holds accuracy constant using a model-predicted control duration.70 - S4's AI arm also includes agent-prioritised queue ordering, and its p-value-based SE is approximate.71 - The populations (developers, IT administrators, security analysts) and AI tools differ.72- Rendering and internal inconsistencies are documented in `data_issues`. None of them changes the harmonised values beyond rounding.73- Process note: during the work a shell working-directory slip briefly created a scratch `out/` folder inside `data/`. It was moved out immediately, and all `data/` hashes were re-verified as unchanged (`out/state/log.md`).74