# T3-synthesis — Does generative-AI assistance shorten the time workers need to complete their tasks? A small evidence synthesis

# Scientific question

Five accessible primary studies report how access to a generative-AI assistant changed the performance of workers (software developers, IT administrators, security analysts) on their tasks. They report their effects under different definitions: a regression coefficient on log time-on-task, a percentage change in completion time, a percentage difference in mean task duration holding accuracy constant, a percentage time saving per processed item with a p-value, and a difference-in-differences coefficient on output counts around the withdrawal of access. This is a complete, bounded evidence-synthesis project: extraction with locators → harmonisation onto one stated definition → pooled estimates with heterogeneity → forest plot → an appropriately limited synthesis. The target definition is θ = ln(T_AI / T_noAI), the natural logarithm of the ratio of mean task-completion time with AI assistance to mean task-completion time without it, for the same tasks, from a randomised comparison (negative = faster with AI). The estimands are: the five extracted values exactly as printed; the harmonised θ and its standard error for every study where both can be obtained from the printed quantities under the conventions below; the inverse-variance fixed-effect and DerSimonian–Laird random-effects pooled θ over the poolable studies, with Q, I² and τ²; and the synthesis conclusion that these numbers support.

# Inputs

• data/studies.json — the five items {"id", "source", "short", "extract"}: extract names the printed quantity to take from each source (which table/row/column or sentence).
• data/sources/arxiv-2410.12944v3.txt — Paradis et al. (2024), How much does AI impact development speed? An enterprise-based randomized controlled trial (Google), CC BY 4.0: abstract, analytic approach, results Sections VI-B/VI-C with Table III (OLS on logged time on task; coefficients with standard errors in parentheses; the four model columns are collapsed into one line per row, see the extract's editorial note).
• data/sources/arxiv-2507.09089v2.txt — Becker et al. (2025), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (METR), CC BY 4.0: abstract, effect-estimation section, opening of the results, Appendix D (regression, confidence intervals, conversion to speedup).
• data/sources/arxiv-2411.01067v2.txt — Bono & Xu (2024), Randomized Controlled Trials for Security Copilot for IT Administrators (Microsoft), CC BY 4.0: abstract, RCT protocol, results on speed with Table 3 (task durations in minutes, holding accuracy constant; significance by asterisks only), Appendix B (how time savings are computed and bootstrapped).
• data/sources/arxiv-2511.13860v1.txt — Bono (2025), Randomized Controlled Trials for Phishing Triage Agent (Microsoft), CC BY 4.0: abstract, methodology (three arms), results Sections 3.1 (true positives per analyst minute, Table 1) and 3.3 (Table 3, time per email with % time savings and p-values), Appendix B.
• data/sources/arxiv-2304.09339v1.txt — Kreitmeir & Raschky (2023), The Unintended Consequences of Censoring Digital Technology – Evidence from Italy's ChatGPT Ban, CC BY 4.0: abstract, outcome definitions (Section 3.1), Section 4 with Table 1 (difference-in-differences coefficients, standard errors in parentheses).
• data/README.md — provenance, retrieval times, licences, SHA-256 of the extracts.

# Information-availability rules

• The extracts are the only admissible evidence. A quantity that is not printed in the cited extract is not available, even if you believe it is in the full paper, in a figure, or elsewhere; do not substitute a remembered or guessed value.
• Harmonisation conventions (state the one you apply in transformation):
  • a coefficient b on an AI indicator in a regression whose dependent variable is the natural log of completion time gives θ = b with SE(θ) = SE(b); a 95% confidence interval [L, U] on that scale gives SE = (U − L)/(2 × 1.96);
  • a percentage change in time c (e.g. "+19%" means c = 0.19, "−29.79%" means c = −0.2979) or a ratio of mean times T_AI / T_noAI gives θ = ln(1 + c) = ln(T_AI / T_noAI);
  • when only a two-sided p-value is printed for an estimate (no SE, no CI), the SE may be recovered as SE(θ) ≈ |θ| / z_{1−p/2} with z from the standard normal; mark the study as poolable with an approximate SE and say so;
  • when the only uncertainty information is significance stars, a p-value threshold, or a confidence interval shown in a figure that is not reproduced, SE(θ) cannot be obtained: the study is harmonisable (θ can be computed) but not poolable (harmonised_se = null, poolable = false, with the reason);
  • a study is not harmonisable (harmonisable = false, harmonised_estimate = null, with the reason) when the printed effect is not a comparison of task-completion times with and without AI access on the same tasks (for example an effect on output counts, or a comparison that is not AI-versus-no-AI).
• Pooling: inverse-variance fixed effect and DerSimonian–Laird random effects over the poolable studies only, on the θ scale, with 95% intervals using 1.96; Q with k − 1 degrees of freedom; I² = max(0, (Q − df)/Q) × 100; τ² = max(0, (Q − df)/C) with C = Σw − Σw²/Σw. heterogeneity_interpretation is "low" if I² < 25, "moderate" if 25 ≤ I² < 50, "substantial" if I² ≥ 50. pooled_significant_5pct is true only if the random-effects 95% interval excludes 0.
• feasibility refers to the project as a whole: the project is complete when every item has either harmonised values or an explicit, justified statement of what cannot be obtained, and the pooled estimates are computed over the poolable items.

# Required outputs

• out/table1.csv — the extraction table, one row per study (5 rows): study, estimate_as_printed, uncertainty_as_printed, sample_as_printed, definition, locator, harmonised_estimate, harmonised_se, harmonisable, poolable, transformation, reason_if_not.
• out/figure1.png — a forest plot of the harmonised estimates (all harmonisable studies; those without an SE shown as points without intervals) with the pooled random-effects estimate.
• out/results.json with the keys:

• out/report.md with the sections: Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues. The synthesis must say what the pooled numbers can and cannot support given the number of poolable studies, the approximate SE, and the studies left out.
• out/analysis.py — running ~/wp29/venv/bin/python out/analysis.py from the workspace root must regenerate out/results.json, out/table1.csv and out/figure1.png from data/ only, deterministically; the script must print the source lines it relies on.
• out/claims.json — a list of the central claims of the report: [{"id": "C1", "text": "...", "results_keys": ["..."], "status": "supported"|"unsupported"|"not_evaluated", "caveat": "..."}].

# Constraints

• No network access. Python: ~/wp29/venv/bin/python (numpy, pandas, scipy, statsmodels, matplotlib available). Do not modify data/.
• Report printed numbers exactly as printed (same sign, same decimals); transformations are applied only in the harmonised columns.
• Budget class: transfer mini-project — 12.00 USD cap, 60 minutes wall, 150 turns per session. The project is designed to be completed in well under 45 minutes of work.