# T3-synthesis: Does generative-AI assistance shorten task-completion time?

## Question
Does access to a generative-AI assistant change how long workers take to complete their tasks? I harmonised the five primary studies onto θ = ln(T_AI / T_noAI) (`target_definition`), the log ratio of mean completion time with AI to mean time without it, for the same tasks, in a randomised comparison. Negative θ means faster with AI. I then pooled the studies that could be pooled with inverse-variance fixed-effect (FE) and DerSimonian–Laird random-effects (RE) models.

## Data and exclusions
The only inputs were the five text extracts under `data/sources/` and `data/studies.json`. All SHA-256 checksums match `data/README.md`, and line counts are in `n_input_rows`. Every one of the 5 items had a printed value to extract (`k_extracted` = 5, `n_analysed` = 5). `out/table1.csv` and `items` give the printed values verbatim with locators. The exclusion ledger (`out/exclusions.csv`) runs 5 assessed → 4 harmonisable → 2 poolable:
- **S5** (Kreitmeir & Raschky, ChatGPT ban; printed −0.278 (0.295), N = 18332) is **not harmonisable**. It is a difference-in-differences coefficient on GitHub output counts around the withdrawal of access, not a comparison of task-completion times on the same tasks (`items.S5.reason_if_not`).
- **S2** (METR; +19%) is harmonisable but **not poolable**. Its confidence interval appears only in Figure 1/15, which the extract does not reproduce, and no SE or p-value is printed (`items.S2.reason_if_not`).
- **S3** (Security Copilot; −29.79%***) is harmonisable but **not poolable**. It reports significance stars only (`items.S3.reason_if_not`).
- That leaves **S1** (Paradis et al.) and **S4** (Bono, phishing agent) in the pool (`k_pooled` = 2, `pooled_studies`).

I found eight input problems, listed in `data_issues`. The main ones:
- S1: n = 96 in Table III but 47 + 46 = 93 in the text, and the two groups' log means appear swapped.
- S3: the printed −29.79% differs slightly from the −29.81% implied by the printed means.
- S4: the units in Table 3 are inconsistent across columns.
- S5: the table note prints "** p<0.5".

None of these changes a harmonised value by more than about 0.0003, apart from the S1 SE choice, which I examine under Uncertainty.

## Method
I applied the TASK conventions to each study; the `items.*.transformation` field records each formula with its inputs:
- **S1:** the Model 2 ExpCon coefficient on log time on task gives θ = b = −0.24, with printed SE 0.14 (`items.S1.harmonised_estimate`, `items.S1.harmonised_se`).
- **S2:** θ = ln(1.19) = 0.174 (`items.S2.harmonised_estimate`).
- **S3:** θ = ln(1 − 0.2979) = −0.354 (`items.S3.harmonised_estimate`).
- **S4:** a printed 11.1% time saving gives c = −0.111 and θ = ln(0.889) = −0.118. Its SE is **approximate**, recovered from the two-sided p = 0.471 as |θ|/z₀.₇₆₄₅ = 0.163 (`items.S4.harmonised_se`).

Pooling used inverse-variance FE and DL RE, with 95% intervals at 1.96, Q on k − 1 = 1 df, I² = max(0, (Q − df)/Q)·100, and τ² = max(0, (Q − df)/C). As a baseline I also computed unweighted means of θ (`baseline_unweighted_mean_theta_poolable`, `baseline_unweighted_mean_theta_harmonised`). The script `out/analysis.py` regenerates everything, uses no randomness (`random_seed` = null), and prints the source lines it parses. Figure: `out/figure1.png`.

## Results
- Harmonised θ: S1 −0.240 (SE 0.140), S2 +0.174 (no SE), S3 −0.354 (no SE), S4 −0.118 (approx. SE 0.163). Keys: `items.*.harmonised_estimate`, `items.*.harmonised_se`. `k_harmonised` = 4.
- **FE pooled θ = −0.188**, SE 0.106, 95% CI [−0.396, 0.020] (`fe_estimate`, `fe_se`, `fe_ci_low`, `fe_ci_high`).
- **RE pooled θ = −0.188**, SE 0.106, 95% CI [−0.396, 0.020] (`re_estimate`, `re_se`, `re_ci_low`, `re_ci_high`). RE equals FE because τ² = 0 (`tau2`).
- On the time scale this is a −17.2% change in mean task time (`pooled_pct_change_re`), with a 95% interval of about −32.7% to +2.0% (`pooled_pct_change_re_ci`).
- Q = 0.324 on 1 df, p = 0.57 (`q_stat`, `q_df`, `q_pvalue`). I² = 0% (`i2`), so heterogeneity is classed as "low" (`heterogeneity_interpretation`).
- The RE interval includes 0, so the pooled effect is not significant at 5% (`pooled_significant_5pct` = false).
- Baselines: the unweighted mean of the two poolable θ is −0.179 (`baseline_unweighted_mean_theta_poolable`). The unweighted mean of all four harmonised θ is −0.134 (`baseline_unweighted_mean_theta_harmonised`). The second is descriptive only, because two of those studies have no SE.

**Synthesis.** The two poolable randomised studies point towards shorter task times with AI, but the pooled interval includes no effect. These data cannot establish that generative-AI assistance shortens task-completion time. The harmonisable studies that could not be pooled disagree in direction: S3 shows a large reduction (−0.354) and S2 a slowdown (+0.174). So the I² = 0% computed from k = 2 says nothing reliable about heterogeneity across settings, and the evidence as a whole is mixed and context-dependent: developers in an enterprise lab task, experienced open-source developers on their own repositories, IT administrators, and SOC analysts.

## Uncertainty
- With k = 2, Q has a single degree of freedom. The DL τ² and I² are extremely imprecise, and a zero τ² estimate does not mean the studies are homogeneous.
- The S4 SE is approximate: it is back-calculated from a p-value under a normal approximation on the log scale, while the test in the paper was not necessarily on that scale. S4 carries 42% of the pooled weight (`weights_fe`).
- **Sensitivity.** Using the S1 SE implied by its printed CI (0.1378 instead of 0.14) gives RE −0.189 [−0.395, 0.017] (`sensitivity_s1_se_from_ci`). The conclusion is unchanged.
- Different estimands sit under one θ. S1 and S2 are regression-adjusted log-time contrasts. S3 holds accuracy constant using predicted control durations. S4 is a ratio of mean time per email in an arm that also re-ordered the queue. Each S1 or S2 coefficient equals the log of a ratio of means only under the assumptions in the papers.

## Limitations and open issues
- Only 2 of 5 studies could be pooled (`k_pooled`). The pooled estimate rests on one 96-developer coding task and one phishing-triage experiment, and it leaves out the two other randomised time studies, which have the largest effects in each direction (S2 +0.174, S3 −0.354). If they could be pooled the result could move substantially. Their SEs are not printed in the extracts, and I did not substitute values from figures or memory, as the information rules require.
- `feasibility` = "complete" in the TASK's sense: every item has either harmonised values or an explicit, justified reason why none can be obtained, and the pool covers all poolable items. This does not mean the evidence base is adequate.
- The source inconsistencies in `data_issues` (S1 sample size and log means, S4 units) cannot be resolved from the extracts.
- No assessment of risk of bias or publication bias was possible or attempted. Three of the studies come from vendors (Google, Microsoft ×2).