{
  "task_id": "T3-synthesis",
  "target_definition": "theta = ln(T_AI / T_noAI): natural log of the ratio of mean task-completion time with generative-AI assistance to mean completion time without it, on the same tasks, from a randomised comparison (negative = faster with AI).",
  "items": {
    "S1": {
      "estimate_as_printed": "ExpCon coefficient (Model 2, LogToT) = -0.24 (marked '.', p<.1); text: effect of AI = 21% faster",
      "uncertainty_as_printed": "SE (0.14); 95% CI = [-0.51, 0.03]; p=0.086",
      "sample_as_printed": "n = 96 (Table III caption); Section VI-B: N=47 AI, N=46 no-AI",
      "definition": "OLS coefficient on experimental condition (AI vs no-AI exposure) with natural-log time on task as dependent variable, controlling for developer-level factors (AvgProgHrsDay, Level); one complex enterprise coding task, randomised",
      "locator": "arxiv-2410.12944v3, Table III, row 'ExpCon', column (2) 'Dev' (value -0.24., SE line (0.14)); CI and p-value in Section VI-C text",
      "harmonisable": true,
      "harmonised_estimate": -0.24,
      "harmonised_se": 0.14,
      "poolable": true,
      "transformation": "theta = b = -0.24; SE(theta) = SE(b) = 0.14 (printed). Cross-check from printed CI: (0.03-(-0.51))/(2*1.96) = 0.1378",
      "reason_if_not": null
    },
    "S2": {
      "estimate_as_printed": "+19% (\"allowing AI actually increases completion time by 19%\"; \"implement issues in 19% more time on average\")",
      "uncertainty_as_printed": "none printed in extract (95% CI shown only in Figure 1, not reproduced)",
      "sample_as_printed": "16 developers, 246 tasks (136 issues AI-allowed, 110 AI-disallowed)",
      "definition": "S = E[completion time with AI allowed]/E[completion time with AI disallowed] - 1, estimated as exp(beta)-1 from log-linear OLS of total implementation time on AI indicator and log forecast time; issue-level randomisation",
      "locator": "arxiv-2507.09089v2, Abstract and Section 3 'Results' opening paragraph (19%); estimator in Section 2.3 and Appendix D.1-D.3",
      "harmonisable": true,
      "harmonised_estimate": 0.17395330712343798,
      "harmonised_se": null,
      "poolable": false,
      "transformation": "theta = ln(1 + c) = ln(1 + 0.19) = 0.1740 (equivalently theta = beta-hat, since S = exp(beta)-1)",
      "reason_if_not": "No SE, CI or p-value for the 19% estimate is printed in the extract; its confidence interval appears only in Figure 1, which is not reproduced. SE cannot be obtained, so not poolable."
    },
    "S3": {
      "estimate_as_printed": "-29.79%*** (Treatment 44.29 min, Control 63.10 min; Overall 53.84 min)",
      "uncertainty_as_printed": "significance stars only: *** (0<p<=0.01)",
      "sample_as_printed": "181 subjects completed the task satisfactorily (treatment/control split not printed in extract)",
      "definition": "Percentage difference in mean task duration (minutes), Copilot vs control, holding accuracy constant (control duration predicted at treatment accuracy via a control-group regression of duration on accuracy); all content, randomised",
      "locator": "arxiv-2411.01067v2, Table 3, block 'Aggregate Results', row 'All Content', columns Treatment / Control / Improvement",
      "harmonisable": true,
      "harmonised_estimate": -0.353679434958934,
      "harmonised_se": null,
      "poolable": false,
      "transformation": "theta = ln(1 + c) = ln(1 - 0.2979) = -0.3537 (ratio of printed means: ln(44.29/63.10) = -0.3540)",
      "reason_if_not": "Only significance stars (***, p<=0.01) are printed; no SE, CI or exact p-value, so SE cannot be obtained and the study is not poolable."
    },
    "S4": {
      "estimate_as_printed": "% Time Savings 11.1 (Aware 50.5 s per email, Control 56.9 s per email), Full Corpus",
      "uncertainty_as_printed": "p-value 0.471",
      "sample_as_printed": "Aware N=56, Control N=52 (167 analysts across three arms; 25-email queues)",
      "definition": "Percentage time saving in mean time per email (Aware arm: agent verdicts/rationale and agent-ordered queue vs Control: random order, no AI), across the full queue; randomised",
      "locator": "arxiv-2511.13860v1, Section 3.3, Table 3, column 'Full Corpus', rows 'Aware Time per Email (s)', 'Control Time per Email (s)', '% Time Savings', 'p-value'",
      "harmonisable": true,
      "harmonised_estimate": -0.11765804346823246,
      "harmonised_se": 0.1632205919720678,
      "poolable": true,
      "transformation": "c = -0.111 (11.1% saving); theta = ln(1 - 0.111) = -0.1177; approximate SE = |theta|/z_(1-0.471/2) = 0.1177/0.7209 = 0.1632 (APPROXIMATE SE recovered from p-value). Ratio of printed means: ln(50.5/56.9) = -0.1193",
      "reason_if_not": null
    },
    "S5": {
      "estimate_as_printed": "Treat x Post = -0.278 (Output, column 3, panel A, users with affiliation)",
      "uncertainty_as_printed": "SE (0.295); no stars",
      "sample_as_printed": "N = 18332",
      "definition": "Difference-in-differences coefficient on the aggregate GitHub Output measure (sum of Push, Pull Requests, PR comments, commit comments, Create, Issues events; the text describes log counts for these measures) for Italian vs Austrian/French users, first two business days after vs before the ChatGPT ban",
      "locator": "arxiv-2304.09339v1, Table 1, panel A (Pre 27/28.03 - Post 03/04.04), row 'Treat x Post', column (3) 'Output'; N from row 'N'",
      "harmonisable": false,
      "harmonised_estimate": null,
      "harmonised_se": null,
      "poolable": false,
      "transformation": "none applied (not harmonisable)",
      "reason_if_not": "The printed effect is on output counts (GitHub events), not task-completion time on the same tasks, and it comes from a natural experiment (withdrawal of access via a national ban, DiD) rather than a randomised AI-vs-no-AI comparison of completion times. It cannot be mapped onto theta = ln(T_AI/T_noAI)."
    }
  },
  "k_extracted": 5,
  "k_harmonised": 4,
  "k_pooled": 2,
  "fe_estimate": -0.18814332871148873,
  "fe_se": 0.10626483095451474,
  "fe_ci_low": -0.39642239738233764,
  "fe_ci_high": 0.02013573995936016,
  "re_estimate": -0.18814332871148873,
  "re_se": 0.10626483095451474,
  "re_ci_low": -0.39642239738233764,
  "re_ci_high": 0.02013573995936016,
  "q_stat": 0.3236860522786328,
  "i2": 0.0,
  "tau2": 0.0,
  "heterogeneity_interpretation": "low",
  "pooled_significant_5pct": false,
  "pooled_pct_change_re": -17.15040485348688,
  "feasibility": "complete",
  "feasibility_reason": null,
  "data_issues": [
    {
      "issue": "S1 (arxiv-2410.12944v3, Section VI-B): the AI group is stated to be faster, yet its printed Mean LogToT (4.6) is larger than the no-AI group's (4.3); neither matches the printed raw means (exp(4.6)~99, exp(4.3)~74 vs 96 and 114 min). Internal inconsistency of the source.",
      "handling": "Not used; the harmonised value comes from the Table III Model 2 coefficient as instructed by studies.json."
    },
    {
      "issue": "S1: Table III caption says n = 96 while Section VI-B reports N=47 + N=46 = 93 for the raw means.",
      "handling": "Both sample sizes reported as printed; no effect on theta/SE, which are taken directly from Table III."
    },
    {
      "issue": "S1: Table III rendering collapses the four model columns into one cell per row, and the R^2 / p-value rows have garbled labels ('R^{2} ... Adj.', '... p value').",
      "handling": "Column order Models 1-4 left to right per the editorial note; Model 2 ExpCon = -0.24 (0.14) cross-checked against the text (beta_1 = -0.24, CI [-0.51, 0.03], p = 0.086)."
    },
    {
      "issue": "S1: two possible SEs: printed SE 0.14 (rounded to 2 dp) and the SE implied by the printed 95% CI, (0.03+0.51)/3.92 = 0.1378.",
      "handling": "Primary analysis uses the printed SE 0.14; sensitivity with the CI-based SE reported (FE = -0.1891)."
    },
    {
      "issue": "S2: the 95% CI of the headline 19% estimate is displayed only in Figure 1 (not reproduced); the raw ratio of means (34%) is also printed but is not the regression estimate.",
      "handling": "Harmonised the regression-based 19% (theta = ln 1.19); SE set to null and study not pooled; 34% not used."
    },
    {
      "issue": "S3: the printed improvement -29.79% differs slightly from the ratio of the printed means (44.29/63.10 - 1 = -29.81%), presumably due to rounding of the means; sign conventions differ between text ('29.79% reduction') and table ('-29.79%').",
      "handling": "Used the printed improvement c = -0.2979 per the harmonisation convention; difference in theta < 0.001."
    },
    {
      "issue": "S4: Table 3 header says 'Time per Email (s)', but the Malicious/Non-Malicious columns (2.8, 1.8, 1.9, 2.4) are implausible as seconds next to Full Corpus 50.5/56.9 s and do not reproduce their printed % savings (e.g. 1-2.8/1.9 = -47.4% vs printed -52.9); likely minutes and/or rounding. Also 1-50.5/56.9 = 11.25% vs printed 11.1.",
      "handling": "Only the Full Corpus column is used, with the printed % Time Savings (11.1) as the harmonisation input; sign convention: positive savings means less time with AI, so c = -0.111."
    },
    {
      "issue": "S4: only a two-sided p-value (0.471) is printed for the full-corpus time saving; the test underlying it (likely on raw seconds, not logs) is not described in the extract.",
      "handling": "Approximate SE = |theta|/z_(1-p/2) as permitted; study pooled with an approximate SE and flagged."
    },
    {
      "issue": "S5: Table 1 notes print the significance key as '** p<0.5' (presumably p<0.05); the row label renders as '\\times TreatPost'.",
      "handling": "Recorded as printed; S5 is not harmonisable regardless."
    },
    {
      "issue": "Stray 'p<' / 'p=' lines and footnote numbers in the extracts (rendering artefacts per data/README.md).",
      "handling": "Ignored."
    }
  ],
  "n_input_rows": {
    "studies.json": 5,
    "README.md": 17,
    "arxiv-2304.09339v1.txt": 108,
    "arxiv-2410.12944v3.txt": 112,
    "arxiv-2411.01067v2.txt": 109,
    "arxiv-2507.09089v2.txt": 132,
    "arxiv-2511.13860v1.txt": 162
  },
  "n_analysed": 5,
  "random_seed": null
}