SPB Git

spb/wp29_uqo

Public
4.1 KB · 7 lines csv
1study,estimate_as_printed,uncertainty_as_printed,sample_as_printed,definition,locator,harmonised_estimate,harmonised_se,harmonisable,poolable,transformation,reason_if_not2S1,"ExpCon coefficient, Model 2 (Dev): -0.24 (marked '.'); quoted in VI-C as beta_1=-0.24, i.e. AI effect (1-exp(beta_1))*100 = 21%","SE (0.14); 95% CI = [-0.51, 0.03]; p=0.086 (NS)","n = 96 (Table III caption); VI-B: N=47 (AI), N=46 (no-AI)","OLS coefficient on experimental condition (AI vs no-AI) with developer-level covariates; dependent variable = natural log of time on task (LogToT); randomised enterprise RCT, one complex coding task","arxiv-2410.12944v3, Table III (L66), row ExpCon (L76) and SE row (L78), column (2) 'Dev' = Model 2; CI and p-value in Section VI-C (L58)",-0.24,0.14,True,True,log-time coefficient: theta = b = -0.24; SE(theta) = SE(b) = 0.14 (CI-implied alternative (0.03-(-0.51))/(2*1.96) = 0.1378),3S2,"+19% (""allowing AI actually increases completion time by 19%""; ""they implement issues in 19% more time on average"")","none printed in the extract (95% CI shown only in Figure 1/Figure 15, not reproduced)","16 developers, 246 tasks (136 issues AI-allowed, 110 AI-disallowed)","S = E[time with AI allowed]/E[time with AI disallowed] - 1, estimated as exp(beta)-1 from OLS of log implementation time on AI indicator and log forecast time (issue-level randomisation)","arxiv-2507.09089v2, Abstract (L34); Section 3 Results (L62); estimator Sections 2.3 and D.1-D.3",0.17395330712343798,,True,False,percentage change c = +0.19: theta = ln(1 + 0.19) = ln(1.19),"Not poolable: no SE, CI or p-value for the 19% estimate is printed in the extract; its confidence interval appears only in figures that are not reproduced."4S3,All Content: Overall 53.84 | Treatment 44.29 | Control 63.10 (minutes) | Improvement -29.79%***,significance stars only: *** (0<p<=0.01); no SE or CI printed,181 subjects completed the task satisfactorily (one further subject dropped),"Difference in mean task duration (minutes), Copilot vs control, holding accuracy constant (control duration predicted at the treatment group's accuracy, Appendix B); randomised assignment","arxiv-2411.01067v2, Section 4.2, Table 3 (L87), 'Aggregate Results' block, row 'All Content' (L55), columns Treatment/Control/Improvement",-0.353679434958934,,True,False,percentage change c = -0.2979: theta = ln(1 - 0.2979) = ln(0.7021) (ratio of printed durations 44.29/63.10 would give -0.35396),"Not poolable: uncertainty is reported by significance stars only (***), so SE(theta) cannot be obtained."5S4,Full Corpus: Aware Time per Email 50.5 s | Control Time per Email 56.9 s | % Time Savings 11.1,p-value 0.471,"Aware N=56, Control N=52 (Blind N=59; 167 analysts in total, 13 low-scoring responses filtered); 25-email queue per subject","Mean time per email (seconds), Aware arm (agent verdicts, rationale and agent-ordered queue) vs Control arm (no AI), full queue; % time savings = reduction relative to Control; randomised three-arm RCT","arxiv-2511.13860v1, Section 3.3, Table 3 (L88), rows Aware/Control Time per Email, % Time Savings, p-value (L90-L96), column 'Full Corpus'",-0.11765804346823246,0.1632205919720678,True,True,time saving 11.1% -> c = -0.111: theta = ln(1 - 0.111) = ln(0.889) = -0.11766; approximate SE from two-sided p: SE = |theta| / z_(1-0.471/2) = 0.11766 / 0.72085 = 0.16322 (APPROXIMATE SE),6S5,"Treat x Post (Output, column 3, panel A, users with affiliation): -0.278","SE (0.295), no significance stars",N = 18332 (user-day observations),"Difference-in-differences coefficient on an aggregate GitHub Output count measure (Push, Pull Requests, PR comments, commit comments, Create, Issues) for Italian vs Austrian/French users, pre (27/28.03.2023) vs post (03/04.04.2023) the ChatGPT ban; user FE, user-clustered SE","arxiv-2304.09339v1, Section 4, Table 1 (header L74), panel A, row Treat x Post (L80) and SE row (L82), column (3) Output; N row (L88)",,,False,False,none applied (not harmonisable),"Not harmonisable: the printed effect is on output counts (activity volume) around a withdrawal of access in an observational DiD, not a comparison of task-completion times with and without AI on the same tasks."7
Files · main
Menu

Theme