=== Source lines relied on ===
[arxiv-2410.12944v3 L76] -0.25. -0.24. -0.30* -0.14 ExpCon | | | |
[arxiv-2410.12944v3 L78] (0.14) (0.14) (0.14) (0.17) | | | |
[arxiv-2410.12944v3 L58] However, our confidence intervals are large, and as a result the estimate is not statistically significant at the p<0.05 level (estimate of experimental condition on best-fit Model 2, \beta_{1}=-0.24;   95\%CI = [-0.51,0.03] , p=0.086 , NS). Our second hypothesis is therefore only partially supported.
[arxiv-2410.12944v3 L66] TABLE III: Summaries of regressions using LogToT as the dependent variable, n = 96. Unstandardized estimates provided, along with standard error of the mean (in parentheses).
[arxiv-2410.12944v3 L48] On average, developers who were exposed to AI features completed the task in 96 min ( N=47,SE=9.3 ), compared to 114 min ( N=46,SE=8.1 ) for developers in the no-AI condition. We conducted a Student’s t-test on log time on task to assess the main effect impact of the experimental condition assignment (AI vs no-AI exposure). Results of this analysis indicated that developers who were in the AI exposure condition (Mean LogToT=4.6 ) were significantly faster at completing the task ( t(83.6)=2.11,p=.038 ), compared to those who were in the no-AI exposure condition (Mean LogToT=4.3 ).
[arxiv-2507.09089v2 L34] Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down.
[arxiv-2507.09089v2 L62] Developers complete 136 issues with AI-allowed and 110 issues with AI-disallowed. Section G.7 shows the number of issues completed across repositories and developers, respectively. We find that when developers use AI tools, they implement issues in 19% more time on average (Figure 1), and nearly all quantiles of observed implementation time see AI-allowed issues taking longer (Figure 5). That is, developers are slower when using AI is allowed. Colloquially, we refer to this result that issues with AI-allowed take longer than issues with AI-disallowed as slowdown.
[arxiv-2507.09089v2 L8] Omitted: Sections 1, 2.1-2.2, 2.4, 3.2-3.3, 4-5, Appendices A-C, D.4-D.5, E-G, Tables 1-8, all figures (including Figure 1, which displays the confidence interval of the main estimate graphically), the references.
[arxiv-2411.01067v2 L55] All Content | 53.84 | 44.29 | 63.10 | -29.79%***
[arxiv-2411.01067v2 L51] | Overall | Treatment | Control | Improvement
[arxiv-2411.01067v2 L38] We had 181 subjects complete the task satisfactorily.^3^3
[arxiv-2411.01067v2 L10] Editorial note: the paper marks significance with asterisks only (defined in the methodology as * 0.05<p<=0.1, ** 0.01<p<=0.05, *** 0<p<=0.01); no standard errors or confidence intervals are printed for the time-savings estimates.
[arxiv-2511.13860v1 L88] Metric | Malicious | Non-Malicious | Full Corpus
[arxiv-2511.13860v1 L90] Aware Time per Email (s) | 2.8 | 1.8 | 50.5
[arxiv-2511.13860v1 L92] Control Time per Email (s) | 1.9 | 2.4 | 56.9
[arxiv-2511.13860v1 L94] % Time Savings | -52.9 | 23.6 | 11.1
[arxiv-2511.13860v1 L96] p-value | 0.006 | 0.126 | 0.471
[arxiv-2511.13860v1 L31] Control ( N{=}52 ): Random queue order, no references to AI.
[arxiv-2511.13860v1 L35] Aware ( N{=}56 ): Queue ordered by the agent’s verdicts; participants see the agent’s decision, a brief rationale, and an accuracy calibration.
[arxiv-2511.13860v1 L44] We filter out 13 subject responses that scored less than 40% – significantly below guessing percentage. These included three in the Aware group, six in the Blind group, and four in the Control group. This step does not affect our qualitative results. Their task is to triage a 25-email queue. The queue comes from an original corpus of 93 randomly selected emails reported by Microsoft employees. Through an in-depth analysis by expert graders and security researchers, we arrived at our ground truth of 82 non-malicious and 11 malicious emails.
[arxiv-2304.09339v1 L74] | Events | Release | Output | PushPull | Events | Release | Output | PushPull
[arxiv-2304.09339v1 L80] \times TreatPost | 0.115 | -0.012** | -0.278 | -0.227 | 0.005 | -0.003 | 0.188 | 0.192
[arxiv-2304.09339v1 L82] | (0.109) | (0.005) | (0.295) | (0.207) | (0.120) | (0.005) | (0.221) | (0.189)
[arxiv-2304.09339v1 L88] N | 18332 | 18332 | 18332 | 18332 | 14496 | 14496 | 14496 | 14496
[arxiv-2304.09339v1 L42] We also use the (log of) number of total GitHub events by a user, Events , the sum of Push and Pull events, PushPull and an aggregate Output measure based on Holub and Thies, (2023), which is the sum of Push, Pull Requests, Pull Request Comments, Commit Comments, Create, and Issues. In addition, we use information about the users organisational affiliation to distinguish between GitHub users who have entered an organisational affiliation (i.e. company, research organisation) and those without an organisational affiliation.
=== Results ===
S1 theta = -0.24 SE = 0.14 poolable = True
S2 theta = 0.17395330712343798 SE = None poolable = False
S3 theta = -0.353679434958934 SE = None poolable = False
S4 theta = -0.11765804346823246 SE = 0.1632205919720678 poolable = True
S5 theta = None SE = None poolable = False
FE -0.1881 (SE 0.1063); RE -0.1881 [-0.3964, 0.0201]; Q 0.3237; I2 0.00; tau2 0.00000; pct change RE -17.15%