1SOURCE EXTRACT — arxiv-2410.12944v32Title: How much does AI impact development speed? An enterprise-based randomized controlled trial3Authors: Elise Paradis; Kate Grey; Quinn Madison; Daye Nam; Andrew Macvean; Vahid Meimand; Nan Zhang; Ben Ferrari-Church; Satish Chandra (Google)4Bibliographic record: arXiv:2410.12944v3 [cs.SE], version 3 submitted 11 Nov 2024. Abstract page: https://arxiv.org/abs/2410.12944v35Retrieved: https://arxiv.org/html/2410.12944 on 2026-10-30T04:19:39Z (SHA-256 of the retrieved HTML: 2a5507903fe618f4cd8f91c0268ced5032a265742e9d7c18fa3d45c1170fe2ec)6License: CC BY 4.0 (license shown on the arXiv abstract page; https://creativecommons.org/licenses/by/4.0/). Reproduced in part with attribution.7Extract contents: title, authors and abstract; Section IV-C 'Analytic approach'; Sections VI-B 'RQ1: Impact of AI tools on developer task speed' and VI-C 'RQ2: Impact of covariates on main effect' including Table III (regressions with LogToT as dependent variable).8Omitted: Sections I-III, IV-A, IV-B, V, VI-A, VI-D, VII-IX, Tables I-II, all figures, the references.9Rendering note: converted from the HTML rendering to plain text. Tables are rendered one row per line with " | " separating cells; in regression tables the line following a coefficient row holds the standard errors in parentheses, exactly as printed. Mathematical expressions appear as LaTeX source; non-breaking spaces were replaced by ordinary spaces. Figures are not reproduced: a value that appears only in a figure is not in this extract. No numbers were altered.10Editorial note: in the HTML rendering of Table III the four model columns are collapsed into one cell per row (values for Models 1-4 appear left to right on each line, followed by the variable name); the line below each coefficient line holds the standard errors in parentheses in the same order. Significance markers: '.' p<.1, '*' p<.05, '**' p<.01, '***' p<.001.11================================================================1213# How much does AI impact development speed?1415An enterprise-based randomized controlled trial1617Elise Paradis, Kate Grey, Quinn Madison, Daye Nam,1819Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, Satish Chandra2021###### Abstract2223How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.2425### IV-C Analytic approach2627We ran a two-tailed Student t-test ( \alpha=0.05 ) on logged time on task, comparing mean time for participants in the control versus experimental groups.2829To explore the robustness of our estimate and test our theoretical model, we ran three follow up linear regressions: one with the complete framework, i.e. with the experimental condition, developer-level factors, and task-level factors on logged time on task; one with only the experimental condition and developer-level factors; and one with only the experimental condition and task-level factors.3031The full framework can be represented by the following equation:3233\displaystyle LogToT \displaystyle=\beta_{1}*ExpCon+\beta_{2}*AvgProgHrsDay | | |3435\displaystyle+\beta_{3}*Level+\beta_{4}*NbrHighFreqAreas | | |3637\displaystyle+\beta_{5}*TopLangCpp+\beta_{6}*DataLogExp+\epsilon | | |3839Given the logged dependent variable, the effect of a variable n on time on task in percentages is represented by (1-exp(\beta_{n}))*100 . The impact of AI on time on task (our ExpCon variable), expressed in percentages, is thus represented by (1-exp(\beta_{1}))*100 . We did not use hierarchical linear modeling given the absence of nesting between constructs in our theoretical model.4041Finally, to explore interaction effects between experimental condition and developer-level factors, we ran three separate linear regressions with interaction effects, one for each covariate: average hours coded daily, level (seniority), and frequency of use of AI tooling.4243To identify the best model, we used adjusted R^{2} values and model-level p values.4445### VI-B RQ1: Impact of AI tools on developer task speed4647Figure 7 shows the time distribution by experimental condition in a box and whisker plot.48On average, developers who were exposed to AI features completed the task in 96 min ( N=47,SE=9.3 ), compared to 114 min ( N=46,SE=8.1 ) for developers in the no-AI condition. We conducted a Student’s t-test on log time on task to assess the main effect impact of the experimental condition assignment (AI vs no-AI exposure). Results of this analysis indicated that developers who were in the AI exposure condition (Mean LogToT=4.6 ) were significantly faster at completing the task ( t(83.6)=2.11,p=.038 ), compared to those who were in the no-AI exposure condition (Mean LogToT=4.3 ).4950p<5152### VI-C RQ2: Impact of covariates on main effect5354Table III summarizes the regression findings. When controlling for other factors in our theoretical framework, we find that estimates for the impact of the experimental condition on logged time on task range between -0.30 to -0.24 ( \beta_{1} for ExpCon for Models 1-3 in Table III), suggesting a relatively consistent and negative downward pressure on time on task. Controlling for the developer- and task-level factors described in our theoretical framework (see Section IV-B), we found that this main effect was robust to covariates, making developers between 21% and 26% faster.^2^25525657From Model 2, where \beta_{1}=-0.24 , and therefore the effect of AI =(1-exp(\beta_{1}))*100=21\% (with 95% CI = [-0.51, 0.03]). Similarly, 26% from Model 3, where \beta_{1}=-0.30 .58However, our confidence intervals are large, and as a result the estimate is not statistically significant at the p<0.05 level (estimate of experimental condition on best-fit Model 2, \beta_{1}=-0.24; 95\%CI = [-0.51,0.03] , p=0.086 , NS). Our second hypothesis is therefore only partially supported.5960Controlling for experimental condition and other model covariates, average programming hours per day was significantly associated with time spent on task, with participants reporting five or more hours of development work per day being 32% ( \beta=-0.38 ) faster than participants reporting zero to four hours of development work per day. Similarly, higher levels of seniority were also associated with a decrease in time on task, with one increased level (from 4 to 5, or 5 to 6, for example) being associated with a 15% ( \beta=-0.16 ) decrease in time spent on task.6162Other covariates were not significantly associated with time spent on task. As seen in Table III, the best fit model was Model 2 with developer-level factors only, as evidenced by higher adjusted R^{2} and lower p -value. While the total variance accounted for by Model 2 was low, the model was statistically significant ( p=0.011 ), suggesting an adequate fit to our data.6364p=6566TABLE III: Summaries of regressions using LogToT as the dependent variable, n = 96. Unstandardized estimates provided, along with standard error of the mean (in parentheses).6768Complete Dev Task Interaction | | | |6970(1) (2) (3) (4) | | | |71725.36*** 5.41*** 4.55*** 4.66*** Constant | | | |7374(0.42) (0.37) (0.19) (0.11) | | | |7576-0.25. -0.24. -0.30* -0.14 ExpCon | | | |7778(0.14) (0.14) (0.14) (0.17) | | | |7980-0.38* -0.38* -0.16 AvgProgHrsDay | | | |8182(0.15) (0.15) (0.22) | | | |8384-0.16. -0.16* Level | | | |8586(0.08) (0.08) | | | |8788-0.01 0.00 NbrHighFreqAreas | | | |8990(0.03) (0.03) | | | |91920.02 0.06 TopLangCpp | | | |9394(0.15) (0.15) | | | |95960.05 0.02 DataLogExp | | | |9798(0.09) (0.09) | | | |99100-0.29 EC:APHD | | | |101102(0.29) | | | |103104R^{2} 0.078 0.095 0.017 0.075 Adj. | | | |1051060.040 0.011 0.209 0.018 p value | | | |107108p< p< p< p< .: .1 *: .05. **: .01. ***: 0.001 | | | |109110Note: Estimates are for our logged dependent variable. The effect of each | | | |111112covariate on time on task can be calculated by exponentiating the estimate. | | | |113