1SOURCE EXTRACT — arxiv-2507.09089v22Title: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity3Authors: Joel Becker; Nate Rush; Beth Barnes; David Rein (Model Evaluation & Threat Research, METR)4Bibliographic record: arXiv:2507.09089v2 [cs.AI], version 2 submitted 25 Jul 2025. Abstract page: https://arxiv.org/abs/2507.09089v25Retrieved: https://arxiv.org/html/2507.09089 on 2026-10-30T04:19:45Z (SHA-256 of the retrieved HTML: e860161026afa19c69e7ba332f5cccc8fec6b8d74865868db1316a364d270c6a)6License: CC BY 4.0 (license shown on the arXiv abstract page; https://creativecommons.org/licenses/by/4.0/). Reproduced in part with attribution.7Extract contents: title, authors and abstract; Section 2.3 'Effect Estimation'; the opening of Section 3 'Results' and Section 3.1 'Forecasts' (figure captions included as text); Appendix D.1 'Regression', D.2 'Confidence intervals' and D.3 'Converting to speedup'.8Omitted: Sections 1, 2.1-2.2, 2.4, 3.2-3.3, 4-5, Appendices A-C, D.4-D.5, E-G, Tables 1-8, all figures (including Figure 1, which displays the confidence interval of the main estimate graphically), the references.9Rendering note: converted from the HTML rendering to plain text. Tables are rendered one row per line with " | " separating cells; in regression tables the line following a coefficient row holds the standard errors in parentheses, exactly as printed. Mathematical expressions appear as LaTeX source; non-breaking spaces were replaced by ordinary spaces. Figures are not reproduced: a value that appears only in a figure is not in this extract. No numbers were altered.10================================================================1112# Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity1314Joel Becker^*1516 17Nate Rush^*1819 20Beth Barnes2122 23David Rein2425Affiliation: Model Evaluation & Threat Research (METR)2627###### Abstract2829Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February–June 2025 frontier affect the productivity of experienced open-source developers.3016 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience.31Each task is randomly assigned to allow or disallow usage of early-2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet.32Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%.33After completing the study, developers estimate that allowing AI reduced completion time by 20%.34Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down.35This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter).36To understand this result, we collect and evaluate evidence for 21 properties of our setting that a priori could contribute to the observed slowdown effect—for example, the size and quality standards of projects, or prior developer experience with AI tooling.37Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.3839^1^1footnotetext: Equal contribution. Correspondence to {nate, joel}@metr.org4041#### 2.3 Effect Estimation4243We aim to measure the percentage change in issue time to completion due to AI, defined as:4445S=\frac{\mathbb{E}[\text{completion time with AI allowed}]}{\mathbb{E}[\text{completion time with AI disallowed}]}-1 | |4647A value of -50% would indicate issues assigned to AI-allowed taking half the time of issues assigned to AI-disallowed; 0% would indicate that issues take the same time to complete with and without AI-allowed; 100% would indicate that issues assigned to AI-allowed take twice as long as issues assigned to AI-disallowed. For convenience and clarity, we typically report the absolute value of S , and colloquially refer to negative values of as speedup, and positive values as slowdown^8^84884950This is a mild abuse of language, because speedup and slowdown refer to rates, while the quantity we actually estimate and report is the percentage change in time from AI.5152We use the total implementation time (including both pre- and post-pull request review time) developers self-report spending on each issue to estimate the percentage change in issue completion time due to AI. A small number of issues do not end up getting reviewed/merged for various reasons—we impute the post-review implementation time of these issues with the mean post-review time per treatment condition. Section C.3.4 shows that our results are robust to the specific imputation method used.5354We estimate S via a log-linear regression for total implementation time, incorporating estimates of issue difficulty to reduce noise in our estimate. We also use adapted versions of this regression when examining how performance varies in different subsets of the data (to efficiently account for any post-randomization issue difficulty imbalances that may be more pronounced within subsets). See Appendix D for details of the regression approach used.5556### 3 Results5758Figure 4: Left: Raw average forecasted implementation times. Right: Raw average observed implementation times. The ratio of observed implementation times gives a more extreme slowdown estimate than regression-based estimates (Section D.1) because AI-allowed issues are forecasted (importantly, before treatment assignment) by developers to take slightly longer, which the regression corrects for. Both: Section D.5 describes confidence intervals around ratios of average times.5960Figure 5: Empirical cumulative distribution functions of observed implementation times. Percentile ordering is calculated separately for each treatment group.6162Developers complete 136 issues with AI-allowed and 110 issues with AI-disallowed. Section G.7 shows the number of issues completed across repositories and developers, respectively. We find that when developers use AI tools, they implement issues in 19% more time on average (Figure 1), and nearly all quantiles of observed implementation time see AI-allowed issues taking longer (Figure 5). That is, developers are slower when using AI is allowed. Colloquially, we refer to this result that issues with AI-allowed take longer than issues with AI-disallowed as slowdown.6364#### 3.1 Forecasts6566###### Developer Forecasts and Post hoc Estimates6768Before developers complete each issue, they forecast how long they expect them to take with and without AI assistance. On average, they forecast speedup of 24%. Interestingly, after the experiment they post-hoc estimate that they were sped-up by 20% when using AI is allowed—after they used AI assistance, they estimate similar speedup as before, despite the fact that they are in fact slowed down by 19% (Figure 1).69Figure 4 displays the raw average forecasted and observed implementation times^12^1270127172The raw percentage difference in implementation times between AI-allowed and AI-disallowed issues is 34%, which is larger in absolute value than the 19% slowdown estimated using the regression specified in Section D.1. This is true because AI-allowed issues ended up being slightly more difficult than AI-disallowed issues after randomization, as measured by the forecasted AI-disallowed times (not statistically significant; see Table 4). Our regression accounts for this, while this simple ratio estimator does not. See Figure 13 for results implied by alternative estimators..7374Despite developers forecasting speedup from AI (while they are slowed down), developer forecasts are informative about completion time—the Pearson correlation between the time developers forecast AI-allowed issues taking and the actual time they take is 0.64, and the correlation between the time developers forecast AI-disallowed issues taking and the actual time they take is 0.59. This suggests that developers are broadly well-calibrated on the relative amount of time that issues will take, but their expectations regarding the usefulness of AI assistance are reversed.7576#### D.1 Regression7778For each issue i we observe the realised completion time T_{i}>0 , a binary treatment flag \text{AI}_{i}\in\{0,1\} , and the developer’s ex-ante forecast of how long the task would take without AI, denoted \widehat{T}^{\text{NoAI}}_{i}>0 .7980We estimate the log-linear model8182\log T_{i}=\alpha+\beta\,\text{AI}_{i}+\delta\,\log\widehat{T}^{\text{NoAI}}_{i}+\varepsilon_{i}, | | | (1)8384via ordinary least squares. Random assignment of \text{AI}_{i} guarantees consistency of \hat{\beta} for \beta=\mathbb{E}[\log T\,|\,\text{AI}=1]-\mathbb{E}[\log T\,|\,\text{AI}=0] .8586We include forecasts as a control variable because they serve as a proxy for issue difficulty and are highly predictive of completion times. This substantially increases our statistical power without introducing bias, as forecasts were elicited prior to treatment assignment and thus cannot be affected by treatment status.^18^1887188889We do not generally include developer fixed effects because they explain minimal variation in the outcome conditional on forecasts. Section C.3.5 displays estimates from a regression specification including developer fixed effects.9091Figure 14: Regression diagnostics associated with the regression specification in equation (1).9293Figure 14displays regression diagnostics associated with this specification.9495#### D.2 Confidence intervals9697By default, we report 95% confidence intervals using HC3 standard errors.^19^19981999100Given that our sampling of developers is non-random and treatment is assigned at the issue level, Abadie et al. [2022] do not necessarily recommend developer-level clustering even if residuals are correlated within-developer. If we are willing to treat developer sampling as random then clustering is appropriate. Figure 15 displays 95% confidence intervals from alternative uncertainty estimation procedures. Standard errors clustered at the developer level and bias-corrected cluster-robust standard errors give similar results; a hierarchical bootstrap resampling developers and then issues within developers yields somewhat wider confidence intervals.101102Figure 15: Speedup calculated using the regression specification in equation (1) and alternative uncertainty estimation procedures. CR2 and CR3 standard errors are bias-corrected cluster-robust standard errors Colin Cameron and Miller [2015], Bell and Mccaffrey [2002]. Two-stage bootstrap resamples developers, then issues within developers.103104#### D.3 Converting to speedup105106Taking conditional expectations of equation (1) given treatment status and forecast:107108\displaystyle\mathbb{E}[\log T\,|\,\text{AI}=1,\widehat{T}^{\text{NoAI}}] \displaystyle=\alpha+\beta+\delta\,\log\widehat{T}^{\text{NoAI}} | | | | (2)109110\displaystyle\mathbb{E}[\log T\,|\,\text{AI}=0,\widehat{T}^{\text{NoAI}}] \displaystyle=\alpha+\delta\,\log\widehat{T}^{\text{NoAI}} | | | | (3)111112Therefore, \beta=\mathbb{E}[\log T\,|\,\text{AI}=1,\widehat{T}^{\text{NoAI}}]-\mathbb{E}[\log T\,|\,\text{AI}=0,\widehat{T}^{\text{NoAI}}] represents the average treatment effect on the log scale, conditional on forecast.113114To convert to a speedup measure, we note that:115116\displaystyle\exp(\beta) \displaystyle=\frac{\mathbb{E}[T\,|\,\text{AI}=1,\widehat{T}^{\text{NoAI}}]}{\mathbb{E}[T\,|\,\text{AI}=0,\widehat{T}^{\text{NoAI}}]}\quad | | | | (4)117118\displaystyle=\exp(\beta)-1 | S | | | (5)119120(Step (4) holds provided the disturbance \varepsilon_{i} is independent of both \text{AI}_{i} and \widehat{T}^{\text{NoAI}}_{i} and that \mathbb{E}\!\left[e^{\varepsilon_{i}}\right] exists.^20^2012120122123Because T_{i}=e^{\alpha}e^{\beta\,\text{AI}_{i}}\,(\widehat{T}^{\text{NoAI}}_{i})^{\delta}\,e^{\varepsilon_{i}} , taking expectations conditional on \text{AI}_{i} and \widehat{T}^{\text{NoAI}}_{i} gives \mathbb{E}[T\,|\,\text{AI}=j,\widehat{T}^{\text{NoAI}}]=e^{(\alpha+\beta j)}\,(\widehat{T}^{\text{NoAI}})^{\delta}\,\mathbb{E}[e^{\varepsilon_{i}}] .124The common factor \mathbb{E}[e^{\varepsilon_{i}}] cancels when we form the ratio of these conditional means, yielding \mathbb{E}[T\,|\,\text{AI}=1,\widehat{T}^{\text{NoAI}}]\big/\mathbb{E}[T\,|\,\text{AI}=0,\widehat{T}^{\text{NoAI}}]=\exp(\beta) .)125126Then under standard regularity assumptions (which our diagnostics suggest hold) our OLS estimator \hat{\beta} is normally distributed \mathcal{N}(\beta,\sigma^{2}) for some \sigma which we can estimate using it’s standard error \text{SE}[\hat{\beta}] . This lets us construct a 95% confidence interval for \beta in the usual way:127128\text{CI}_{95\%}=\left[\hat{\beta}-1.96\cdot\text{SE}[\hat{\beta}],\hat{\beta}+1.96\cdot\text{SE}[\hat{\beta}]\right] | | | (6)129130As \text{S}=\exp(\beta)-1 is a monotonic function of \beta we can construct a confidence interval for S by simply applying the function to the endpoints of \beta ’s confidence interval:131132\text{CI}_{95\%}=\left[e^{\hat{\beta}-1.96\cdot\text{SE}[\hat{\beta}]}-1,e^{\hat{\beta}+1.96\cdot\text{SE}[\hat{\beta}]}-1\right] | | | (7)133