SOURCE EXTRACT — arxiv-2411.01067v2
Title: Randomized Controlled Trials for Security Copilot for IT Administrators
Authors: James Bono; Alec Xu (Microsoft Corporation)
Bibliographic record: arXiv:2411.01067v2 [econ.GN], version 2 submitted 12 Nov 2024. Abstract page: https://arxiv.org/abs/2411.01067v2
Retrieved: https://arxiv.org/html/2411.01067 on 2026-10-30T04:20:32Z (SHA-256 of the retrieved HTML: 649d54bc890b793eb99d51e52fcbccf1a93c0cd853953debe0da36a195945c04)
License: CC BY 4.0 (license shown on the arXiv abstract page; https://creativecommons.org/licenses/by/4.0/). Reproduced in part with attribution.
Extract contents: title, authors and abstract; from Section 3 'Methodology' the RCT protocol and subject-count paragraphs; the opening of Section 4 'Results'; Section 4.2 'Speed' in full (Table 3 'Comparison of task duration (minutes) by task type, holding accuracy constant' and figure captions); Appendix B 'Calculating Time Savings Holding Accuracy Constant'.
Omitted: Sections 1-2, the rest of Section 3 (tasks, Table 1), Section 4.1 (accuracy, Table 2), 4.3 (sentiment, Tables 4-5), Section 5, Appendix A, all figures, the references.
Rendering note: converted from the HTML rendering to plain text. Tables are rendered one row per line with " | " separating cells; in regression tables the line following a coefficient row holds the standard errors in parentheses, exactly as printed. Mathematical expressions appear as LaTeX source; non-breaking spaces were replaced by ordinary spaces. Figures are not reproduced: a value that appears only in a figure is not in this extract. No numbers were altered.
Editorial note: the paper marks significance with asterisks only (defined in the methodology as * 0.05<p<=0.1, ** 0.01<p<=0.05, *** 0<p<=0.01); no standard errors or confidence intervals are printed for the time-savings estimates.
================================================================

# Randomized Controlled Trials for Security Copilot for IT Administrators

James Bono
^†^†thanks: james.bono@microsoft.com
  
Alec Xu
^†^†thanks: alecxu@microsoft.com

Microsoft Corporation

October 2024

###### Abstract

As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft’s Security Copilot (“Copilot”) on information technology administrators (“IT admins”) through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft’s Entra and Intune admin centers. Across three IT admin scenarios – sign-in troubleshooting, device policy management, and device troubleshooting – Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.

## 3 Methodology

We follow a standard RCT protocol to identify and estimate the causal effect of Copilot on accuracy and task completion time. That is, we randomly assign half of our subjects to the treatment group, which gives them the ability to use the Entra and Intune admin centers with Copilot in responding to the experimental tasks. The other half of the subjects are assigned to the control group and have use of the Entra and Intune admin centers without Copilot. To make our design as realistic as possible, we placed no restrictions on how subjects used other tools, such as other AI tools or web searches.^2^2
2

Indeed, after they completed the task, we asked control subjects whether they used any other AI tools, and 25% indicated they used some AI tool to help them in this task – most said they used ChatGPT. Therefore, our measurements reflect the incremental benefit of Copilot on top of the admin centers and other tools like web searches and non-Copilot AI. This makes our estimates more conservative than they would be if we had tightly restricted tool use beyond Copilot.

We recruited our subject pool through Upwork, a marketplace for freelancers. We required subjects to be proficient in reading and writing English and to have a positive reputation on Upwork. We told them to expect the tasks to take approximately two hours to complete. We offered performance incentives for combined speed and accuracy. Specifically, we multiplied the percentiles of each subject’s speed and accuracy and awarded payments on the following scale:

We had 181 subjects complete the task satisfactorily.^3^3
3

We dropped one subject who finished in unreasonably short time and scored worse than guessing percentage. Our subject pool embodied a diverse range of experience levels across identity and network device management. See table 1 for a full breakdown of subjects by experience level.

## 4 Results

The results answer RQ1 in the affirmative: Copilot does offer speed and accuracy improvements in common IT scenarios. Across all tasks, we measure a 34.53%*** increase in accuracy and a 29.79%*** reduction in task completion time. In the subsections on accuracy and speed below, we present detailed results answering RQ1-RQ4.

### 4.2 Speed

We measure statistically significant time savings in all scenarios. Table 3 summarizes these results by task type and scenario. The “Aggregate Results, All Content (Holding accuracy constant)” numbers represent our headline speed improvements, a 29.79%*** time savings for Copilot users. We also note that Copilot generally involves greater latency than typical page-loading in the admin centers, which is common among GAI tools. This necessarily slowed the Copilot users. Product improvements should reduce this latency and further increase the time savings for users with Copilot.

| Overall | Treatment | Control | Improvement

Aggregate Results | | | |

All Content | 53.84 | 44.29 | 63.10 | -29.79%***

SATA | 17.81 | 15.25 | 20.14 | -24.28%***

MC | 10.91 | 12.02 | 9.60 | 25.13%***

MC (Drop first) | 7.17 | 7.47 | 6.71 | 10.92%

Free Response | 30.98 | 17.03 | 43.82 | -61.14%***

Sign-In Troubleshooting | | | |

All Content | 24.04 | 16.63 | 30.46 | -45.41%***

SATA | 6.41 | 5.04 | 7.63 | -33.98%***

Free Response Content | 17.37 | 11.59 | 22.40 | -48.23%***

Device Policy Management | | | |

All Content | 20.00 | 17.45 | 22.23 | -21.64%***

MC | 10.91 | 12.02 | 9.60 | 25.13%***

MC (Drop first) | 7.17 | 7.47 | 6.71 | 10.92%

Free Response Content | 12.17 | 5.43 | 19.27 | -71.80%***

Device Troubleshooting | | | |

SATA | 11.24 | 10.21 | 12.11 | -15.69%***

Table 3: Comparison of task duration (minutes) by task type, holding accuracy constant

We measure time savings for Copilot across all three scenarios. The time savings included 45.41%*** for sign-in troubleshooting, 21.64%*** for device policy management, and 15.69%*** for device troubleshooting.

Figure 2: The difference in time spent between Copilot and control users is greatest on the first MC question, suggesting a possible learning phenomenon.

We find that Copilot’s time savings track its accuracy improvements across task types. That is, the biggest gains again come from free response (61.14%*** time savings), followed by SATA (24.28%***). Lastly, the MC questions show a decrease in speed for Copilot users (25.13%*** time increase). For these MC questions we also estimate a negative relationship between accuracy and time spent for the control group. This is likely because control subjects who knew the answer responded quickly, and those who did not know it gained little accuracy by spending more time by, for example, searching the internet. This can explain the data we see, where even though Copilot provided an advantage in responding accurately, controlling for accuracy in calculating its time savings does not make up for the disadvantage of its latency.

We also notice that Copilot users spent much more time than the control users on the first MC question compared to the subsequent three questions (see figure 2). We think this suggests Copilot users were learning how to use Copilot for this type of task. Therefore, we drop the first question and recompute the time differences across the remaining tasks, which results in no statistically significant difference in task completion time (see the “Drop first” entry in table 3).

Figure 3: Copilot’s effect heterogeneity by experience level on speed, holding accuracy constant

As we did with accuracy, we examine time savings across different experience levels and domains. Our findings once again reveal statistically significant point estimates of time savings for all groups, as shown in figure 3. However, we do not observe statistically significant differences in estimated effect sizes between any of the groups. The left panel illustrates Copilot’s impact on task completion times for the sign-in troubleshooting task, with a 50.1%*** and 45.2%*** time savings for Copilot users with 0-3 and 3+ years of experience, respectively. These point estimates are consistent with intuition and our accuracy findings.

The right panel of figure 3 shows time savings of 12.6%*** and 30.2%*** for 0-3 and 3+ years of experience, respectively, for device management tasks. This suggests that experienced network management subjects benefit more from Copilot in terms of time savings than their less experienced counterparts, which runs counter to intuition and expectation. This is driven by the fact that less experienced control subjects are driving the negative relationship between task completion time and accuracy for the device policy management MC question, mentioned above. This shrinks the possible time savings for the less experienced Copilot subjects.

## Appendix B Calculating Time Savings Holding Accuracy Constant

Note that a simple regression of time on treatment and score assumes the returns to additional time spent are the same between the treatment and control groups, which is unlikely true. Therefore, to compare speed holding accuracy constant, we employ a linear regression framework that proceeds in three steps. First, we estimate the task duration as a function of accuracy for the control group. Then we predict the task duration the control group would need to achieve the same accuracy as the Copilot group. Finally, we compute the difference between task durations for the two groups, using a bootstrap to compute the level of certainty of this finding.

Let T_{g} and A_{g} represent the set of task durations and accuracy scores for each subject in group g\in\{(t)reatment,(c)ontrol\} . And let \bar{T}_{g} and \bar{A}_{g} represent the sample means. We then estimate the effect of accuracy on task duration for the control group via the following regression and ordinary least squares.

T_{c}=\alpha+\beta A_{c}+\epsilon | |