1SOURCE EXTRACT — arxiv-2511.13860v12Title: Randomized Controlled Trials for Phishing Triage Agent3Authors: James Bono (Microsoft Corporation)4Bibliographic record: arXiv:2511.13860v1 [econ.GN], version 1 submitted 17 Nov 2025. Abstract page: https://arxiv.org/abs/2511.13860v15Retrieved: https://arxiv.org/html/2511.13860 on 2026-10-30T04:20:39Z (SHA-256 of the retrieved HTML: 9a25378878d5735130154b510c032744445d4089e70415e3f1c3b2ca066b984a)6License: CC BY 4.0 (license shown on the arXiv abstract page; https://creativecommons.org/licenses/by/4.0/). Reproduced in part with attribution.7Extract contents: title, author and abstract; Section 2 'Methodology' (arms, subjects, queue construction); Section 3 'Results' opening and Section 3.1 'Productivity Gains: True Positives per Analyst Minute' (Table 1); Section 3.3 'Behavior: Effect of Agent on Analyst Behavior' (Table 3 'Comparison of Email Processing Times'); Appendix B 'TP per Analyst Minute'.8Omitted: Section 1, Sections 3.2 and 3.4 onward (Tables 2 and 4), Section 4, Appendices A, C and D, all figures, the references.9Rendering note: converted from the HTML rendering to plain text. Tables are rendered one row per line with " | " separating cells; in regression tables the line following a coefficient row holds the standard errors in parentheses, exactly as printed. Mathematical expressions appear as LaTeX source; non-breaking spaces were replaced by ordinary spaces. Figures are not reproduced: a value that appears only in a figure is not in this extract. No numbers were altered.10================================================================1112# Randomized Controlled Trials for Phishing Triage Agent1314James Bono15^†^†thanks: james.bono@microsoft.com; This work would not be possible without the support of the following contributors: Eden Abadi, Costas Boulis, Michael Browning, Jael Fafner, Corina Feuerstein, Justin Grana, Paula Greve, Cristina Da Gama Henriquez, Adel Khaldi, Amit Kollman, Aviel Lavie, Katerina Marazopoulou, Hani Neuvirth-Telem, Inbar Rotem, Felicity Scott-Milligan, Cole Sims, Holly Stewart, and Tracy Zhen1617Microsoft Corporation1819October 20252021###### Abstract2223Security operations centers (SOCs) face a persistent challenge: efficiently triaging a high volume of user-reported phishing emails while maintaining robust protection against threats. This paper presents the first randomized controlled trial (RCT) evaluating the impact of a domain-specific AI agent—the Microsoft Security Copilot Phishing Triage Agent—on analyst productivity and accuracy. Our results demonstrate that agent-augmented analysts achieved up to 6.5 times as many true positives per analyst minute and a 77% improvement in verdict accuracy compared to a control group. The agent’s queue prioritization and verdict explanations were both significant drivers of efficiency. Behavioral analysis revealed that agent-augmented analysts reallocated their attention, spending 53% more time on malicious emails, and were not prone to rubber-stamping the agent’s malicious verdicts. These findings offer actionable insights for SOC leaders considering AI adoption, including the potential for agents to fundamentally change the optimal allocation of SOC resources.2425## 2 Methodology2627We conduct an RCT with three arms, randomly assigned:28291.3031Control ( N{=}52 ): Random queue order, no references to AI.32332.3435Aware ( N{=}56 ): Queue ordered by the agent’s verdicts; participants see the agent’s decision, a brief rationale, and an accuracy calibration.36373.3839Blind ( N{=}59 ): Identical ordering as Aware but without any AI references; neither verdicts nor rationales are revealed.4041Our subjects are 167 external security analysts.^1^14214344We filter out 13 subject responses that scored less than 40% – significantly below guessing percentage. These included three in the Aware group, six in the Blind group, and four in the Control group. This step does not affect our qualitative results. Their task is to triage a 25-email queue. The queue comes from an original corpus of 93 randomly selected emails reported by Microsoft employees. Through an in-depth analysis by expert graders and security researchers, we arrived at our ground truth of 82 non-malicious and 11 malicious emails.4546On this corpus, the agent’s precision and recall were both 100%. The agent’s accuracy is very good, but it is not always perfect. It will occasionally have false negative (FN) and false positive (FP) verdicts. However, without FNs and FPs in our corpus, we cannot measure the effect of the agent on analyst trust and diligence. Hence, we create synthetic false positives (SFPs) and synthetic false negatives (SFNs). We create SFNs from each of the malicious emails by changing the agent verdict and swapping the justification for a generic justification of a non-malicious verdict that fits the content of the malicious email. For example, for a malicious email impersonating Microsoft 365, we used the justification from a non-malicious email from Microsoft 365. We create fake false negatives (SFNs) with a similar technique, e.g., by using the justification from a non-malicious email from Microsoft 365 for a malicious email impersonating Microsoft 365. In both cases, we ensure no details in the fake justification are in obvious conflict with the SFN and SFP emails. Using this method, we ended up with five SFPs and 10 SFNs.4748To construct each subject’s 25 email queue, we randomly select four malicious emails, 19 non-malicious emails, one SFN, and one SFP. This sampling scheme mimics agent precision and recall of 80%, which aligns closely to Microsoft’s internal measurements on large random samples. It also includes 20% malicious samples.^2^24925051Our results are not sensitive to this malicious percentage, since we calculate the time to triage for TP, TN, SFP, and SFN for all groups separately. We need only that the malicious percentage does not dramatically affect the expectations of the Control group, which 20% should not do. For the Aware and Blind groups, we randomly order the four malicious and one SFP at the front of the queue, i.e., in positions one through five. We then randomly order the 19 non-malicious and one SFN in queue positions six through 25. This implies agent accuracy with precision=recall=0.8. Each email is presented as a timed, forced-choice classification: Malicious vs. Non-Malicious. Responses are final (no backtracking).5253To reduce the importance of learning effects and to control for the heterogeneous beliefs of subjects using the agent for the first time, we calibrate our Aware group’s beliefs about the agent’s accuracy with the following information: “On a very large sample, the agent’s accuracy is approximately as follows: Malicious verdicts are 80% correct; Not Malicious verdicts are 95% correct.”5455For each email in the queue, subjects have a link to a read-only OneDrive folder with the email’s artifacts, including a screenshot (annotated with primary URL), EML and HTML text, URL list, URL detonation screenshots, and sanitized attachments.^3^35657## 3 Results5859The results show that the agent offers dramatic improvements in both productivity and protection. These improvements are driven by both the agent’s ability to prioritize the queue as well as its verdicts and explanations. In addition, the agent’s accuracy and its effect on analyst accuracy both strongly support implementing the resolve-benign protocol.6061### 3.1 Productivity Gains: True Positives per Analyst Minute6263The Aware group identifies 6.5 times as many TPs per analyst minute (550% productivity gain) on the corpus ground truth. This number reflects the resolve-benign protocol.64Under the pessimistic counterfactual, analysts identify 3.1 times as many TPs per minute. At a high malicious rate of 30%, we still find an effect of 2.0-2.6 times. These measurements are all statistically significant above the 99% level. Figure 1 illustrates the relationship between the malicious rate and the agent’s productivity gain (as a factor improvement) for both the case where agent precision = 1.0 and the case where agent precision = 0.8. The shaded region is the 99% confidence interval for the case where agent precision = 0.8.6566Figure 1: Factor Increase in TPs per unit of Analyst Time6768Next, by applying the resolve-benign protocol to the Blind group’s responses, we measure how much of this gain is due to the resolve-benign protocol versus the subject being aware of the agent, including its verdicts and explanations. We find that the resolve-benign protocol, which dramatically reduces the number of non-malicious emails for review, is the biggest factor driving the productivity gain, accounting for between 79-84% of the total agent-driven productivity gain. As precision decreases, the information conveyed in the agent’s verdicts and explanations becomes relatively more important. However, even when the agent’s precision is maximized, the information conveyed in the agent’s verdicts and explanations remains relatively important, about 13-17% of the total improvement.6970Table 1: Productivity Improvements by Agent Feature7172Scenario | Factor Increase | Resolve-Benign (%) | Agent Awareness (%)7374Corpus ground truth | 6.53 | 83 | 177576Accuracy counterfactual | 5.04 | 78 | 227778Malicious rate counterfactual | 3.97 | 87 | 137980Pessimistic counterfactual | 3.07 | 83 | 178182### 3.3 Behavior: Effect of Agent on Analyst Behavior8384Next, we look at subject behavior across the entire queue, not just the agent-prioritized portion, i.e., this is how analysts would perform if we removed the resolve-benign protocol. First, we find that the Aware group spends 53% more time per malicious email. They spend marginally less on non-malicious email, and the overall time is roughly the same. This indicates they are utilizing the agent to help optimize their time and attention.8586Table 3: Comparison of Email Processing Times8788Metric | Malicious | Non-Malicious | Full Corpus8990Aware Time per Email (s) | 2.8 | 1.8 | 50.59192Control Time per Email (s) | 1.9 | 2.4 | 56.99394% Time Savings | -52.9 | 23.6 | 11.19596p-value | 0.006 | 0.126 | 0.4719798We also find that the Aware group is no more likely to confirm the agent’s TPs or miss the agent’s SFPs, which are precisely the samples the Aware group would see under the resolve-benign protocol. Taken together with the evidence that Aware subjects spend more time on malicious emails, this means they are not rubber-stamping the agent’s malicious verdicts. Rather, they use the agent’s insights to guide their own investigation.99100However, we do find they are more likely to confirm agent TNs by 21 percentage points and more likely to miss agent SFNs by 29 percentage points. This lends further support to the use of the resolve-benign protocol. In addition to the productivity gains documented above, the protection benefit of having analysts spend time on emails the agent has already classified as non-malicious is minimal because analysts are likely to agree with the agent anyway.101102## Appendix B TP per Analyst Minute103104Let p denote the malicious email rate in the corpus. We define the following parameters and variables:105106### B.1 Control Group Performance107108For the control group, the expected time to process an email is:109110\mathbb{E}[T_{c}]=p\cdot\bar{T}_{c,m}+(1-p)\cdot\bar{T}_{c,b} | |111112where:113114•115116\bar{T}_{c,m} is the mean time (in seconds) for control group to process malicious emails117118•119120\bar{T}_{c,b} is the mean time (in seconds) for control group to process benign emails121122The true positive rate for the control group is defined as:123124\text{TPR}_{c}=\frac{\text{TP}_{c}}{n_{m}} | |125126where \text{TP}_{c} is the number of true positives identified by control subjects and n_{m} is the total number of malicious emails in the sample.127128The productivity metric for the control group (true positives per minute) is:129130\rho_{c}=\frac{p\cdot\text{TPR}_{c}}{\mathbb{E}[T_{c}]} | |131132### B.2 Aware Group Performance (Priority Queue)133134For the Aware group processing a priority queue filtered by an agent with precision \pi and recall \sigma , the queue composition changes. The priority queue contains only items flagged by the agent as malicious. The expected time to process an item in the priority queue is:135136\mathbb{E}[T_{a}]=\pi\cdot\bar{T}_{a,\text{TP}}+(1-\pi)\cdot\bar{T}_{a,\text{FP}} | |137138where:139140•141142\bar{T}_{a,\text{TP}} is the mean time for Aware group to process agent-identified true positives143144•145146\bar{T}_{a,\text{FP}} is the mean time for Aware group to process agent-identified false positives147148The accuracy of the Aware group on agent-identified true positives is:149150\alpha_{a,\text{TP}}=\mathbb{E}[\mathbbm{1}_{\text{correct}}|\text{Agent}=\text{TP}] | |151152The productivity metric for the Aware group (true positives per minute) is:153154\rho_{a}=\frac{\pi\cdot\alpha_{a,\text{TP}}}{\mathbb{E}[T_{a}]} | |155156### B.3 Productivity Factor157158The productivity factor, representing the multiplicative improvement of the Aware group over the control group, is:159160\phi^{(p)}=\frac{\rho_{a}}{\rho_{c}} | |161162A value of \phi^{(p)}>1 indicates improved productivity, with \phi^{(p)}=2 representing a doubling of productivity (twice as many true positives identified per unit time).163