# Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR

DevFeed: [Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR](<https://devfeed.tech/articles/closing-the-sim-to-real-gap-an-evaluation-framework-for-autonomous-cyber-defense-configuration-of-commercial-edr-94500.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.08168>)

Author: Kerri Prinos, Lilianne Brush

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Endpoint Security & XDR](<https://devfeed.tech/topics/endpoint-security-xdr.md>), [Agentic SOC](<https://devfeed.tech/topics/agentic-soc.md>)

Tags: [active-directory](<https://devfeed.tech/tags/active-directory.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [crowdstrike](<https://devfeed.tech/tags/crowdstrike.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cyber](<https://devfeed.tech/tags/cyber.md>), [cyber-defense](<https://devfeed.tech/tags/cyber-defense.md>), [defense](<https://devfeed.tech/tags/defense.md>), [edr](<https://devfeed.tech/tags/edr.md>), [endpoint-detection-and-response](<https://devfeed.tech/tags/endpoint-detection-and-response.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [first](<https://devfeed.tech/tags/first.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gap](<https://devfeed.tech/tags/gap.md>), [hardening](<https://devfeed.tech/tags/hardening.md>), [language-model](<https://devfeed.tech/tags/language-model.md>)

## AI overview

The framework evaluates autonomous defense agents hardening commercial EDR in a setting where the EDR itself makes vendor-specific autonomous decisions. It is instantiated in a Game of Active Directory lab with Horizon3's NodeZero as the pentester and Microsoft Defender XDR as the primary EDR, then extended to CrowdStrike Falcon and SentinelOne Singularity. A sample benchmark uses Claude Sonnet 4.6 and Cisco Foundation-Sec-8B. The authors identify three evaluation challenges: commercial EDR telemetry is designed for SOC workflows rather than measuring policy effectiveness; per-policy attribution is needed to distinguish agent actions from EDR actions; and results vary across runs, largely because containment varies and accumulates. The abstract motivates methods that account for these black-box behaviors when benchmarking enterprise defense agents.

## Source excerpt

arXiv:2606.08168v2 Announce Type: replace Abstract: Leading commercial endpoint detection and response (EDR) products have shifted from operator-configured rule sets to multi-component systems where autonomous AI components operate alongside, and increasingly in place of, operator-deployed policies. Autonomous defense agents using commercial EDR as their hardening tool are no longer tuning a passive tool, but a black-box autonomous system capable of making vendor-specific decisions. We present the first evaluation framework for autonomous defense agents hardening commercial EDR. We instantiate it in a Game of Active Directory (GOAD) lab with Horizon3's NodeZero as the autonomous pentester and Microsoft Defender XDR as the primary EDR under test. We extend the same framework to CrowdStrike Falcon and SentinelOne Singularity to test whether our Microsoft Defender XDR findings hold across vendors. We run a sample benchmark of defense agents with two large language model (LLM) backbones (Claude Sonnet 4.6 and Cisco Foundation-Sec-8B). We report three lessons learned that neither simulation nor open source EDR evaluation can surface: (i) commercial EDR telemetry is engineered for Security Operations Center (SOC) analyst workflows rather than for measuring policy effectiveness; (ii) per-policy attribution is important for separating the defense agent's actions from the EDR's autonomous actions; and (iii) EDR behavior varies between runs, largely because containment varies and accumulates. Together, these findings highlight a sim-to-real gap for enterprise defense and motivate evaluation methodology for benchmarking autonomous defense agents in environments with black-box, autonomous tools.