# SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud Incidents

DevFeed: [SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud Incidents](<https://devfeed.tech/articles/sweops-bench-evaluating-agents-for-end-to-end-resolution-of-software-induced-cloud-incidents-118066.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06099>)

Author: Danwen Chen, Guangba Yu, Pengfei Chen

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [AIOps for Kubernetes](<https://devfeed.tech/topics/aiops-for-kubernetes.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [bugfixes](<https://devfeed.tech/topics/bugfixes.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-operations](<https://devfeed.tech/tags/cloud-operations.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [defects](<https://devfeed.tech/tags/defects.md>), [detection](<https://devfeed.tech/tags/detection.md>), [development](<https://devfeed.tech/tags/development.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fail](<https://devfeed.tech/tags/fail.md>), [functional](<https://devfeed.tech/tags/functional.md>), [functional-verification](<https://devfeed.tech/tags/functional-verification.md>), [gap](<https://devfeed.tech/tags/gap.md>), [harness](<https://devfeed.tech/tags/harness.md>), [software](<https://devfeed.tech/tags/software.md>)

## AI overview

SWEOps-Bench evaluates whether agents can resolve software-induced cloud incidents end to end, combining runtime investigation with source or configuration repair. It provides 55 reproducible tasks based on confirmed defects in 16 Kubernetes operator projects: each starts with a healthy cluster, then a controlled workload triggers the historical fault. Agents submit repairs through a terminal interface, and a harness builds and deploys them under a shared time budget. Across six agent-model configurations, faulty-state detection was 72.7%-90.9%, but verified overall recovery was only 14.5%-52.7%. The evaluation also measures stage-wise accuracy, lifecycle latency with penalties, and token use. Reported root-cause diagnoses only partly matched recovery outcomes: some tasks recovered despite an incorrect diagnosis, while clearing runtime symptoms did not always pass fix-specific functional verification.

## Source excerpt

arXiv:2611.06099v1 Announce Type: new Abstract: As autonomous agents advance, the traditional boundary between software development and cloud operations gradually blurs, increasing the need for integrated incident detection, localization, and remediation. Existing benchmarks evaluate isolated diagnostic queries or repository-level program repair, failing to test continuous incident resolution across operational clusters and source repositories. We introduce SWEOps-Bench, an interactive benchmark comprising 55 reproducible incident tasks derived from confirmed historical defects across 16 Kubernetes operator projects. Each task preserves the original software repository and initializes in a healthy cluster before a controlled workload triggers the historical fault. Agents investigate runtime evidence and submit source or configuration repairs through a terminal interface, while the harness builds and deploys the changes within a shared time budget. Evaluation covers stage-wise accuracy, verified recovery, penalized lifecycle latency, and token consumption. Across six agent-model configurations, faulty-state detection ranges from 72.7% to 90.9%, whereas overall recovery ranges from only 14.5% to 52.7%, exposing a substantial gap between recognizing incidents and resolving them end to end. Trajectory analysis further shows that reported diagnosis and final recovery only partially coincide: some tasks achieve verified recovery despite an incorrect reported RCA, while runtime symptom clearance can still fail fix-specific functional verification. SWEOps-Bench establishes a rigorous foundation for evaluating autonomous agents in integrated development and operational environments.