# kw-bench: Breaking Frontier Harnesses on Real-World Tasks

DevFeed: [kw-bench: Breaking Frontier Harnesses on Real-World Tasks](<https://devfeed.tech/articles/kw-bench-breaking-frontier-harnesses-on-real-world-tasks-103128.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36405>)

Author: George Juraj Salapa

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [claude-opus-5-5](<https://devfeed.tech/tags/claude-opus-5-5.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [harness](<https://devfeed.tech/tags/harness.md>), [knowledge-work](<https://devfeed.tech/tags/knowledge-work.md>), [opencode](<https://devfeed.tech/tags/opencode.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

## AI overview

kw-bench tests investigative knowledge work by asking agents to reconstruct five years of VAT data from undocumented, inconsistent extracts spanning three generations of SAP; grading is deterministic and uses no LLM judge. In a two-hour run with DeepSeek v4.1 Flash, model-agnostic OpenCode and micro-cc scored 0.542 and 0.528, while Claude Code and Codex scored 0 away from their home model. The authors attribute the gap to difficulty retaining weak signals over long sessions. With Claude Opus 5.5, micro-cc and Claude Code scored 0.9986 and 1.000 in 22 and 25 minutes. These results suggest harness choice mattered substantially below the frontier on this task; the abstract does not establish performance across other task families.

## Source excerpt

arXiv:2610.36405v1 Announce Type: new Abstract: Each new generation of frontier model pushes the edge of what is possible. Inside a harness -- a loop, a shell, a filesystem, the ability to write and run its own code -- a model can take on increasingly difficult work. A growing body of work treats the coding-agent harness as infrastructure for knowledge work rather than a coding tool, and a new wave of benchmarks tests it there. Most of them are generative: build a financial model, a memo, a spreadsheet from a brief, judged by a rubric or by another model. Few test the expensive part of knowledge work, which is investigative. In audit, tax, due diligence, regulatory reporting, claims or any dispute over numbers, the answer already exists -- hidden in incomplete and inconsistent evidence spread across systems -- and has to be recovered and defended to the cent. We introduce kw-bench, a benchmark built from real engagements specialists described to us. Its first task family is a five-year VAT reconstruction across three generations of SAP -- legacy ECC, a partial S/4HANA migration and a consultant's redesign -- delivered as raw system extracts: page headers, local number formats, partial link tables, no documentation of how anything connects. Grading is deterministic, with no LLM judge. We ran four harnesses -- Claude Code, Codex, OpenCode and micro-cc -- on the same model, DeepSeek v4.1 Flash, for two hours. The two independent, model-agnostic harnesses delivered (OpenCode 0.542, micro-cc 0.528); the two lab harnesses, away from their home model, delivered nothing (0.00 each). What defeated them was inertia: keeping many weak signals alive across a long session. On a frontier model the task collapses. micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes. A weeks-long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.