# Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

DevFeed: [Harvesting SWE Benchmarks from Wild Vibe Coding Sessions](<https://devfeed.tech/articles/harvesting-swe-benchmarks-from-wild-vibe-coding-sessions-113527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02440>)

Author: Ao Lu, Yuanhao Zeng, Shaocheng Lan, Sijie Xu, Jiacheng Bao, Kuang Yan, Heng Lin, Kan Ren

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [commits](<https://devfeed.tech/tags/commits.md>), [container](<https://devfeed.tech/tags/container.md>), [context](<https://devfeed.tech/tags/context.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [execution](<https://devfeed.tech/tags/execution.md>), [filter](<https://devfeed.tech/tags/filter.md>), [framework](<https://devfeed.tech/tags/framework.md>), [harness](<https://devfeed.tech/tags/harness.md>), [history](<https://devfeed.tech/tags/history.md>), [languages](<https://devfeed.tech/tags/languages.md>), [logs](<https://devfeed.tech/tags/logs.md>), [model](<https://devfeed.tech/tags/model.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [vibe-coding](<https://devfeed.tech/tags/vibe-coding.md>)

## AI overview

SessionFarmer turns coding-agent conversations and repository checkpoints into executable software engineering tasks by filtering unrelated sessions, grouping work around engineering goals, reconstructing hermetic containers, and creating context-based task statements, rubrics, and tests. Its closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptance of incomplete solutions. The resulting CropChat benchmark contains 242 tasks from 61 public repositories across 14 languages; CropEnterprise is built from proprietary logs. In a CropEnterprise audit, original developers endorsed the vast majority of reconstructed episodes, statements, and rubrics. Full-episode context captured substantially more user requirements than single-turn or single-session inputs. Test refinement improved balanced accuracy against reference patch labels by 7-9 percentage points across solvers. On CropChat, the strongest of 12 model-harness configurations reached 39.3% Pass@1, while harness choice shifted results by up to 8.4 points for the same model.

## Source excerpt

arXiv:2611.02440v1 Announce Type: new Abstract: Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.