# ViBench 2.0: A Benchmark for Long-Horizon Software Building

DevFeed: [ViBench 2.0: A Benchmark for Long-Horizon Software Building](<https://devfeed.tech/articles/vibench-2-0-a-benchmark-for-long-horizon-software-building-118054.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05620>)

Author: Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong, Michele Catasta

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [bugfixes](<https://devfeed.tech/topics/bugfixes.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-engineering](<https://devfeed.tech/tags/agentic-engineering.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [browser](<https://devfeed.tech/tags/browser.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [family](<https://devfeed.tech/tags/family.md>), [fast](<https://devfeed.tech/tags/fast.md>), [feature](<https://devfeed.tech/tags/feature.md>), [full](<https://devfeed.tech/tags/full.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [gap](<https://devfeed.tech/tags/gap.md>), [harness](<https://devfeed.tech/tags/harness.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [software](<https://devfeed.tech/tags/software.md>)

## AI overview

ViBench 2.0 evaluates whether agents can build complete web products through successive feature requests, addressing benchmarks focused on single tasks. It covers 17 products and 241 requests; finished builds are judged by a browser agent running 544 test plans, and a build passes only if every plan passes. Across nine models, the three strongest pass up to 98% of plans but produce a fully working product in at most 65% of builds; no open-weight model does so. Across three releases of one model family, pass@1 on public products rose from 3% to 53% while cost per build fell by more than half. The release includes eight products, a harness and grader, with nine products held out.

## Source excerpt

arXiv:2611.05620v1 Announce Type: cross Abstract: On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.