# What 50,000 Runs of a 5-Line Eval Taught Us

DevFeed: [What 50,000 Runs of a 5-Line Eval Taught Us](<https://devfeed.tech/articles/what-50-000-runs-of-a-5-line-eval-taught-us-4058.md>)

Original publisher: [Read original article](<https://code.visualstudio.com/blogs/2026/06/19/what-50000-runs-taught-us>)

Author: Julia Kasper

Published: 2026-06-19T00:00:00Z

Content type: article

Language: en

Sources: [Visual Studio Code - Code Editing. Redefined.](<https://devfeed.tech/sources/visual-studio-code-code-editing-redefined.md>)

Topics: [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blog](<https://devfeed.tech/tags/blog.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cost](<https://devfeed.tech/tags/cost.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [tool](<https://devfeed.tech/tags/tool.md>), [vs-code](<https://devfeed.tech/tags/vs-code.md>)

## AI overview

The article examines more than 50,000 runs of a minimal VS Code agent evaluation that asks a model to write text to a file. It argues that stable, simple evaluations reveal differences in model efficiency, tool use, reliability, and cost.

## Source excerpt

How AI coding models calibrate effort, token cost, and tool use on even the simplest task, and what that means for model selection and cost. Read the full article