# The Agents on Rails Files: designing black-box evals for coding agents

DevFeed: [The Agents on Rails Files: designing black-box evals for coding agents](<https://devfeed.tech/articles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents-92955.md>)

Original publisher: [Read original article](<https://evilmartians.com/chronicles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents>)

Author: Travis Turner (richardturner@evilmartians.com)

Published: 2026-10-21T00:00:00Z

Content type: article

Language: en

Sources: [Evil Martians](<https://devfeed.tech/sources/evil-martians.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Ruby on Rails](<https://devfeed.tech/topics/ruby-on-rails.md>), [AI in frontend development](<https://devfeed.tech/topics/ai-in-frontend-development.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Application Development](<https://devfeed.tech/topics/application-development.md>), [gpt-6-astra](<https://devfeed.tech/topics/gpt-6-astra.md>)

Tags: [agentic-development](<https://devfeed.tech/tags/agentic-development.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [apps](<https://devfeed.tech/tags/apps.md>), [browser](<https://devfeed.tech/tags/browser.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [designing](<https://devfeed.tech/tags/designing.md>), [dx](<https://devfeed.tech/tags/dx.md>), [e2e](<https://devfeed.tech/tags/e2e.md>), [evals](<https://devfeed.tech/tags/evals.md>), [llm-benchmarking](<https://devfeed.tech/tags/llm-benchmarking.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [mutation-testing](<https://devfeed.tech/tags/mutation-testing.md>), [rails](<https://devfeed.tech/tags/rails.md>), [robust](<https://devfeed.tech/tags/robust.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tricks](<https://devfeed.tech/tags/tricks.md>), [ui](<https://devfeed.tech/tags/ui.md>), [ui-ux](<https://devfeed.tech/tags/ui-ux.md>), [ux](<https://devfeed.tech/tags/ux.md>), [verification](<https://devfeed.tech/tags/verification.md>)

## AI overview

For coding agent evals, write black-box checks against product requirements and calibrate them so varied valid interfaces pass while broken behavior fails. The article describes Campfire Bots, where a hybrid white-box suite sped development before browser E2E checks; equivalent UI mutants tested tolerance for harmless markup changes, while faulty mutants checked detection of broken requirements. For greenfield Quizzy, the authors combined accessibility requirements with a few HTML/CSS mockups to give deterministic tests stable anchors without fully specifying the interface. Browser-test techniques include waiting for form requests to complete before navigation, replaying authenticated write traffic as an anonymous user to check authorization, and manually closing WebSockets because CDP offline mode leaves persistent connections open. The authors recommend production-like runs, diverse model trials, mutation calibration, and reviewing helper timeouts. In their reported trials, Rust cost more than Rails on Campfire Bots, largely because its Rails-shaped port had about 13 times more code; frontier models solved both Quizzy stacks, while cheaper models solved Rails but not Rust. These

## Source excerpt

Tips and tricks on writing robust evals for coding agents that test apps whose code we've never seen: different UI/UX contract strategies, UI mutation testing, browser-based E2E verification gotchas.