# eval Published articles for eval. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How sandbox boundaries affect AI coding agent evaluations DevFeed: [How sandbox boundaries affect AI coding agent evaluations]() Original publisher: [Read original article]() Author: Waldek Mastykarz Published: 2026-09-16T09:09:51Z Content type: opinion Language: en Sources: [Developer Blogs]() Topics: [AI-assisted coding](), [Testing](), [Artificial Intelligence](), [coding]() Tags: [agent-experience](), [ai](), [ai-coding](), [ax](), [coding](), [developers](), [eval](), [evaluation](), [measurement](), [sandbox]() ### AI overview The article explains that an AI coding agent evaluation can produce misleading scores when the agent retrieves answers from its prompt, environment, or other accessible resources. It argues that evaluators should define the capability being tested and set information boundaries accordingly, including restricting access to evidence when measuring internal model knowledge. ### Source excerpt Your AI coding agent passed the eval. But did the model know the answer, or did it find it somewhere on your machine? A correct answer can still invalidate your measurement. The post Your AI coding agent evaluation is only as good as its sandbox appeared first on Microsoft for Developers. ## Debugging our AI search assistant with agent tracing DevFeed: [Debugging our AI search assistant with agent tracing]() Original publisher: [Read original article]() Author: Dominik Buszowiecki; Shaun Kaasten Published: 2026-09-11T09:00:00Z Content type: article Language: en Sources: [Sentry Blog]() Topics: [AI search](), [agent observability](), [debugging](), [tracing](), [Large Language Model]() Tags: [agent](), [ai](), [api](), [debugging](), [errors](), [eval](), [llm](), [search](), [sentry](), [tracing]() ### AI overview Sentry engineers describe how they debugged the Search Query Assistant, which converts natural-language prompts into Sentry Syntax queries. They used evals for performance measurement and AI Conversation tracing to investigate failures, including a bug involving custom numerical attributes that caused queries to return no results. ### Source excerpt See how Sentry engineers used AI Conversations to debug a natural language search assistant and fix a tricky query generation bug. ## What's in a tag name? JavaScript, apparently DevFeed: [What's in a tag name? JavaScript, apparently]() Original publisher: [Read original article]() Author: Gareth Heyes Published: 2026-08-25T14:24:32Z Content type: article Language: en Sources: [PortSwigger Research]() Topics: [JavaScript](), [browser](), [HTML](), [Developer Tools](), [Web Development]() Tags: [browser](), [eval](), [event](), [html](), [javascript](), [payload](), [vector]() ### AI overview The article explores unusual HTML tag-name and attribute behaviors that can be combined with JavaScript features to create XSS vectors and potentially bypass web application firewalls. It discusses case transformations, event handlers, the Function constructor, eval, focusability, and alternative HTML attributes across browsers. ### Source excerpt I was on my laptop, as I often am when there's rubbish on telly, and found myself wondering what characters are allowed in a tag. I knew they had to begin with "a-zA-Z", but what about after that? I t ## One AI Output Is an Example, Not an Evaluation DevFeed: [One AI Output Is an Example, Not an Evaluation]() Original publisher: [Read original article]() Author: Raluca Budiu Published: 2026-08-14T17:00:00Z Content type: article Language: en Sources: [NN/g latest articles and announcements]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [LLM evaluation / benchmarking](), [benchmarking](), [User experience (UX)]() Tags: [ai](), [article](), [benchmarking](), [confidence-interval](), [eval](), [evaluation](), [language-models](), [llm](), [metrics](), [nondeterminism](), [performance](), [research](), [statistical-significance](), [usability]() ### AI overview One AI output is only an example, not a reliable evaluation. Because AI systems can produce different results from the same input, teams should assess them with multiple representative inputs, repeated runs, quantitative metrics, and confidence intervals. ### Source excerpt One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. ## Eval-driven development: Lessons from evaluating GenAI at scale DevFeed: [Eval-driven development: Lessons from evaluating GenAI at scale]() Original publisher: [Read original article]() Author: Rohit Girme Published: 2026-07-28T17:01:03Z Content type: article Language: en Sources: [The Airbnb Tech Blog - Medium]() Topics: [Generative AI](), [Large Language Model](), [Software Engineering](), [Artificial Intelligence](), [Tooling]() Tags: [agentic-ai](), [ai](), [article](), [development](), [engineering](), [eval](), [evals](), [evaluation](), [genai](), [generation](), [generative-ai](), [llm](), [software](), [software-testing](), [testing](), [tooling](), [workflows]() ### AI overview This article presents eval-driven development as a core engineering discipline for trustworthy Generative AI products. It explains why evaluating LLM systems is difficult, including non-deterministic outputs, subjective correctness, AI-based evaluation risks, and failures across retrieval, reasoning, tool calls, and generation. It shares foundational evaluation practices and cautions that teams should plan evaluation early and ground success criteria in their data. ### Source excerpt How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.Nestled into the lush hillside, this stunning modern retreat features striking natural wood architecture, terraced balconies, and a serene landscape. By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly Introduction Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and "correct" is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently. At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what's working, guiding where we improve the product next. Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb. In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals. 1. Foundation Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen: False confidence: A generic "helpfulness" metric scores well, you ship, ## How we brought agentic workflows to Cloud SIEM with the Datadog MCP Server DevFeed: [How we brought agentic workflows to Cloud SIEM with the Datadog MCP Server]() Original publisher: [Read original article]() Author: Chelsea Xu; Eddie Cai; Romain Kirszbaum; Mohamed Hachem Ouertani Published: 2026-07-17T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [SIEM, Security](), [MCP Server](), [Model Context Protocol](), [Security & compliance, Cloud security](), [real user monitoring]() Tags: [agent-observability](), [agentic](), [ai](), [ai-impact](), [cloud-security](), [context-window](), [eval](), [mcp](), [mcp-server](), [real-user-monitoring](), [security](), [tool](), [workflows]() ### AI overview This article explains how Datadog built MCP tools for Cloud SIEM to support agentic security workflows. It covers tool scoping based on user behavior, progressive disclosure for managing a shared context window, custom evaluation of non-deterministic agent behavior, and governance of a growing multi-team toolset. ### Source excerpt See how we built MCP tools for Cloud SIEM, using usage data, progressive disclosure, and a custom eval framework to keep a multi-team agentic toolset reliable. ## From weeks to a day: how we made LLM evaluation fast enough to iterate on DevFeed: [From weeks to a day: how we made LLM evaluation fast enough to iterate on]() Original publisher: [Read original article]() Author: Baharak Saberidokht Published: 2026-07-14T17:01:03Z Content type: article Language: en Sources: [The Airbnb Tech Blog - Medium]() Topics: [LLM evaluation / benchmarking](), [Large Language Model](), [Software Engineering](), [data](), [Machine learning]() Tags: [ai](), [data](), [engineering](), [eval](), [evaluation](), [infrastructure](), [llm](), [models](), [software-engineering](), [techniques](), [technology](), [validation]() ### AI overview The article explains how Airbnb made production LLM evaluation fast enough for rapid iteration by addressing infrastructure challenges across four layers. It focuses on diagnosing evaluation noise from data and judging uncertainty, validating complete system paths, and applying classical software engineering techniques to make measurements more trustworthy. ### Source excerpt Training an LLM is the easy part. The hard part is designing experiments and evaluations that you can trust enough to know whether the new model is actually an improvement. By: Baharak Saberidokht Introduction Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques. At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we've made; the fourth is the integration layer that ties the rest together -- the one that is easiest to overlook, because each individual component looks fine in isolation. The approach rests on two observations: the seams are where things break, and finding those breaks requires exercising the full path, not just validating each component in isolation. Figure 1. The four layers of the production LLM stack. Bounded model mutation requires trustworthy measurement, and end-to-end validation requires the eval foundation to be fast enough to run on the combined path. Layer 1: Name it before trying to remove it Layer 1 is diagnostic framing of evaluation noise. This layer addresses two different sources of indeterminacy: data and judging uncertainty. Classical ML metrics are deterministic: F1, BLEU, and accuracy return the same number on the same input. With LLMs in the evaluation loop, that assumption dies. Judges score identical inputs differently across runs, and LLM-generated references regenerate as different strings. A two percent score movement can mean the model improved, the judge drifted, the references shifted, or some combination. We cannot tell which without naming which kind of noise we are looking ## Hugging Face Models on Foundry Managed Compute DevFeed: [Hugging Face Models on Foundry Managed Compute]() Original publisher: [Read original article]() Author: Manoj Bableshwar; Osi Published: 2026-07-07T15:20:06Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Microsoft](), [AI Development](), [Artificial Intelligence](), [Cloud](), [SDKs](), [Open Source](), [C#](), [JavaScript](), [Python](), [anthropic](), [deepseek](), [OpenAI]() Tags: [agentic-ai](), [agents](), [ai](), [azure](), [c-sharp](), [deployment](), [eval](), [gpu](), [javascript](), [microsoft](), [microsoft-foundry](), [monitoring](), [observability](), [open-source](), [orchestration](), [platform](), [python](), [sdks](), [security]() ### AI overview Microsoft Foundry Managed Compute is presented as a managed GPU platform for deploying open-source and custom models within Foundry. It uses model-oriented configuration while Microsoft manages GPU topology, runtime updates, and security patches across supported runtimes. The platform shares endpoints, SDKs, authentication, observability, and billing with Foundry's other deployment options, while integrating open-source models with Foundry Agents. ### Source excerpt Microsoft Foundry is a platform for building and operating agentic AI applications. Foundry starts with the widest model selection on any cloud -- models from Microsoft, OpenAI, Anthropic, Meta, Mistral, DeepSeek, Hugging Face, and others, spanning frontier, open-source, and custom weights -- all accessible through a single endpoint and a single set of SDKs in Python, C#, JavaScript, and Java. ## The good, the bad, and the AI apps DevFeed: [The good, the bad, and the AI apps]() Original publisher: [Read original article]() Author: Phoebe Sajor Published: 2026-07-03T07:40:00Z Content type: article Language: en Sources: [Stack Overflow Blog]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [App](), [Generative AI](), [Open Source]() Tags: [ai](), [apps](), [eval](), [evaluation](), [generative](), [generative-ai](), [open-source](), [podcast](), [se-stackoverflow](), [se-tech]() ### AI overview Ryan welcomes Benny Chen, co-founder of Fireworks AI, to discuss what makes an AI application good or bad, how qualitative signals and quantitative metrics can be balanced in AI evaluation, and how open-source protocols and community efforts are shaping evaluation standards. ### Source excerpt Ryan welcomes Benny Chen, co-founder of Fireworks AI, to the show to explore what actually makes an AI application good or not, how to balance qualitative signals with quantitative metrics when evaluating AI, and how open-source eval protocols and community efforts are setting the standard for AI evaluation. ## What an AI Agent Failure Revealed About Model Tradeoffs, Evals, and Traces DevFeed: [What an AI Agent Failure Revealed About Model Tradeoffs, Evals, and Traces]() Original publisher: [Read original article]() Author: Sergiy Dybskiy Published: 2026-07-01T09:00:00Z Content type: article Language: en Sources: [Sentry Blog]() Topics: [Traces](), [Artificial Intelligence](), [Prompt Engineering](), [vercel ai sdk](), [gateway]() Tags: [agent](), [ai](), [eval](), [evals](), [model](), [tool](), [tools](), [trace](), [traces]() ### AI overview An experiment with an AI-powered conference schedule builder showed that an agent could invent speakers, falsely claim to have retrieved them from an API, and appear grounded because a tool had run. The article discusses model tradeoffs, evaluation limitations, and the value of inspecting agent traces. ### Source excerpt I gave the free tier a cheaper model and it invented conference speakers who don't exist. What that taught me about model tradeoffs, evals, and reading agent traces. ## Debug and evaluate your AI app from your coding agent with Datadog Agent Observability DevFeed: [Debug and evaluate your AI app from your coding agent with Datadog Agent Observability]() Original publisher: [Read original article]() Author: Michael Bevilacqua-Linn; Till W; Tanguy Renaudie; Mehul Sonowal; Gabriele Lorenzo; Alex Barksdale Published: 2026-06-30T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [agent observability](), [AI-assisted coding](), [AI Engineering](), [observability](), [MCP](), [MCP Server](), [Command-line interface](), [LLM evaluation / benchmarking](), [datasets](), [Traces](), [debug](), [experiments]() Tags: [agent-observability](), [agent-skills](), [agents](), [ai](), [ai-engineering](), [ai-observability](), [analysis](), [api](), [claude](), [claude-code](), [cli](), [code](), [codex](), [coding](), [command-line](), [cursor](), [datasets](), [debug](), [eval](), [how-to](), [observability]() ### AI overview This article explains how to use Datadog Agent Observability from coding agents such as Claude Code, Cursor, and Codex CLI. It presents the Datadog MCP Server, Pup CLI, and Agent Skills as ways to access traces, evaluation results, experiment metrics, and other telemetry for classifying sessions, debugging production failures, creating evaluation datasets, and generating fixes. ### Source excerpt Learn how to give your coding agent access to Datadog Agent Observability data to classify failures, run RCA, bootstrap evaluators, and generate fixes. ## LLM-судья для нейроразбора резюме на hh DevFeed: [LLM-судья для нейроразбора резюме на hh]() Original publisher: [Read original article]() Author: Mentalitet (hh.ru) Published: 2026-06-22T05:31:17Z Content type: article Language: ru Sources: [HeadHunter RU]() Topics: [Large Language Model](), [Machine learning]() Tags: [ai](), [ai-quality](), [eval](), [hh-ru](), [hr](), [llm](), [llm-10b5a83db3c5](), [llm-as-a-judge](), [llm-as-judge](), [quality-assurance](), [tag-61cd5a476b1d](), [tag-a939ae104bb0](), [tag-b6914c0b0244](), [tag-d96e238f1419](), [tag-e7e38deac50c]() ### AI overview This article describes how hh.ru designed an LLM judge for its resume-analysis feature in an AI assistant. It explains why reliable evaluation requires its own rubrics, datasets, quality metrics, operating-cost considerations, and controls for hallucinations and interpretation errors. The system assesses candidates against criteria relevant to a specific position, which may be extracted from a vacancy or defined by an employer. ### Source excerpt Создать LLM-судью легко. Гораздо сложнее сделать так, чтобы его оценкам можно было доверять. Мы убедились в этом на практике при разработке нейроразбора резюме для ИИ-помощника hh.ru. Быстро выяснилось, что хороший LLM-судья -- это отдельный продукт со своими рубриками, датасетами, метриками качества и стоимостью эксплуатации. Меня зовут Женя Орлов, я LLM Eval Lead. В этой статье расскажу, как мы проектировали систему оценки для нейроразбора резюме, почему отказались от наивных подходов и какие выводы сделали по ходу разработки. Читать далее ## Jonathan Frankle on Infrastructure for Specifying and Evaluating Frontier AI Systems DevFeed: [Jonathan Frankle on Infrastructure for Specifying and Evaluating Frontier AI Systems]() Original publisher: [Read original article]() Author: UX Magazine Team Published: 2026-06-11T10:46:59Z Content type: opinion Language: en Sources: [UX Magazine]() Topics: [Artificial Intelligence](), [long-context](), [large-language-models](), [databricks](), [LLM evaluation / benchmarking](), [systems](), [User experience (UX)](), [multimodal](), [Prompt Engineering](), [Retrieval Augmented Generation (RAG)]() Tags: [ai](), [benchmarks](), [claude](), [context](), [databricks](), [eval](), [language-models](), [large-language-models](), [multimodal](), [systems]() ### AI overview Jonathan Frankle argues that frontier language models have advanced faster than the infrastructure needed to specify, test, and deploy them reliably. The discussion emphasizes requirements, evaluation, composable tools, context strategies, retrieval, and prompts as parameters. ### Source excerpt Jonathan Frankle on why frontier models outran our ability to specify, evaluate, and curate, and what that means for design leaders shipping agentic systems. The metaphor lands because it names the embarrassment. On his return to Invisible Machines, a conversation recorded last summer, Jonathan Frankle, Chief AI Scientist at Databricks and head of Mosaic Research, The post Nuclear Fusion, No Power Lines appeared first on UX Magazine. ## How Braintrust turns customer requests into code with Codex DevFeed: [How Braintrust turns customer requests into code with Codex]() Original publisher: [Read original article]() Published: 2026-05-29T12:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [codex](), [observability](), [Artificial Intelligence](), [Code](), [experiments](), [Command-line interface](), [Terminal]() Tags: [ai](), [codex](), [coding](), [customer](), [development](), [eval](), [experimentation](), [experiments](), [feature](), [observability](), [real-time](), [sandbox](), [speed](), [tool](), [workflow]() ### AI overview Braintrust engineers use Codex to turn customer feature requests into preview branches and working ideas within minutes. The article describes faster terminal performance, real-time customer iteration, and sandboxed experiments driven by tests. ### Source excerpt How Braintrust engineers use Codex with GPT-5.5 to run experiments and code faster. ## A New Framework for Evaluating Voice Agents (EVA) DevFeed: [A New Framework for Evaluating Voice Agents (EVA)]() Original publisher: [Read original article]() Author: Tara Bogavelli; Gabrielle Gauthier Melancon; Katrina Stankiewicz; Nifemi Bamgbose; Hoang Nguyen; Raghav Mehndiratta; Hari Subramani; Fanny Riols Published: 2026-03-24T02:01:52Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [voice ai](), [Benchmark](), [Bot](), [dataset](), [speech-to-speech](), [asr](), [Large Language Model]() Tags: [agent](), [agents](), [architecture](), [audio](), [benchmark](), [code](), [demo](), [eval](), [evaluation](), [llm](), [models](), [speech](), [speech-to-speech](), [transcription](), [voice]() ### AI overview EVA is an end-to-end framework for evaluating conversational voice agents across both task accuracy and conversational experience. It scores complete multi-turn spoken conversations, includes an airline dataset of 50 scenarios, and reports benchmark results for cascade and audio-native systems. The article highlights a recurring tradeoff: stronger task completion can coincide with worse user experience. ### Source excerpt Conversational voice agents present a distinct evaluation challenge: they must simultaneously satisfy two objectives -- accuracy (completing the user's task correctly and faithfully) and conversational experience (doing so naturally, concisely, and in a way appropriate for spoken interaction). ## Games as Model Eval: 1-Click Deploy AI Town on Fly.io DevFeed: [Games as Model Eval: 1-Click Deploy AI Town on Fly.io]() Original publisher: [Read original article]() Published: 2025-08-11T00:00:00Z Content type: opinion Language: en Sources: [The Fly Blog]() Topics: [LLM evaluation / benchmarking](), [Artificial Intelligence](), [Kaggle](), [fly.io]() Tags: [ai-models](), [benchmarks](), [cdn](), [close-to-users](), [deploy-app-servers](), [docker](), [elixir](), [eval](), [fly](), [fly-io](), [games](), [heroku-alternative](), [heroku-competitor](), [hosting](), [i](), [networking](), [postgresql-clusters](), [servers]() ### AI overview The article argues that games can make AI model evaluation more rigorous and engaging. It highlights the limits of conventional benchmarks and subjective output comparisons, points to Google's Kaggle Game Arena, and presents game environments as tests of strategic reasoning, long-term planning, and dynamic adaptation. ### Source excerpt Recently, I suggested that The Future Isn't Model Agnostic, that it's better to pick one model that works for your project and build around it, rather than engineering for model flexibility. If you buy that, you also have to acknowledge how important comprehensive model evaluation becomes. Benchmarks tell us almost nothing about how a model will actually behave in the wild, especially with long contexts, or when trusted to deliver the tone and feel that defines the UX we're shooting for. Even the best evaluation pipelines usually end in subjective, side-by-side output comparisons. Not especially rigorous, and more importantly, boring af. Can we gamify model evaluation? Oh yes. And not just because we get to have some fun for once. Google backed me up this week when it announced the Kaggle Game Arena. A public platform where we can watch AI models duke it out in a variety of classic games. Quoting Google; "Current AI benchmarks are struggling to keep pace with modern models... it can be hard to know if models trained on internet data are actually solving problems or just remembering answers they've already seen." When models boss reading comprehension tests, or ace math problems, we pay attention. But when they fail to navigate a simple conversation with a virtual character or completely botch a strategic decision in a game environment, we tell ourselves we're not building a game anyway and develop strategic short-term memory loss. Just like I've told my mom a thousand times, games are great at testing brains, and it's time we take this seriously when it comes to model evaluation. Why Games Don't Lie Games provide what benchmarks can't, "a clear, unambiguous signal of success." They give us observable behavior in dynamic environments, the kind that would be extremely difficult (and tedious) to simulate with prompt engineering alone. Games force models to demonstrate the skills we actually care about; strategic reasoning, long-term planning, and dynamic adaptation in in ## Codestral DevFeed: [Codestral]() Original publisher: [Read original article]() Published: 2024-05-29T09:00:00Z Content type: article Language: en Sources: [Mistral AI Blog]() Topics: [Artificial Intelligence](), [Generative AI](), [Code generation](), [coding](), [code-completion](), [Benchmark](), [context window](), [Python](), [Java](), [C](), [C++](), [JavaScript]() Tags: [ai](), [bash](), [benchmark](), [c](), [c-plus-plus](), [code](), [code-completion](), [code-generation](), [coding](), [context-window](), [errors](), [eval](), [java](), [javascript](), [latency](), [models](), [open](), [python]() ### AI overview Mistral introduces Codestral, a 22B open-weight generative AI model designed for code generation. It supports code completion, fill-in-the-middle generation, function writing, and test writing across more than 80 programming languages. The article describes its 32k context window, benchmark performance, latency characteristics, and research/testing license. ### Source excerpt The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.