# Ground truth / benchmark quality The quality and reliability of ground-truth labels and benchmarks used to evaluate machine-learning systems. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Measuring and Improving Consistency in Repeated Agent Runs DevFeed: [Measuring and Improving Consistency in Repeated Agent Runs]() Original publisher: [Read original article]() Author: Evelyn Duesterwald; Lilian Ngweta; Vatche Isahagian; Jayaram Radhakrishnan; Vinod Muthusamy; Gaodan Fang; Ashwath Vaithinathan Aravindan; Punleuk Oum; G Thomas; Merve Unuvar; Ayhan Sebin; Michał Ulewi Published: 2026-09-15T16:00:44Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Benchmark](), [Ground truth / benchmark quality](), [model-deployment]() Tags: [agent](), [benchmarks](), [consistency](), [evaluation](), [gpt](), [inference](), [leaderboard](), [model](), [reports](), [standard](), [technical]() ### AI overview This article presents the Consistency Analyzer, a diagnostic for finding decision points where an agent's behavior may change across repeated runs. It introduces consistency guidelines in ALTK-Evolve and reports that they reduced the consistency gap from 24.4 percentage points to 12.0 points without reducing average accuracy. ### Source excerpt That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. ## Benchmaxxing: When the Benchmark Becomes the Target DevFeed: [Benchmaxxing: When the Benchmark Becomes the Target]() Original publisher: [Read original article]() Author: Nathan Danneman Published: 2026-09-12T11:17:51.295154Z Content type: article Language: en Sources: [Blog]() Topics: [Benchmark](), [Artificial Intelligence](), [Cybersecurity](), [Ground truth / benchmark quality](), [Detection engineering]() Tags: [ai](), [ai-and-cybersecurity](), [benchmark](), [benchmarking](), [benchmarks](), [blog](), [cybersecurity](), [leaderboards](), [securing-ai](), [security](), [validation]() ### AI overview The article explains how public AI and cybersecurity benchmarks can become targets for optimization, a practice it calls "benchmaxxing." It argues that gaming, ceiling effects, data leakage, binary scoring, omitted costs, and aggregate scores can make benchmark results poor proxies for real-world defensive capability. The article proposes task-coupled internal benchmarks intended to evaluate end-to-end cyber agents and support rigorous science rather than visibility-driven score optimization. ### Source excerpt The more attention a benchmark receives, the stronger the incentive to optimize for it. In AI and cybersecurity, this can have significant consequences. ## Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering DevFeed: [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering]() Original publisher: [Read original article]() Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [LLM evaluation / benchmarking](), [Ground truth / benchmark quality](), [Hallucination detection]() Tags: [benchmark](), [computer-vision](), [evaluation](), [metrics](), [multimodal](), [research](), [video]() ### AI overview The article introduces CapQuiz, a reference-free benchmark for evaluating video-caption quality through human-verified multiple-choice questions. It also proposes CapF1, combining factuality and visual-information coverage, and reports stronger correlation with human judgments than existing metrics. ### Source excerpt Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the "one-to-many" nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage... ## How engineering teams can measure AI coding ROI with cost attribution and outcome metrics DevFeed: [How engineering teams can measure AI coding ROI with cost attribution and outcome metrics]() Original publisher: [Read original article]() Author: Kelsey Rosen Published: 2026-09-08T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [ai-coding](), [Ground truth / benchmark quality](), [coding](), [Finance]() Tags: [ai](), [ai-coding](), [analysis](), [blog](), [coding](), [cost](), [datadog](), [engineering](), [governance](), [harness](), [incident](), [lifecycle](), [measurement](), [metrics](), [numbers](), [production](), [team](), [tokens](), [tooling](), [workflow]() ### AI overview A Harness panel discussion outlines ways engineering organizations can measure AI coding ROI. The panel recommends attributing costs to specific work, combining throughput metrics with qualitative evidence, and using governance and process changes to interpret results. ### Source excerpt Harness panelists share practical strategies for measuring AI coding ROI, from granular cost attribution and lifecycle metrics to budgeting, governance. | Blog ## Benchmarking vector indexes DevFeed: [Benchmarking vector indexes]() Original publisher: [Read original article]() Author: Evgeniy Patlan Published: 2026-08-27T13:35:32Z Content type: article Language: en Sources: [Blog - Percona]() Topics: [benchmarking](), [Ground truth / benchmark quality](), [Databases](), [ann](), [datasets](), [Sorting]() Tags: [ai-vector](), [ann](), [benchmark](), [benchmarking](), [benchmarks](), [blog](), [blog-post](), [data](), [database-performance](), [database-trends](), [databases](), [datasets](), [embeddings](), [index](), [measurements](), [model](), [open-source](), [percona](), [points](), [report](), [run](), [search](), [sorting](), [vector-search](), [vectorsearch]() ### AI overview This article explains how Percona's vector-bench measures vector indexes by building database engines from pinned versions, running them in identical containers with the same data and hardware, and comparing consistent measurements. It describes embeddings, exact search, approximate nearest neighbour search, and ground truth as the brute-force reference needed to evaluate index accuracy. ### Source excerpt Nearly every database has vector search now, and every one of them has a blog post with a big number in it. Almost none of those numbers can be checked, because the thing that makes them meaningful is usually missing. We built a vector-bench to stop guessing. You name the engines you want, build them ... Continued The post Benchmarking vector indexes appeared first on Percona. ## SOP-Bench: A new benchmark for evaluating AI agents on real business procedures DevFeed: [SOP-Bench: A new benchmark for evaluating AI agents on real business procedures]() Original publisher: [Read original article]() Author: Rohith Nama; Nandi Subhrangshu Published: 2026-08-21T15:57:17Z Content type: article Language: en Sources: [Amazon Science homepage]() Topics: [Benchmark](), [Artificial Intelligence](), [Ground truth / benchmark quality]() Tags: [agentic-ai](), [ai](), [ai-agents](), [benchmark](), [benchmarks](), [conversational-ai](), [testing]() ### AI overview SOP-Bench is an openly available benchmark for evaluating how well AI agents execute real standard operating procedures authored by domain experts. It combines genuine enterprise procedures, functioning tools, and ground-truth answers to test interpretation, memory, judgment, and tool selection during complete procedures. ### Source excerpt Extendable framework enables testing agents on the full set of capabilities required to successfully complete a procedure, not isolated proxy tasks. ## Sidekick's continual learning loop DevFeed: [Sidekick's continual learning loop]() Original publisher: [Read original article]() Author: Andrew McNamara Published: 2026-08-05T14:52:54Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Artificial Intelligence](), [Shopify](), [GraphQL](), [Ground truth / benchmark quality](), [Latency](), [Code]() Tags: [agent](), [ai](), [frontier-model](), [latency](), [model](), [production](), [retrieval](), [routing](), [safety]() ### AI overview Shopify describes a continual learning loop for its GraphQL agent that turns production failures, user corrections, and sampled traffic into ground truth and model-weight improvements. The approach is reported to exceed frontier-model quality while reducing latency and serving costs by 96%. ### Source excerpt How we compress production failures into model weights every day, beat frontier-model quality, and cut serving costs 96%. ## Outlier Handling at Scale in Experimentation DevFeed: [Outlier Handling at Scale in Experimentation]() Original publisher: [Read original article]() Author: Margarida Moreira da Silva Published: 2026-07-01T13:44:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [data](), [Simulation](), [Ground truth / benchmark quality](), [plotting]() Tags: [data](), [experimentation](), [experiments](), [false-positive](), [outlier-detection](), [simulation](), [variance]() ### AI overview The article examines how extreme values affect experimentation at Booking.com. It describes permutation tests and simulated A/A experiments for diagnosing distorted p-value distributions, and reports that increasing outlier magnitude and frequency can cause test failures. ### Source excerpt At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don't create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention. The Problem When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com's scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes. A Diagnostic Tool: the Permutation Test One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results. Plot 1: P-value distributions from simulated A/A tests. Clean normally-distributed estimated effects produce a uniform distribution (left), while the presence of extreme outliers results in skewed p-values (right), indicating a distorted false positive rate.Simulation Evidence: What Drives Failure? We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value di ## How Evaluation-Driven Development (EDD) Works DevFeed: [How Evaluation-Driven Development (EDD) Works]() Original publisher: [Read original article]() Author: Paul Iusztin Published: 2026-06-23T08:57:02Z Content type: tutorial Language: en Sources: [Decoding ML]() Topics: [Development](), [AI Agent](), [Ground truth / benchmark quality](), [dataset]() Tags: [agent](), [ai](), [ai-evals](), [case-study](), [development](), [evaluation](), [experiment](), [saas](), [test](), [tests]() ### AI overview This case study explains Evaluation-Driven Development (EDD) for AI agents: measure a new feature, compare results before and after changes, and detect regressions before merging. It also discusses generating realistic test data when historical datasets, traces, or ground truth are unavailable. ### Source excerpt Turn every AI agent change into a measured experiment you compare before and after to detect regressions and measure performance. ## Towards passive heart health monitoring via smartphone camera DevFeed: [Towards passive heart health monitoring via smartphone camera]() Original publisher: [Read original article]() Published: 2026-06-04T19:47:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Monitoring](), [webcam](), [dataset](), [Deep learning](), [Google](), [Ground truth / benchmark quality]() Tags: [deep-learning](), [devices](), [google](), [health](), [health-bioscience](), [heart-rate-monitoring](), [human-computer-interaction-and-visualization](), [machine-intelligence](), [model](), [monitoring](), [publication](), [research](), [resource](), [smartphones]() ### AI overview Google Research presents PHRM, a research system that passively estimates heart rate and resting heart rate from facial video captured by a smartphone's front-facing camera during everyday use. The system applies deep learning to video recorded after face unlock events and reports accuracy comparable to electrocardiogram-derived ground truth and wearable trackers. The publication also releases a large, diverse smartphone-video dataset and the pre-trained PHRM-mini model for qualified researchers. ### Source excerpt Health & Bioscience ## QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard DevFeed: [QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard]() Original publisher: [Read original article]() Author: Leen AlQadi; Ahmed Alzubaidi; Mohammed Alyafeai; Maitha Alhammadi; Shaikha Alsuwaidi; Omar saif alkaabi; Basma Boussaha; Hakim Hacid Published: 2026-04-21T10:09:58Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Ground truth / benchmark quality](), [LLM evaluation / benchmarking](), [Benchmark](), [LLMs](), [Natural language processing](), [datasets](), [Inference](), [Open Source](), [Code]() Tags: [benchmark](), [benchmarks](), [blog](), [code](), [data](), [evaluation](), [hugging-face](), [inference](), [leaderboard](), [llm](), [nlp](), [open-source]() ### AI overview QIMMA is a quality-first Arabic LLM leaderboard that validates benchmark data before evaluating models. It addresses translation issues, annotation errors, encoding problems, cultural bias, reproducibility gaps, and fragmented task coverage. The platform combines native Arabic content, systematic validation, code evaluation, and public per-sample inference outputs across 109 subsets from 14 benchmarks and more than 52,000 samples. ### Source excerpt A Blog post by Technology Innovation Institute on Hugging Face ## Building better AI benchmarks: How many raters are enough? DevFeed: [Building better AI benchmarks: How many raters are enough?]() Original publisher: [Read original article]() Published: 2026-03-31T16:16:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [LLM evaluation / benchmarking](), [Human-AI evaluation](), [Ground truth / benchmark quality](), [benchmarking](), [data]() Tags: [ai](), [algorithms-theory](), [benchmarking](), [benchmarks](), [cost](), [evaluation](), [machine-intelligence](), [machine-learning](), [research]() ### AI overview Google Research presents an evaluation framework for machine-learning models that balances the number of rated items with the number of human raters per item. The research addresses reproducibility, human disagreement, benchmark quality, and the cost of collecting evaluation data, arguing that the common practice of using one to five raters per item can miss meaningful disagreement. ### Source excerpt Algorithms & Theory ## Google Research at The Check Up: from healthcare innovation to real-world care settings DevFeed: [Google Research at The Check Up: from healthcare innovation to real-world care settings]() Original publisher: [Read original article]() Published: 2026-03-17T19:47:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Google](), [Artificial Intelligence](), [AI Research](), [datasets](), [data](), [Ground truth / benchmark quality]() Tags: [ai](), [ai-research](), [data](), [datasets](), [google](), [health](), [health-bioscience](), [healthcare](), [innovation](), [machine-intelligence](), [research](), [science](), [uk](), [us]() ### AI overview Google Research highlights AI applications in healthcare, including a Personal Health Agent for preventative care, multimodal analysis of wearable data, and diagnostic research for improving breast cancer detection. The article emphasizes collaboration with healthcare professionals and the use of diverse datasets and expert-validated ground truth data. ### Source excerpt Health & Bioscience ## Introducing Groundsource: Turning news reports into data with Gemini DevFeed: [Introducing Groundsource: Turning news reports into data with Gemini]() Original publisher: [Read original article]() Published: 2026-03-12T10:42:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [datasets](), [Ground truth / benchmark quality](), [data](), [Resilience](), [Google]() Tags: [climate-sustainability](), [datasets](), [flash](), [gemini](), [generative-ai](), [natural-language-processing](), [open-source-models-datasets](), [research](), [resilience](), [sustainability]() ### AI overview Google Research introduces Groundsource, a scalable methodology that uses Gemini to convert unstructured global news reports into verified historical data. Its first open-access dataset contains 2.6 million flash-flood records from more than 150 countries, supporting research, forecasting, and urban crisis resilience. ### Source excerpt Climate & Sustainability ## Why we no longer evaluate SWE-bench Verified DevFeed: [Why we no longer evaluate SWE-bench Verified]() Original publisher: [Read original article]() Published: 2026-02-23T11:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Ground truth / benchmark quality]() Tags: [analysis](), [autonomous](), [benchmark](), [bug](), [coding](), [engineering](), [evaluation](), [model](), [open-source](), [openai](), [performance](), [research](), [software](), [software-engineering](), [training]() ### AI overview OpenAI says SWE-bench Verified no longer reliably measures frontier autonomous software-engineering capability because many remaining tasks have flawed tests and the benchmark is contaminated by training exposure. ### Source excerpt SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro. ## How to use Vertex AI Prompt Optimizer with ground truth data DevFeed: [How to use Vertex AI Prompt Optimizer with ground truth data]() Original publisher: [Read original article]() Author: Alexander Nohe; Elena Erbiceanu Tener Published: 2026-01-20T00:00:00Z Content type: tutorial Language: en Sources: [Firebase Blog]() Topics: [Prompt Engineering](), [Artificial Intelligence](), [Ground truth / benchmark quality](), [Google](), [CSV](), [data](), [Firebase]() Tags: [ai](), [cloud-storage](), [data](), [evaluation](), [firebase](), [format](), [gemini](), [google](), [google-cloud](), [how-to](), [learn](), [quality](), [vertex](), [vertex-ai]() ### AI overview This tutorial explains how to use Vertex AI Prompt Optimizer with ground truth data to iteratively tune prompts and evaluate output quality. It describes preparing scripts and target descriptions in a Google Sheet, exporting the data as CSV, uploading it to Google Cloud Storage, and configuring optimization in a Colab Enterprise notebook. ### Source excerpt Learn how to use the Vertex AI Prompt Optimizer to automatically tune your prompts to get better results by iterating on your prompts and then running an evaluation on the outputs assessing their quality to see if it has improved. ## Introducing Mistral OCR 3 DevFeed: [Introducing Mistral OCR 3]() Original publisher: [Read original article]() Published: 2025-12-17T15:00:00Z Content type: release Language: en Sources: [Mistral AI Blog]() Topics: [AI, ML & Data Engineering](), [Ground truth / benchmark quality](), [SDKs]() Tags: [ai](), [api](), [benchmarks](), [document-ai](), [html](), [json](), [models](), [ocr]() ### AI overview Mistral introduces OCR 3, a document-processing model for extracting text and embedded images, reconstructing tables, and producing Markdown or structured JSON. The article describes benchmark comparisons with ground truth and API integration for developers. ### Source excerpt The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models. ## Use Test-Controlled Data as the Ground Truth for Web Page Tests DevFeed: [Use Test-Controlled Data as the Ground Truth for Web Page Tests]() Original publisher: [Read original article]() Author: Gleb Bahmutov Published: 2025-10-03T04:00:00Z Content type: tutorial Language: en Sources: [Gleb Bahmutov]() Topics: [Testing](), [Cypress](), [Web](), [Ground truth / benchmark quality](), [data]() Tags: [advice](), [cypress](), [data](), [network](), [process](), [source-of-truth](), [test](), [testing](), [web]() ### AI overview This tutorial explains why tests should not compare two values rendered on a web page as if either value were authoritative. It recommends stubbing the data or network response so the test controls the expected value, while allowing limited duplication when it improves clarity and maintainability. ### Source excerpt Imagine you are testing a web page showing the purchase receipt. ## Evaluating GPT-5 for hallucination handling in RAG applications DevFeed: [Evaluating GPT-5 for hallucination handling in RAG applications]() Original publisher: [Read original article]() Author: Pamela Fox (noreply@blogger.com) Published: 2025-08-11T18:40:00Z Content type: opinion Language: en Sources: [Pamela Fox]() Topics: [Retrieval-Augmented Generation](), [Artificial Intelligence](), [OpenAI](), [SDKs](), [Azure](), [Ground truth / benchmark quality]() Tags: [ai](), [azure](), [evaluation](), [gpt](), [hallucinations](), [llm](), [models](), [openai](), [rag](), [sdk]() ### AI overview The article evaluates GPT-5 models in an Azure AI Foundry RAG template. It reports that GPT-5 identified when the source documents lacked enough information to answer a question, and describes broader bulk evaluations using the azure-ai-evaluations SDK across 50 question-and-answer pairs. ### Source excerpt OpenAI released the GPT-5 model family today, with an emphasis on accurate tool calling and reduced hallucinations. For those of us working on RAG (Retrieval-Augmented Generation), it's particularly exciting to see a model specifically trained to reduce hallucination. There are five variants in the family: gpt-5 gpt-5-mini gpt-5-nano gpt-5-chat: Not a reasoning model, optimized for chat applications gpt-5-pro: Only available in ChatGPT, not via the API As soon as GPT-5 models were available in Azure AI Foundry, I deployed them and evaluated them inside our popular open source RAG template. I was immediately impressed - not by the model's ability to answer a question, but by it's ability to admit it could not answer a question! You see, we have one test question for our sample data (HR documents for a fictional company's) that sounds like it should be an easy question: "What does a Product Manager do?" But, if you actually look at the company documents, there's no job description for "Product Manager", only related jobs like "Senior Manager of Product Management". Every other model, including the reasoning models, has still pretended that it could answer that question. For example, here's a response from o4-mini: However, the gpt-5 model realizes that it doesn't have the information necessary, and responds that it cannot answer the question: As I always say: I would much rather have an LLM admit that it doesn't have enough information instead of making up an answer. Bulk evaluation But that's just a single question! What we really need to know is whether the GPT-5 models will generally do a better job across the board, on a wide range of questions. So I ran bulk evaluations using the azure-ai-evaluations SDK, checking my favorite metrics: groundedness (LLM-judged), relevance (LLM-judged), and citation_match (regex based off ground truth citations). I didn't bother evaluating gpt-5-nano, as I did some quick manual tests and wasn't impressed enough - plus, we've never ## Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms DevFeed: [Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms]() Original publisher: [Read original article]() Author: Jodi Sloan Published: 2021-04-01T18:45:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Algorithms, Complexity](), [Data Science](), [Ground truth / benchmark quality](), [Kafka](), [Shopify]() Tags: [algorithms](), [data](), [data-science](), [kafka](), [search](), [shopify](), [tests](), [the-challenge]() ### AI overview Shopify describes a three-step framework for evaluating experimental search algorithms. The process uses Kafka and annotation to collect data, establishes a ground-truth relevance dataset, and measures algorithm performance through offline and online A/B tests. The article illustrates the approach by comparing Vanilla Pagerank with Query-specific Pagerank. ### Source excerpt The three-step framework Shopify's Data Science & Engineering team built for evaluating new search algorithms.