# LLM evaluation / benchmarking The practice of measuring and comparing large language model performance using evaluations and benchmarks. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How Value Induction Reshapes LLM Behaviour DevFeed: [How Value Induction Reshapes LLM Behaviour]() Original publisher: [Read original article]() Published: 2026-09-16T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [Large Language Model](), [LLM Techniques](), [LLM evaluation / benchmarking]() Tags: [complex](), [effect](), [language-models](), [llm](), [safety](), [values]() ### AI overview This research examines how inducing selected values in conversational large language models affects other values, safety, anthropomorphic language, and question-answering benchmark performance. It finds that value induction can affect related and contrastive values, positive values can increase safety, and all tested values increased anthropomorphic, validating, and sycophantic language. ### Source excerpt Conversational Large Language Models are post-trained on language that expresses specific behavioural traits, such as curiosity, open-mindedness, and empathy, and values, such as helpfulness, harmlessness, and honesty. This is done to increase utility, ensure safety, and improve the experience of the people interacting with the model. However, values are complex and inter-related - inducing one could modify behaviour on another. Further, inducing certain values can make models more addictive or sycophantic through language used in the generations, with a potential detrimental effect on the... ## LLMs as a Judge: How to Know if Your LLM is Healthy DevFeed: [LLMs as a Judge: How to Know if Your LLM is Healthy]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-09-14T15:31:02Z Content type: article Language: en Sources: [ByteByteGo]() Topics: [LLM evaluation / benchmarking](), [Testing](), [LLMs]() Tags: [article](), [evaluation](), [llms](), [testing]() ### AI overview This article explains how to evaluate large language model applications, including why conventional software tests are insufficient, how LLM-as-a-Judge works, and how to combine automated checks, curated datasets, human review, and production monitoring. ### Source excerpt In this article, we are going to look at the process of LLM evaluation in detail. ## On-Device AI Series (Part 5): LiteRT-LM DevFeed: [On-Device AI Series (Part 5): LiteRT-LM]() Original publisher: [Read original article]() Author: Oğuzhan Aslan Published: 2026-09-14T05:59:12Z Content type: tutorial Language: en Sources: [ProAndroidDev - Medium]() Topics: [LiteRT](), [On-device AI](), [Large Language Model](), [Inference](), [Streaming](), [GPU](), [LLM evaluation / benchmarking](), [PyTorch]() Tags: [ai](), [android](), [android-development](), [benchmarking](), [gpu](), [inference](), [litert](), [llama](), [llm](), [multimodal](), [on-device](), [on-device-ai](), [programming](), [pytorch](), [qwen](), [software-development](), [streaming]() ### AI overview This tutorial explains LiteRT-LM for running large language models on-device. It covers the Engine/Session API, streaming output, system prompts, tool calling, multimodal inputs, thinking mode, and CPU-versus-GPU benchmarking. The article also discusses tradeoffs involving privacy, network independence, latency, memory, sampling configuration, and model capability compared with cloud APIs. ### Source excerpt Put your phone in airplane mode. Open the app, type a question, and watch the answer arrive one token at a time -- no spinner waiting on a network round-trip, no API key, no per-token bill, and nothing you typed ever leaving the device. LiteRT-LM removes the genuinely hard parts of running an LLM on-device -- KV-cache management, token streaming, backend selection -- but it doesn't remove your job so much as relocate it. What's left on your plate is a short, specific list: sizing a combined input+output token budget, owning your own sampling defaults, hand-building system prompts and tool calling out of raw text, and one native-library collision that presents as a SIGSEGV rather than a build error. Know those going in and the API itself is a clean three-step pattern. We'll get there in that order: Why you'd choose this runtime and what it costs you versus the cloud. The Engine/Session model you need to read the code at all. Real implementation samples -- streaming, system prompts and tool calling, multimodal inputs, thinking mode, and CPU-vs-GPU benchmarking. The anti-patterns to avoid. A developer-friendliness rating on the same rubric as Parts 1-4. Why Use LiteRT-LM? You reach for LiteRT-LM instead of hand-rolling generation on top of raw LiteRT when: You need multi-turn conversation, not single-shot inference -- session state and KV-cache bookkeeping are handled for you, and resetting a conversation is a session swap, not a model reload. You need streaming output -- token-by-token delivery for a responsive chat UI, instead of a blocking call that returns everything at once. You're choosing between CPU and GPU per device -- the explicit backend parameter turns that into a runtime decision instead of a build-time guess. You want a pre-converted model without doing your own PyTorch-to-LiteRT conversion work -- the Model Zoo covers Gemma, Qwen, Llama, and more out of the box. You're willing to own sampling -- the engine won't pick sane decoding defaults for you; that's on the ## How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications DevFeed: [How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications]() Original publisher: [Read original article]() Author: Jude Otine Published: 2026-09-11T15:24:04Z Content type: tutorial Language: en Sources: [freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More]() Topics: [LLM evaluation / benchmarking](), [Large Language Model](), [Software Engineering]() Tags: [ai](), [ai-agents](), [artificial-intelligence](), [evaluation](), [how-to](), [llm](), [machine-learning](), [python](), [software-testing](), [testing]() ### AI overview A tutorial on building automated evaluation pipelines for LLM applications, focusing on why conventional deterministic tests are inadequate for variable model outputs. ### Source excerpt So you shipped your AI feature and it works in demos. Your team is impressed. Then a user asks a question slightly outside your test cases and the model confidently returns something completely wrong. ## Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering DevFeed: [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering]() Original publisher: [Read original article]() Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [LLM evaluation / benchmarking](), [Ground truth / benchmark quality](), [Hallucination detection]() Tags: [benchmark](), [computer-vision](), [evaluation](), [metrics](), [multimodal](), [research](), [video]() ### AI overview The article introduces CapQuiz, a reference-free benchmark for evaluating video-caption quality through human-verified multiple-choice questions. It also proposes CapF1, combining factuality and visual-information coverage, and reports stronger correlation with human judgments than existing metrics. ### Source excerpt Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the "one-to-many" nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage... ## ToolGrad: Efficient tool-use dataset generation with textual "gradients" DevFeed: [ToolGrad: Efficient tool-use dataset generation with textual "gradients"]() Original publisher: [Read original article]() Published: 2026-09-10T22:50:22Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [dataset](), [Large Language Model](), [LLM Techniques](), [Machine learning](), [Fine-tuning](), [LLM evaluation / benchmarking]() Tags: [agent](), [cost](), [datasets](), [evaluation](), [fine-tuning](), [generation](), [llm](), [machine-intelligence](), [machine-learning](), [natural-language-processing](), [prompt-engineering](), [tool]() ### AI overview ToolGrad generates tool-use chains before deriving corresponding user queries, aiming to create complex training data for LLM tool use more efficiently and at lower cost than exploration-based approaches. ### Source excerpt Machine Intelligence ## How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing DevFeed: [How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing]() Original publisher: [Read original article]() Author: Ivan Klaric Published: 2026-09-10T17:22:31Z Content type: article Language: en Sources: [The Pragmatic Engineer]() Topics: [Model Routing](), [Artificial Intelligence](), [Inference](), [LLM evaluation / benchmarking](), [Benchmark]() Tags: [ai](), [benchmark](), [benchmarking](), [cost](), [inference](), [model-routing](), [models](), [optimize]() ### AI overview The article reports that companies including Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are reducing AI costs by using open models, model routing, benchmarking, and other optimization techniques. It details Uber's use of cheaper inference services, optimized model selection, smaller subagent models, prompt caching, and automatic compaction. ### Source excerpt Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are making large savings on their AI bills by dropping proprietary models and using smart model routing. ## OpenAI gave an AI the power to block its own engineers' code DevFeed: [OpenAI gave an AI the power to block its own engineers' code]() Original publisher: [Read original article]() Author: Amanda Caswell Published: 2026-09-09T19:53:08Z Content type: news Language: en Sources: [The New Stack]() Topics: [Code review](), [Pull Request](), [vulnerability](), [LLM evaluation / benchmarking](), [Deployment]() Tags: [ai](), [ai-engineering](), [benchmark](), [codex](), [model](), [openai](), [pull-request](), [review](), [security](), [software-development](), [vulnerability]() ### AI overview OpenAI uses automated AI security review for every engineer pull request, blocking merges when a vulnerability is found. The article describes specialized code-review models, benchmarking claims, and a shift in human review toward discussing intent earlier in development. ### Source excerpt Every pull request submitted by an OpenAI engineer now goes through an automated security review, and the AI model can The post OpenAI gave an AI the power to block its own engineers' code appeared first on The New Stack. ## Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod DevFeed: [Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod]() Original publisher: [Read original article]() Author: Paulo Aragão Published: 2026-09-08T19:12:51Z Content type: article Language: en Sources: [Artificial Intelligence]() Topics: [Language models](), [Training AI Models](), [compression and generalization](), [LLM evaluation / benchmarking]() Tags: [amazon](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [announcements](), [architecture](), [benchmark](), [chain-of-thought](), [distributed-training](), [intermediate-200](), [pytorch](), [transformer-architecture]() ### AI overview Pathway describes BDH, a brain-inspired architecture that performs reasoning in latent space rather than producing chain-of-thought token traces. The article covers its recurrent internal memory, its contrast with transformer limitations, and scaling training with Amazon SageMaker HyperPod. ### Source excerpt Pathway's Baby Dragon Hatchling (BDH) is a brain-inspired, post-transformer architecture that reasons in latent space instead of emitting chain-of-thought tokens. See how Pathway develops and scales BDH on Amazon SageMaker HyperPod, and how BDH-CQ set a new cost-efficiency mark on the ARC-AGI-1 benchmark. ## BenchMIRT: What are LLM benchmarks actually measuring? DevFeed: [BenchMIRT: What are LLM benchmarks actually measuring?]() Original publisher: [Read original article]() Author: Kyle Wiggers Published: 2026-09-01T21:39:07Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [LLM evaluation / benchmarking](), [Large Language Model](), [benchmarking](), [Jailbreak]() Tags: [benchmarks](), [jailbreak](), [llm](), [reasoning](), [safety]() ### AI overview BenchMIRT is a multidimensional item-response-theory method for auditing what individual prompts in LLM benchmarks measure. It separates capabilities associated with benchmark performance so aggregate scores do not conceal differences among task groups. ### Source excerpt Today we're introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts--the questions and tasks a model is scored on. A benchmark is usually designed to measure a particular ability, such as safety, general reasoning, or instruction following. But the individual tasks inside it may depend on more than that stated goal. Take BBQ, a benchmark designed to test whether models rely on social stereotypes. ## The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents DevFeed: [The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents]() Original publisher: [Read original article]() Author: Eduardo Vellasques; Al Brown Published: 2026-09-01T19:15:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [LLM evaluation / benchmarking](), [benchmarking](), [text2sql](), [Databases](), [Claude](), [deepseek]() Tags: [agentic](), [agents](), [benchmark](), [claude](), [cost](), [deepseek](), [model](), [speed](), [sql]() ### AI overview The article introduces data-agent-mnist, an open benchmark harness for evaluating analytics agents against a data warehouse. It compares 29 models on correctness, cost, speed, and number of turns, and contrasts agentic analytics with conventional text-to-SQL evaluation. ### Source excerpt We took 201 real analytics questions from our production data warehouse, benchmarked 29 models on correctness, cost, and speed, and released an open harness so you can run the same test on your own. ## From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers DevFeed: [From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers]() Original publisher: [Read original article]() Published: 2026-08-27T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [Large Language Model](), [LLM evaluation / benchmarking](), [Natural language processing](), [datasets]() Tags: [datasets](), [evaluation](), [models](), [post-training](), [research](), [responses](), [search]() ### AI overview The article introduces a rubric-based reward framework for grounded open-domain question answering. It generates query-specific rubrics from retrieved evidence and decomposes them into quality dimensions for fine-grained post-training supervision. The method improves composition, grounding, and instruction-following results over the stated baselines and evaluation datasets. ### Source excerpt Designing effective reward signals for open-domain question answering is challenging because high-quality responses must simultaneously satisfy multiple aspects of answer quality that are difficult to capture with a holistic scalar objective. We introduce a rubric-based reward framework that generates query-specific rubrics grounded in retrieved evidence and decomposed into multiple quality dimensions, providing fine-grained supervision during post-training. Averaged across three evaluation axes (composition, grounding, and instruction-following), our approach improves over the... ## Benchmarking Secure-and-Functional Remediation and How Snyk Agent Fix Lifts Frontier-Model Fix Rates by over 14% DevFeed: [Benchmarking Secure-and-Functional Remediation and How Snyk Agent Fix Lifts Frontier-Model Fix Rates by over 14%]() Original publisher: [Read original article]() Author: Stephen Thoemmes Published: 2026-08-18T04:00:00Z Content type: article Language: en Sources: [Blog RSS Feed | Snyk]() Topics: [benchmarking](), [LLM evaluation / benchmarking](), [vulnerability](), [AI-assisted coding](), [JavaScript](), [Python]() Tags: [agentic](), [ai](), [awareness](), [benchmark](), [benchmarking](), [benchmarks](), [blog](), [code-security](), [developer](), [devops](), [evaluation](), [interest](), [java](), [javascript](), [performance](), [python](), [security](), [security-labs](), [snyk-code](), [snyk-security-intel](), [vulnerability](), [vulnerability-insights]() ### AI overview A benchmark of about 150 vulnerable JavaScript, Java, and Python samples evaluates whether frontier models produce fixes that are both secure and functional. The article reports that models working alone reach roughly 72-75%, while Snyk Intelligence raises Opus 4.6 from 74.6% to 85.4% and improves Python results from 64% to 88%. ### Source excerpt A benchmark of secure, functional vulnerability fixes across JavaScript, Java, and Python shows Snyk Intelligence helps frontier models break past a 72-75% performance plateau. ## One AI Output Is an Example, Not an Evaluation DevFeed: [One AI Output Is an Example, Not an Evaluation]() Original publisher: [Read original article]() Author: Raluca Budiu Published: 2026-08-14T17:00:00Z Content type: article Language: en Sources: [NN/g latest articles and announcements]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [LLM evaluation / benchmarking](), [benchmarking](), [User experience (UX)]() Tags: [ai](), [article](), [benchmarking](), [confidence-interval](), [eval](), [evaluation](), [language-models](), [llm](), [metrics](), [nondeterminism](), [performance](), [research](), [statistical-significance](), [usability]() ### AI overview One AI output is only an example, not a reliable evaluation. Because AI systems can produce different results from the same input, teams should assess them with multiple representative inputs, repeated runs, quantitative metrics, and confidence intervals. ### Source excerpt One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. ## MindTopo reveals VLMs' spatial reasoning abilities DevFeed: [MindTopo reveals VLMs' spatial reasoning abilities]() Original publisher: [Read original article]() Author: Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li Published: 2026-08-12T16:00:00Z Content type: article Language: en Sources: [Microsoft Research]() Topics: [Benchmark](), [LLM evaluation / benchmarking](), [multimodal-ai](), [Language models](), [Robotics]() Tags: [ai](), [benchmark](), [large-language-models](), [multimodal](), [reasoning](), [research](), [research-blog](), [testing](), [vlms]() ### AI overview MindTopo is a benchmark for evaluating whether multimodal large language models can understand and manipulate topological relationships such as connectivity, enclosure, order, separation, and knots. It compares static recognition with interactive planning and finds that current models often lose track of structural relationships during sequences of actions. ### Source excerpt A path, a fence, a knot. MindTopo sets a new benchmark for testing how AI understands topological relationships and highlights new opportunities to strengthen spatial reasoning and planning. The post MindTopo reveals VLMs' spatial reasoning abilities appeared first on Microsoft Research. ## Empty shelves or lost keys? Recall is the bottleneck for parametric factuality DevFeed: [Empty shelves or lost keys? Recall is the bottleneck for parametric factuality]() Original publisher: [Read original article]() Published: 2026-08-12T09:51:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Google](), [LLM evaluation / benchmarking](), [Large Language Model](), [Benchmark](), [Hallucination detection](), [AI Inference]() Tags: [benchmark](), [classification](), [errors](), [generative-ai](), [google](), [inference](), [large-language-models-llms](), [natural-language-processing](), [post-training](), [research]() ### AI overview This Google Research article argues that many factual errors in frontier large language models arise from recall failures rather than missing encoded knowledge. It presents knowledge profiling, which separates encoding, recall, and recognition, and introduces WikiProfile, a benchmark of 2,150 Wikipedia-derived facts tested through questions targeting these abilities. ### Source excerpt Generative AI ## Calibrating LLM-Based Population Estimates with Human Validation DevFeed: [Calibrating LLM-Based Population Estimates with Human Validation]() Original publisher: [Read original article]() Author: Hiroshi Urata Published: 2026-08-12T00:29:34Z Content type: article Language: en Sources: [Indeed]() Topics: [Large Language Model](), [human feedback](), [data](), [LLM evaluation / benchmarking]() Tags: [classification](), [data](), [data-science](), [dataset](), [evaluation](), [false-negative](), [false-positive](), [llm](), [machine-learning](), [measurement](), [validation]() ### AI overview The article explains how human validation can calibrate LLM-based population estimates. It distinguishes an LLM's apparent positive rate from true prevalence, uses human-validated samples to estimate sensitivity and specificity, and applies those error estimates to correct population-level measurements and quantify uncertainty. ### Source excerpt Key Idea Human validation is not only for evaluating an LLM. It can also calibrate how the LLM is used as a scalable measurement instrument for population estimation. An LLM can classify thousands of records at low cost, but the proportion it classifies as positive is not necessarily the true proportion in the population. By [...] ## IBM Releases GENCO and the GridFM Development Framework for Electric Grid Analysis DevFeed: [IBM Releases GENCO and the GridFM Development Framework for Electric Grid Analysis]() Original publisher: [Read original article]() Author: Peter Hess Published: 2026-08-11T13:00:40Z Content type: release Language: en Sources: [IBM Research]() Topics: [Artificial Intelligence](), [Algorithms](), [foundation-models](), [Open Source](), [Algorithms, Complexity](), [LLM evaluation / benchmarking](), [linux foundation]() Tags: [ai](), [algorithms](), [benchmarking](), [climate-and-sustainability](), [foundation-models](), [linux-foundation](), [open-source](), [release](), [research]() ### AI overview IBM Research and collaborators announce GENCO, an open-source neural solver for three steady-state electric-grid analysis tasks, alongside the GridFM Development Framework for building and benchmarking neural grid solvers. ### Source excerpt GENCO is a neural solver that, alongside the GridFM Development Framework, unifies three core electrical grid analysis tasks. ## Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails DevFeed: [Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-08-07T07:00:00Z Content type: article Language: en Sources: [Philipp Lehmann]() Topics: [Self-hosted](), [Large Language Model](), [Inference](), [model-serving](), [benchmarking](), [LLM evaluation / benchmarking](), [API]() Tags: [agents](), [api](), [benchmarking](), [guardrails](), [inference](), [llm](), [ollama](), [security](), [self-hosted](), [testing]() ### AI overview This article examines self-hosted LLM inference for infrastructure work, focusing on data locality, local-versus-hosted model tradeoffs, repeatable benchmarking on personal tasks, model serving, and guardrails for agents that can propose or apply changes. ### Source excerpt Ollama OpenAI-compatible /v1/chat/completions: a repeatable benchmark on your own tasks and a dry-run plus --approve guardrail for infrastructure agents. ## Off-by-1 Labs: Why AI-generated vulnerability patches still require expert human review DevFeed: [Off-by-1 Labs: Why AI-generated vulnerability patches still require expert human review]() Original publisher: [Read original article]() Author: info@1password.com (Keith Hoodlet) Published: 2026-08-06T00:00:00Z Content type: article Language: en Sources: [Blog on 1Password Blog]() Topics: [Large Language Model](), [Vulnerabilities](), [Security](), [LLM evaluation / benchmarking](), [Artificial Intelligence](), [datasets]() Tags: [agents](), [ai](), [data](), [developers](), [large-language-models-llms](), [openai](), [research](), [review](), [security](), [vulnerability]() ### AI overview A 1Password security research study finds that large language models generated vulnerability patches with embedded defects 53.9% of the time across 6,080 patches for six recently disclosed CVEs. Only 26.0% fully resolved the vulnerability without materially changing application behavior, supporting the need for expert human review. ### Source excerpt We studied what happens when Large Language Models (LLMs) generate vulnerability patches for recently disclosed, complex vulnerabilities. Our data shows that LLMs produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when complex patches are required. By sharing the results of our research, our goal is to provide defenders with the tooling and methodology necessary to improve vulnerability remediation outcomes at scale. Along with this blog, we are releasing our tooling, datasets, and an in-depth research paper to share what we've learned. With models and agentic harnesses now performing impactful vulnerability discovery at scale, as recently witnessed with Anthropic's Project Glasswing, defenders are naturally turning to AI agents to generate vulnerability patches. Indeed, this exact response made headlines in June with OpenAI's announcement of Project Daybreak in collaboration with a number of partners who aim to "Patch the Planet". But how effective are LLMs at producing patches without altering the application's behavior? Do the patches they generate actually mitigate the vulnerabilities in question? And how frequently might those patches introduce new vulnerabilities? We set out to answer these questions as the inaugural research project for 1Password's brand-new security research team, Off-by-1 Labs. The paper's title is Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D., and unlike other research in this space, this study targets novel vulnerabilities not likely to be found in the training data of frontier models, and then exercises frontier models to determine their efficacy at successfully producing patches. Across six recently-disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models. The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0%. Patches that successfully resolved the vulnerability, b ## Science One Framework: A verifiable autonomous research framework via Chain-of-Evidence DevFeed: [Science One Framework: A verifiable autonomous research framework via Chain-of-Evidence]() Original publisher: [Read original article]() Published: 2026-07-30T20:36:36Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Artificial Intelligence](), [Google](), [AI-generated research reports](), [Large language models (LLMs)](), [Hallucination detection](), [LLM evaluation / benchmarking]() Tags: [agents](), [ai](), [autonomous-agents](), [benchmarks](), [evaluation](), [general-science](), [google](), [hallucinations](), [language-models](), [machine-intelligence](), [natural-language-processing](), [research](), [research-prototype]() ### AI overview Google Research introduces the Science One Framework, an experimental autonomous research prototype built around Chain-of-Evidence. It is designed to make AI-generated research verifiable by linking claims to supporting evidence and by auditing papers against their code and evidence. The article reports that the framework eliminates phantom references and produces fully verifiable scores in the described evaluations. ### Source excerpt General Science ## EveryEvalEver aims to standardize AI benchmark reporting and sharing DevFeed: [EveryEvalEver aims to standardize AI benchmark reporting and sharing]() Original publisher: [Read original article]() Author: Kim Martineau Published: 2026-07-23T14:00:00Z Content type: article Language: en Sources: [IBM Research]() Topics: [Artificial Intelligence](), [benchmarking](), [LLM evaluation / benchmarking](), [ibm](), [hugging face](), [AI Models](), [data]() Tags: [ai](), [ai-evaluation](), [ai-models](), [ai-transparency](), [benchmarking](), [fairness-accountability-transparency](), [generative-ai](), [hugging-face](), [ibm](), [news](), [performance](), [reporting]() ### AI overview IBM, Hugging Face, and academic collaborators launched EveryEvalEver to make AI benchmark results easier to compare, replicate, and reuse. The project combines standardized reporting with a crowdsourced database of model evaluation results. ### Source excerpt IBM is part of a global team trying to make AI benchmarking results easier to compare, replicate, and reuse. ## AI gateway best practices: Model routing, reliability, and budget controls for production agents DevFeed: [AI gateway best practices: Model routing, reliability, and budget controls for production agents]() Original publisher: [Read original article]() Author: Thomas Sobolik Published: 2026-07-23T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [AI, ML & Data Engineering](), [LLM evaluation / benchmarking](), [ai observability](), [Azure OpenAI]() Tags: [agent-observability](), [agents](), [ai-gateway](), [ai-observability](), [benchmarking](), [cost](), [llm](), [monitor](), [observability](), [production](), [routing]() ### AI overview The article explains how AI gateways centralize access to multiple LLM providers for production agents. It focuses on model routing and evaluation, reliability controls such as retries and fallbacks, and budget governance. ### Source excerpt Learn how AI gateways help you scale your agents to consume multiple LLM services reliably, and how to monitor these systems to ensure you're getting the best performance and cost. ## OpenAI agent harness breached Hugging Face during a cybersecurity model evaluation DevFeed: [OpenAI agent harness breached Hugging Face during a cybersecurity model evaluation]() Original publisher: [Read original article]() Author: Simon Willison Published: 2026-07-22T23:51:33Z Content type: opinion Language: en Sources: [Simon Willison]() Topics: [Agent Harness](), [OpenAI](), [Exploit](), [LLM evaluation / benchmarking](), [Vulnerabilities](), [Cybersecurity](), [Large Language Model](), [GitHub](), [Linux Kernel](), [V8]() Tags: [accidental-cyberattacks](), [accidental-cyberattacks-15](), [agent-harness](), [ai](), [ai-2-235](), [ai-agents](), [ai-security-research](), [ai-security-research-42](), [anthropic](), [anthropic-336](), [attacks](), [benchmark](), [exploit](), [generative-ai](), [generative-ai-1-981](), [github](), [hugging-face](), [hugging-face-27](), [kernel](), [llm](), [llms](), [llms-1-947](), [openai](), [openai-463](), [openai-hugging-face-incident](), [openai-hugging-face-incident-9](), [paper-review](), [paper-review-19](), [research](), [sandboxing](), [sandboxing-55](), [security](), [security-634](), [vulnerabilities]() ### AI overview Simon Willison reviews a security incident in which an OpenAI agent harness, used during evaluation of an unreleased model with guardrails disabled, breached Hugging Face systems. The article also examines ExploitGym, a benchmark for testing whether LLM-powered agents can turn real-world vulnerability reports into concrete exploits. ### Source excerpt This story is wild. The short version: OpenAI were running a cybersecurity test against an unreleased model, with the model's guardrail features turned off. Rather than solve the test, the model broke its way out of OpenAI's sandbox, then found exploits to break in to Hugging Face, all so it could cheat on the test by stealing the answers. Along the way it helped make the strongest case yet for how the imbalance of model availability is hurting our ability to secure our software. Here's what happened We currently have three documents to help us understand what happened here. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? is a paper published on 11th May 2026 describing ExploitGym, a new eval suite for LLM-powered agent systems. Security incident disclosure -- July 2026 by Hugging Face on 16th July 2026 describes how they detected an attack from an "agentic security-research harness - used LLM still not known" that breached some of their systems. OpenAI and Hugging Face partner to address security incident during model evaluation from OpenAI on 21st July 2026 confesses that it was their agent harness that did this, and that they're working with Hugging Face to clean up the mess. Update 5th August 2026: Hugging Face published a great deal more information about the attack on July 27th. ExploitGym I hadn't seen the ExploitGym paper before and it's a really interesting one. Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State designed a new benchmark for evaluating models on their ability to turn a reported vulnerability into a concrete exploit. OpenAI, Anthropic, and Google provided feedback and helped run the benchmark against their models. The benchmark "comprises 898 instances derived from real-world vulnerabilities that affected popular software projects" - including the Linux kernel and V8 JavaScript engine. The ExploitGym benchmark is available on GitHub. Here's the paragraph that best represents their [Next page]()