# Measuring and Improving Consistency in Repeated Agent Runs

DevFeed: [Measuring and Improving Consistency in Repeated Agent Runs](<https://devfeed.tech/articles/your-agent-aced-the-task-will-it-do-it-again-26920.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ibm-research/altk-evolve-consistency>)

Author: Evelyn Duesterwald; Lilian Ngweta; Vatche Isahagian; Jayaram Radhakrishnan; Vinod Muthusamy; Gaodan Fang; Ashwath Vaithinathan Aravindan; Punleuk Oum; G Thomas; Merve Unuvar; Ayhan Sebin; Michał Ulewi

Published: 2026-09-15T16:00:44Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [inference](<https://devfeed.tech/tags/inference.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [model](<https://devfeed.tech/tags/model.md>), [reports](<https://devfeed.tech/tags/reports.md>), [standard](<https://devfeed.tech/tags/standard.md>), [technical](<https://devfeed.tech/tags/technical.md>)

## AI overview

This article presents the Consistency Analyzer, a diagnostic for finding decision points where an agent's behavior may change across repeated runs. It introduces consistency guidelines in ALTK-Evolve and reports that they reduced the consistency gap from 24.4 percentage points to 12.0 points without reducing average accuracy.

## Source excerpt

That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions.