# A New Framework for Evaluating Voice Agents (EVA)

DevFeed: [A New Framework for Evaluating Voice Agents (EVA)](<https://devfeed.tech/articles/a-new-framework-for-evaluating-voice-agents-eva-7048.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ServiceNow-AI/eva>)

Author: Tara Bogavelli; Gabrielle Gauthier Melancon; Katrina Stankiewicz; Nifemi Bamgbose; Hoang Nguyen; Raghav Mehndiratta; Hari Subramani; Fanny Riols

Published: 2026-03-24T02:01:52Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [voice ai](<https://devfeed.tech/topics/voice-ai.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Bot](<https://devfeed.tech/topics/bot.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [speech-to-speech](<https://devfeed.tech/topics/speech-to-speech.md>), [asr](<https://devfeed.tech/topics/asr.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [audio](<https://devfeed.tech/tags/audio.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [demo](<https://devfeed.tech/tags/demo.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [speech](<https://devfeed.tech/tags/speech.md>), [speech-to-speech](<https://devfeed.tech/tags/speech-to-speech.md>), [transcription](<https://devfeed.tech/tags/transcription.md>), [voice](<https://devfeed.tech/tags/voice.md>)

## AI overview

EVA is an end-to-end framework for evaluating conversational voice agents across both task accuracy and conversational experience. It scores complete multi-turn spoken conversations, includes an airline dataset of 50 scenarios, and reports benchmark results for cascade and audio-native systems. The article highlights a recurring tradeoff: stronger task completion can coincide with worse user experience.

## Source excerpt

Conversational voice agents present a distinct evaluation challenge: they must simultaneously satisfy two objectives -- accuracy (completing the user's task correctly and faithfully) and conversational experience (doing so naturally, concisely, and in a way appropriate for spoken interaction).