# Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

DevFeed: [Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions](<https://devfeed.tech/articles/beyond-naturalness-probing-automated-text-to-speech-evaluators-on-linguistically-grounded-dimensions-97959.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.09930>)

Author: Oluwanifemi Bamgbose, Jash Shah, Simon Rosen, Lindsay Devon Brin, Hoang H Nguyen, Anke Koelzer, Rachel Hansen, Tara Bogavelli, Fanny Riols

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [voice ai](<https://devfeed.tech/topics/voice-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [audio](<https://devfeed.tech/tags/audio.md>), [automated](<https://devfeed.tech/tags/automated.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [detection](<https://devfeed.tech/tags/detection.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [errors](<https://devfeed.tech/tags/errors.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [opinion](<https://devfeed.tech/tags/opinion.md>), [speech](<https://devfeed.tech/tags/speech.md>), [text-to-speech](<https://devfeed.tech/tags/text-to-speech.md>)

## AI overview

The study benchmarks four MOS predictors and four Audio-LLM judges against 860 utterances annotated by trained linguists across 10 perceptual dimensions of speech. MOS predictors are more sensitive to acoustic properties at prosodic and paralinguistic levels; Audio-LLM judges selectively detect errors, favor word-level errors, and vary with prompting. Neither group reliably captures a broad range of linguistically structured, human-perceptible errors. The authors release the dataset, annotation schema, and evaluation code to support targeted TTS evaluation.

## Source excerpt

arXiv:2608.09930v2 Announce Type: replace-cross Abstract: Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.