# Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA

DevFeed: [Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA](<https://devfeed.tech/articles/evidence-sufficiency-and-model-comparisons-in-mexregqa-an-item-level-audit-of-mexican-spanish-regulatory-qa-93534.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25710>)

Author: Fred Hu (Huawei Technologies de M\\'exico, Mexico), Yeqi Wang (Huawei Technologies de M\\'exico, Mexico), Xing Zhou (Northeastern University, USA)

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [financial](<https://devfeed.tech/tags/financial.md>), [independent](<https://devfeed.tech/tags/independent.md>), [machine](<https://devfeed.tech/tags/machine.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [qa](<https://devfeed.tech/tags/qa.md>), [regulatory](<https://devfeed.tech/tags/regulatory.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>)

## AI overview

Similarity-gated retrieval coverage can miss facts needed to answer regulatory questions: in MexRegQA, 35 of 135 covered items lacked some or all load-bearing facts. On the post-hoc full-evidence slice, eleven prompt-matched models scored 95-98/98 with no significant differences from the anchor, which the authors say does not establish equivalence; excluding evidence-defective items changed the nominal significance status of 14 of 55 pairwise comparisons. Across 56 zero-evidence items, accuracy ranged from 33.9% to 89.3%. Removing evidence under an abstention-permitting prompt reduced accuracy by 63-81 percentage points in all ten tested legs, recovering no items. The authors release MexRegQA v1.0 with provenance and evidence-status labels, while noting the audit was outcome-aware, later comparisons reused the questions, and evaluations lack independent human validation.

## Source excerpt

arXiv:2610.25710v1 Announce Type: new Abstract: Similarity-gated coverage does not establish that retrieved evidence contains the facts needed to answer a question. We audit Mexican-Spanish financial-regulatory QA and find that 35 of 135 covered items lack some or all load-bearing facts. On the post-hoc full-evidence slice (n=98), eleven prompt-matched models score 95-98/98 and none differs significantly from the anchor; this does not establish equivalence. Removing evidence-defective items changes the nominal significance status of 14 of 55 pairwise comparisons. On 56 zero-evidence items, accuracy spans 33.9%-89.3%. Removing evidence from the audited items under an abstention-permitting prompt reduces accuracy by 63-81 percentage points in all ten tested legs, with no recovered items. Building on sufficient-context research, we contribute MexRegQA v1.0: 192 items with provenance and evidence-status labels, frozen retrieval evidence for 191 evaluated items, predictions, judge outputs and offline checks. The audit was outcome-aware, the later model/panel sweep reused the same questions, and all reported scores and labels rely on machine evaluation without independent human validation. We propose releasing evidence-status labels alongside grounded QA benchmarks so that model comparisons can be examined separately from evidence availability.