# Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets

DevFeed: [Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets](<https://devfeed.tech/articles/pairing-expert-critique-with-llm-driven-refinement-an-iterative-approach-for-curating-quality-ai-evaluation-datasets-118372.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05749>)

Author: Honson Ling, Kidus Admassu, Stephanie Rosenthal

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [AI response quality](<https://devfeed.tech/topics/ai-response-quality.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-evaluation](<https://devfeed.tech/tags/ai-evaluation.md>), [ai-workflow](<https://devfeed.tech/tags/ai-workflow.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [authoring](<https://devfeed.tech/tags/authoring.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [business](<https://devfeed.tech/tags/business.md>), [cost](<https://devfeed.tech/tags/cost.md>), [critiques](<https://devfeed.tech/tags/critiques.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [domain](<https://devfeed.tech/tags/domain.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experts](<https://devfeed.tech/tags/experts.md>), [finance](<https://devfeed.tech/tags/finance.md>), [financial](<https://devfeed.tech/tags/financial.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [llm](<https://devfeed.tech/tags/llm.md>), [quality](<https://devfeed.tech/tags/quality.md>)

## AI overview

SME-C combines lightweight critiques from subject matter experts with LLM analysis to iteratively curate response datasets and evaluation rubrics. In financial business briefs, 411 professionals submitted 3,853 critiques on 520 queries; a blind survey of 100 additional professionals found that both critique-guided refinement and refinement based on preference votes improved quality over experts' top-ranked responses. Gains were concentrated among weaker originals. The strategies had different strengths: written critiques led to more domain-grounded improvements, while vote-based refinement better identified and removed irrelevant content. The resulting resources support reproducible offline evaluation and benchmarking. The evidence provided covers this financial domain and does not describe results in other domains.

## Source excerpt

arXiv:2611.05749v1 Announce Type: cross Abstract: Evaluating and improving domain-specific LLM responses requires high-quality reference data and rubrics, yet expert authoring is slow and expensive, and synthetic datasets may fail to capture practitioner preferences unless they are grounded in expert judgment. We present SME-C (Subject Matter Expert Critique), a human-AI workflow that captures expertise through lightweight critiques of candidate responses and complements it with LLM-driven analysis and refinement to iteratively curate response datasets and evaluation rubrics. We validate SME-C in the domain of financial business briefs, collecting 3,853 critiques from 411 finance professionals on 520 queries across two survey rounds. We evaluate two LLM-driven refinement strategies: one guided by the experts' written critiques and one that infers critique solely from their preference votes. In a blind validation survey with 100 additional professionals, we found that both strategies significantly improved quality over the experts' top-ranked baseline responses. The gains were concentrated among weaker original responses, and the two strategies excelled in different ways: critique-based refinement produced more domain-grounded improvements, whereas vote-based refinement was more effective at identifying and removing irrelevant content. The resulting rubrics and refined dataset support reproducible offline evaluation and benchmarking of LLMs, suggesting that structured expert critique is a viable alternative to expert authoring for building high-quality, domain-specific evaluation resources more quickly, at scale, and at lower cost.