# Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

DevFeed: [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](<https://devfeed.tech/articles/putting-captions-to-the-test-evaluating-video-caption-quality-through-multiple-choice-question-answering-6736.md>)

Original publisher: [Read original article](<https://machinelearning.apple.com/research/video-caption-quality>)

Published: 2026-09-11T00:00:00Z

Content type: article

Language: en

Sources: [Apple Machine Learning Research](<https://devfeed.tech/sources/apple-machine-learning-research.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [research](<https://devfeed.tech/tags/research.md>), [video](<https://devfeed.tech/tags/video.md>)

## AI overview

The article introduces CapQuiz, a reference-free benchmark for evaluating video-caption quality through human-verified multiple-choice questions. It also proposes CapF1, combining factuality and visual-information coverage, and reports stronger correlation with human judgments than existing metrics.

## Source excerpt

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the "one-to-many" nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage...