# Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation

DevFeed: [Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation](<https://devfeed.tech/articles/definitions-label-names-or-demonstrations-auditing-what-governs-llm-annotation-124685.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09457>)

Author: Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [enhancing AI model accuracy](<https://devfeed.tech/topics/enhancing-ai-model-accuracy.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [definition](<https://devfeed.tech/tags/definition.md>), [examples](<https://devfeed.tech/tags/examples.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [nearest-neighbor](<https://devfeed.tech/tags/nearest-neighbor.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [science](<https://devfeed.tech/tags/science.md>), [social](<https://devfeed.tech/tags/social.md>), [training](<https://devfeed.tech/tags/training.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

## AI overview

Across three political-text datasets, LLMs rarely follow category definitions when those definitions are reassigned to output labels; standard fine-tuning reinforces the original label mapping without reliably improving adherence to the new definitions. For retrieval-augmented annotation, retrieving the nearest training examples generally beats retrieving examples separately by category, even with fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74-93% of the improvement, suggesting neighborhood labels carry much of the signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations to reassigned definitions improves adherence. The findings show that accuracy gains alone do not establish codebook adherence, so definitions, labels, demonstrations, retrieval, and training data need joint validation.

## Source excerpt

arXiv:2611.09457v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct \(k\)-nearest-neighbor classification recovers 74--93\% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.