# The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics

DevFeed: [The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics](<https://devfeed.tech/articles/the-unsampled-truth-quantifying-prompt-artifacts-in-lm-psychometrics-120044.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.03357>)

Author: Nils Schwager, Christoph Hau, Simon M\\"unker, Achim Rettinger

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [prompt](<https://devfeed.tech/topics/prompt.md>)

Tags: [artifacts](<https://devfeed.tech/tags/artifacts.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [components](<https://devfeed.tech/tags/components.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [framework](<https://devfeed.tech/tags/framework.md>), [inventory](<https://devfeed.tech/tags/inventory.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [survey](<https://devfeed.tech/tags/survey.md>), [target](<https://devfeed.tech/tags/target.md>), [weight](<https://devfeed.tech/tags/weight.md>)

## AI overview

Across 13 open-weight models (0.6B-14B) assessed on the Big Five Inventory and Short Dark Triad, task-instruction wording and option symbols often shifted response distributions more than changes to persona or item wording. The study's diagnostic crosses five baseline personas with five equivalent variants of each of four prompt components, then uses 1-Wasserstein distance and variation partitioning to separate target effects from prompt artifacts. For a substantial share of items, artifacts account for over half of explained variation, with non-semantic prompt changes sometimes exceeding baseline-persona effects. The authors propose quantifying these artifacts before interpreting model psychometric responses.

## Source excerpt

arXiv:2606.03357v3 Announce Type: replace-cross Abstract: When prompting language models for psychometric assessment, researchers assume that the responses reflect the injected persona and the meaning of the survey item. We test this premise using a diagnostic design that crosses five semantically distinct baseline personas with five semantically equivalent variants of each of four prompt components (persona wording, task instruction, item wording, option symbol). Measuring the 1-Wasserstein distance between the resulting response distributions and partitioning the variation among the five components allows for the separation of target effects from prompt artifacts. We apply the framework to 13 open-weight small language models (0.6B to 14B) on the Big Five Inventory and the Short Dark Triad. We find that in most models, the task instruction and option symbol displace response distributions further than paraphrasing the persona description or the item itself. For a substantial share of items, the artifact share of explained variation exceeds 50%; non-semantic changes of the prompt account for more response variation than the baseline personas. Our framework lets researchers quantify these prompt artifacts before interpreting psychometric output.