# Improving Prompt Consistency with Structured Generations

DevFeed: [Improving Prompt Consistency with Structured Generations](<https://devfeed.tech/articles/improving-prompt-consistency-with-structured-generations-7188.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/evaluation-structured-outputs>)

Author: Will Kurt; Remi Louf; Clémentine Fourrier

Published: 2024-04-30T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Human-AI evaluation](<https://devfeed.tech/topics/human-ai-evaluation.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [compare](<https://devfeed.tech/tags/compare.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [evals](<https://devfeed.tech/tags/evals.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [leaderboards](<https://devfeed.tech/tags/leaderboards.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [research](<https://devfeed.tech/tags/research.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

## AI overview

The article examines how superficial changes to prompt formatting can substantially affect large language model benchmark scores and model rankings. Using MMLU evaluations across multiple prompt formats and models, it shows that prompt structure introduces significant variance, sometimes because of tokenizer issues, and motivates structured generations to improve consistency.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.