# Prompt-agnostic Language Models

DevFeed: [Prompt-agnostic Language Models](<https://devfeed.tech/articles/prompt-agnostic-language-models-107942.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.37683>)

Author: Frederic Sadrieh

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [operators](<https://devfeed.tech/tags/operators.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>), [variance](<https://devfeed.tech/tags/variance.md>)

## AI overview

Prompt-agnostic Language Models addresses prompt sensitivity during instruction tuning rather than relying on manual inference-time prompt optimization. The authors report that gradients from paraphrased prompts conflict in 57% of trainable parameters, while naive averaging discards about one fifth of gradient magnitude. Their methods, Prompt-TIES and Prompt-DARE, apply gradient-merging techniques to reduce this interference. Across four models from two families at two scales, prompt phrasing and example interactions account for 54% of per-example variability on average, exceeding 70% on ANLI and WiC. Gradient merging nearly halves per-example variance on fully fine-tuned models with little cost to overall accuracy; the other evaluated train-time approaches improve raw performance but do not reliably reduce that variance. Evidence is from the supplied abstract.

## Source excerpt

arXiv:2610.37683v1 Announce Type: cross Abstract: Despite their strong performance, large language models remain highly sensitive to prompt formulation. This issue is typically addressed at inference-time through manual prompt optimization, which introduces overhead and lacks scalability. Conversely, we focus on train-time approaches that build prompt robustness into the model during instruction tuning. A common strategy is to mix multiple paraphrases of each prompt into training. We demonstrate that across trainable parameters, the per-paraphrasing gradients interfere, with 57% exhibiting sign conflicts and the naive gradient average discarding approximately a fifth of the magnitude. Therefore, we develop two gradient-merging methods, Prompt-TIES and Prompt-DARE, that address this interference by leveraging interference-resolution operators from model and gradient merging and applying them to prompt gradients. In this study, we compare the efficacy of our methods to increase model robustness to train-time robustness methods spanning all three established classes: data construction, consistency regularization, and contrastive alignment. The evaluation is conducted across four models from two families at two scales. We observe the interaction between prompt phrasing and the specific examples accounts for 54% of per-example variability on average, and exceeds 70% on tasks such as ANLI and WiC. The gradient-merging methods directly improve this local robustness and nearly halve per-example variance on fully fine-tuned models at little cost to overall accuracy. The other train-time methods enhance raw performance, but fail to improve per-example variance reliably. Code: https://github.com/FSadrieh/prompt-agnostic-llms. Model checkpoints: https://huggingface.co/prompt-agnostic-language-models.