# The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance

DevFeed: [The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance](<https://devfeed.tech/articles/the-intrusion-pathway-how-non-evidential-context-degrades-language-model-performance-120247.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07778>)

Author: Dikshant Kukreja, Gautam Gupta, Avni Mittal, Avinash Anand, Kritarth Prasad, Zhengkui Wang, Timothy Liu, Rajiv Ratn Shah, Aik Beng Ng, Simon See, Erik Cambria

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [long-context](<https://devfeed.tech/topics/long-context.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [account](<https://devfeed.tech/tags/account.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [avoid](<https://devfeed.tech/tags/avoid.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [call](<https://devfeed.tech/tags/call.md>), [class](<https://devfeed.tech/tags/class.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [contributions](<https://devfeed.tech/tags/contributions.md>), [corruption](<https://devfeed.tech/tags/corruption.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [model](<https://devfeed.tech/tags/model.md>), [performance](<https://devfeed.tech/tags/performance.md>)

## AI overview

The paper attributes some language-model accuracy loss from irrelevant or off-distribution prompt context to sparse "intrusion heads." Their prefix-derived writes promote distractor-related content; reduced support for the correct answer stems mainly from weaker non-prefix contributions, rather than direct prefix suppression. These heads transfer across tasks and distractor types and are largely weight-disjoint from heads that integrate relevant evidence. Replacing the corrupted final-token residual with its clean counterpart recovers 97.6% of the distractor-induced drop on average across four tasks, but requires a matched clean forward pass. A deployable knockout masking about ten heads (roughly 1% of attention heads) recovers 57%, with bounded cost to helpful context.

## Source excerpt

arXiv:2611.07778v1 Announce Type: new Abstract: Language models lose accuracy when a prompt contains non-evidential context: irrelevant statements from unrelated domains, or random off-distribution tokens, with no competing answer and no semantic overlap with the query. Such noise pervades retrieval-augmented, tool-using, and long-context deployments, yet the failure has lacked a mechanistic account. We trace the corruption to the Intrusion Pathway, which starts with MLPs enriching both query-subject and irrelevant-token representations, parroting heads carry prefix-token content to the prediction position, and relational readers largely avoid unrelated sources. Nevertheless, preserved subject enrichment and selective reading do not prevent intrusion in the broader computation. We then localize part of this interference to a sparse set of attention heads, which we call intrusion heads. Their prefix-derived writes promote distractor-related content, while their reduced support for the correct answer arises primarily from weaker non-prefix contributions rather than direct suppression by the prefix. The same heads transfer across tasks and distractor types and are largely weight-disjoint from the heads that integrate relevant contextual evidence, enabling a class-conditional intervention. Replacing the corrupted last-token residual with its clean counterpart, an oracle probe that requires a matched clean forward pass, recovers 97.6% on average of the distractor-induced drop across four downstream tasks. A deployable mask level knockout of about ten heads, roughly 1% of the model's attention heads, recovers 57% at bounded cost to helpful context.