# KV-Cache Patching: Tracing How and Where Fine-Tuned Behaviors Arise

DevFeed: [KV-Cache Patching: Tracing How and Where Fine-Tuned Behaviors Arise](<https://devfeed.tech/articles/kv-cache-patching-tracing-how-and-where-fine-tuned-behaviors-arise-111592.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41480>)

Author: Jonas Loos, Oliver Eberle

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [prompt](<https://devfeed.tech/topics/prompt.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [base](<https://devfeed.tech/tags/base.md>), [behaviors](<https://devfeed.tech/tags/behaviors.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [generation](<https://devfeed.tech/tags/generation.md>), [influence](<https://devfeed.tech/tags/influence.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [lora](<https://devfeed.tech/tags/lora.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [misalignment](<https://devfeed.tech/tags/misalignment.md>), [model](<https://devfeed.tech/tags/model.md>), [patched](<https://devfeed.tech/tags/patched.md>), [patching](<https://devfeed.tech/tags/patching.md>), [processing](<https://devfeed.tech/tags/processing.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

## AI overview

The study probes where LoRA fine-tuned behaviors arise by swapping key-value caches between a fine-tuned model and its unmodified base. For some behaviors, including responding in German and emergent misalignment, fine-tuned cache states transfer the behavior to the base model; base-model cache states can partly or fully remove it from the fine-tuned model. Patching effects increase with the portion of cache replaced and concentrate at a few pivotal positions, with system-prompt and chat-template tokens exerting disproportionate influence.

## Source excerpt

arXiv:2610.41480v1 Announce Type: cross Abstract: Fine-tuning a language model changes how it responds, but where is this fine-tuned behavior instantiated? To better understand this, we evaluate whether the change in model behavior arises only during the generation of the answer tokens where the change manifests, or whether altered processing of previous tokens causes it via the attention mechanism. For this, we patch the KV cache from a LoRA fine-tuned model into the unmodified base model, and vice versa. Our results show that for some behaviors, like responding in German or emergent misalignment, patching the KV cache can indeed transfer them from the fine-tuned to the base model. Correspondingly, patching the KV cache from base to fine-tune can partially or fully remove fine-tuned behavior. Furthermore, we find that the effect of patching grows with how much of the KV cache is patched and concentrates at a few pivotal positions, with system-prompt and chat-template tokens having disproportionate influence.