# CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

DevFeed: [CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation](<https://devfeed.tech/articles/compilerkv-risk-adaptive-kv-compression-via-offline-experience-compilation-118192.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.08686>)

Author: Ning Yang, Chengzhi Wang, Yibo Liu, Baoliang Tian, Haijun Zhang

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [prompt](<https://devfeed.tech/topics/prompt.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [clamp](<https://devfeed.tech/tags/clamp.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [compression](<https://devfeed.tech/tags/compression.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experience](<https://devfeed.tech/tags/experience.md>), [input](<https://devfeed.tech/tags/input.md>), [model](<https://devfeed.tech/tags/model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [oom](<https://devfeed.tech/tags/oom.md>), [policy](<https://devfeed.tech/tags/policy.md>), [portable](<https://devfeed.tech/tags/portable.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [risk](<https://devfeed.tech/tags/risk.md>)

## AI overview

CompilerKV improves prefill-only KV compression by compiling corrective retention tables offline from a calibration corpus. After the standard observation-window scan, online correction uses O(1) lookups and a budget clamp, addressing per-head reliability and prompt sensitivity that the abstract says are noisy when estimated from one prompt. Across four backbones, table rankings transfer across disjoint corpora (mean Spearman 0.90); transferring tables between models costs 0.4-0.8 LongBench points on average. At a 512-token budget, it beats the strongest prefill-only baseline by 1.67 points on average (95% CI 1.08-2.37). Under pressure, it scores about 73 on 128k RULER versus FullKV's 79 and SnapKV's 38, and 0.89 versus SnapKV's 0.42 on 32k NIAH. At 32k input, batch-16 serving remains feasible with 1.56% of prefill KV retained, while FullKV runs out of memory. Evidence is from the supplied abstract.

## Source excerpt

arXiv:2602.08686v4 Announce Type: replace-cross Abstract: Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction. The retention decision is therefore irreversible, yet existing methods estimate the corrective signals it relies on, per-head reliability and prompt-level compression sensitivity, online from a single noisy prompt. We argue this is the wrong statistical unit: these signals exhibit far higher cross-prompt regularity than within-prompt signal-to-noise. We introduce \textsc{CompilerKV}, a KV-retention policy whose corrective tables are compiled offline from a calibration corpus, reducing online correction after the standard observation-window scan to $O(1)$ lookups plus a budget clamp. We find that compiled retention tables behave as portable architectural priors: rankings transfer across disjoint corpora on four backbones (mean Spearman $\bar\rho{=}0.90$), and direct model-to-model table transfer costs only $0.4$--$0.8$ LongBench points on average. At a 512-token budget, \textsc{CompilerKV} attains compressed-SOTA on all four backbones, improving over the strongest prefill-only baseline by $+1.67$ points on average (task-bootstrap 95\% CI $[+1.08,+2.37]$). Pressure regimes amplify the gap: under a fixed $512/32k$ cache ratio, \textsc{CompilerKV} remains the strongest compressed method through 128k RULER ($\sim\!73$ vs.\ FullKV $\sim\!79$, SnapKV $\sim\!38$); on 32k NIAH it reaches $0.89$ vs.\ SnapKV $0.42$; and at 32k input, retaining only $1.56\%$ of the prefill KV, batch-16 serving remains feasible where FullKV is OOM.