# BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

DevFeed: [BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing](<https://devfeed.tech/articles/benice-closing-looping-s-inference-memory-gap-via-kv-sharing-97093.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31633>)

Author: Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [quality](<https://devfeed.tech/tags/quality.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>)

## AI overview

Looped models create separate KV-caches at each recurrence, so inference memory grows with depth. BeNice! targets sharing the final recurrence's cache with earlier recurrences through two stages: stochastic depth during pretraining encourages representations to converge across recurrences, then post-training broadcasts the final cache into the first recurrence and trains compatible residuals. The authors report enabling KV-sharing while matching standard finetuning on unshared validation perplexity. They say it composes with memory-efficient sequence mixers and low-precision KV-cache quantization. In simulated low-memory, long-context inference, sharing improved cache hit rates by up to 31.4%, increased throughput by up to 23.6%, and reduced p90 time-to-first-token by up to 83.6%.

## Source excerpt

arXiv:2610.31633v1 Announce Type: new Abstract: Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.e., sharing KV-caches across depth), finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss, a property we denote as KV-shareability. However, the mechanisms underlying KV-shareability remain poorly understood, with even identically trained looped architectures not consistently exhibiting it. In this paper, we study how KV-shareability can be enabled in looped models. We propose a two-stage method: in the first stage, we use stochastic depth during pretraining, which induces contractive residual dynamics that drive representations toward a common limit across recurrences. In the second stage, we propose BeNice!, a post-training method that broadcasts the final recurrence's KV-cache into the first recurrence, training it to produce compatible residuals while preserving the original model's quality. As a result, BeNice! enables KV-shareability while matching standard finetuning's unshared validation perplexity. Using BeNice!, we show that KV-sharing is complementary to alternative KV-compression methods, composing with memory-efficient sequence mixers (e.g., multi-latent attention, Kimi delta attention) and low-precision KV-cache quantization (e.g., fp8, mxfp8, nvfp4). Finally, in simulated low-memory, long-context inference, KV-sharing improves cache hit rates by up to 31.4%, increases throughput by up to 23.6%, and reduces p90 time-to-first-token by up to 83.6%.