# Cache

A cache is a high-speed data storage layer that stores data for faster reuse and retrieval.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Uber Cuts Translation Cache Memory by up to 90% While Keeping P99 Latency Below 1 MS

DevFeed: [Uber Cuts Translation Cache Memory by up to 90% While Keeping P99 Latency Below 1 MS](<https://devfeed.tech/articles/uber-cuts-translation-cache-memory-by-up-to-90-while-keeping-p99-latency-below-1-ms-146238.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/uber-go-translation-cache/>)

Author: Leela Kumili

Published: 2026-11-20T14:38:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [memory handling](<https://devfeed.tech/topics/memory-handling.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [cost-optimization](<https://devfeed.tech/topics/cost-optimization.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [development](<https://devfeed.tech/tags/development.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [garbagecollection](<https://devfeed.tech/tags/garbagecollection.md>), [go](<https://devfeed.tech/tags/go.md>), [go-language](<https://devfeed.tech/tags/go-language.md>), [hit](<https://devfeed.tech/tags/hit.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [hybrid-architecture](<https://devfeed.tech/tags/hybrid-architecture.md>), [hybrid-work](<https://devfeed.tech/tags/hybrid-work.md>), [keys](<https://devfeed.tech/tags/keys.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [memory](<https://devfeed.tech/tags/memory.md>), [news](<https://devfeed.tech/tags/news.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [p99](<https://devfeed.tech/tags/p99.md>), [performance-tuning](<https://devfeed.tech/tags/performance-tuning.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [storage](<https://devfeed.tech/tags/storage.md>), [translation](<https://devfeed.tech/tags/translation.md>), [uber](<https://devfeed.tech/tags/uber.md>), [uber-go-translation-cache](<https://devfeed.tech/tags/uber-go-translation-cache.md>), [values](<https://devfeed.tech/tags/values.md>)

### AI overview

Uber reduced memory use in its Go translation cache by keeping keys and frequently accessed values in memory while storing other values in disk-backed files. Replacing sync.Map with a map protected by an RWMutex cut object allocations and improved CPU utilization by more than 30% in an affected service, but did not address dataset size or refreshes that temporarily held two full caches. The hybrid design shards storage across 256 locks and files, uses compact hash-based keys with collision fallback, and preloads recently used entries during refresh. Reported results include memory falling from about 2.7 GB to 750 MB, cache hit rates above 99.9%, and p99 disk-backed miss latency below 100 microseconds. In one workload, around 30,000 of more than four million entries were in memory.

### Source excerpt

Uber redesigned its Go-based translation cache to reduce its initial memory footprint by more than 90% while maintaining a cache hit rate above 99.9% and p99 disk-backed cache miss latency below 100 microseconds. The hybrid architecture combines in-memory keys and frequently accessed values with disk-backed storage, reducing garbage-collection overhead and memory consumption. By Leela Kumili

## FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

DevFeed: [FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models](<https://devfeed.tech/articles/frontierreplay-split-free-exact-prefix-caching-for-hybrid-language-models-145128.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26020>)

Author: Zekun Wang, Yizhuo Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [attention](<https://devfeed.tech/tags/attention.md>), [branch](<https://devfeed.tech/tags/branch.md>), [caching](<https://devfeed.tech/tags/caching.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [load](<https://devfeed.tech/tags/load.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [prefix-caching](<https://devfeed.tech/tags/prefix-caching.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [question](<https://devfeed.tech/tags/question.md>)

### AI overview

FrontierReplay targets exact prefix caching in hybrid language models, where recurrent state updates limit reuse to saved positions. It identifies predictable structural boundaries, such as turn ends and known branch points, and creates exact checkpoints there without interrupting prefill: it logs a short window of recurrent-layer inputs and uses a cross-layer state-only replay kernel, reported to cost 6-37x less per token than full recomputation without host synchronization. Its retention policy prioritizes active anchors and proven branches and lends unused checkpoint slots to the attention KV cache. Across the evaluated architectures, scales, and workloads, the abstract reports 16-89% lower median time to first token, up to 1.73x higher capacity at an 80% SLO, and up to 4.26x higher goodput (3.67x at larger scales) versus existing exact caches. Versus approximate replay, it reports up to 1.54x higher goodput while preserving exact task accuracy; approximation suffered up to a 17% accuracy drop on long contexts.

### Source excerpt

arXiv:2611.26020v1 Announce Type: new Abstract: Hybrid language models combine self-attention with recurrent layers whose in-place state updates restrict prefix reuse to exact saved positions. Saving states on a coarse grid wastes reusable prefixes, whereas existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load. We observe that across multi-turn chat, agent loops, document question answering, and tree search, prefix reuse almost always ends at structural boundaries predictable in advance, such as turn ends and known branch points. We present FrontierReplay, a prefix caching system that materializes exact checkpoints at these boundaries without cutting the prefill. While the forward pass runs uncut, FrontierReplay logs a short window of recurrent-layer inputs and invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization. A frontier-aware retention policy prioritizes active anchors and proven branches while lending dead checkpoint slots to the attention KV cache. Across diverse hybrid architectures, model scales, and workloads, FrontierReplay delivers clear gains over existing exact caches, lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up to 4.26x (3.67x at larger scales). Compared with approximate replay methods, FrontierReplay delivers up to 1.54x higher goodput while preserving exact task accuracy, avoiding the up to 17% accuracy drop suffered by approximation on long contexts.

## TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

DevFeed: [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](<https://devfeed.tech/articles/triaxialkv-toward-extreme-low-precision-kv-cache-quantization-for-agentic-inference-tasks-144895.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.17170>)

Author: Hanzhang Shen, Haoran Wu, Yiren Zhao, Robert Mullins

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [compression](<https://devfeed.tech/tags/compression.md>), [computer](<https://devfeed.tech/tags/computer.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom](<https://devfeed.tech/tags/custom.md>), [decode](<https://devfeed.tech/tags/decode.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [image](<https://devfeed.tech/tags/image.md>), [implement](<https://devfeed.tech/tags/implement.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

TriAxialKV reduces KV-cache memory for agentic inference by tagging tokens across recency, modality, and semantic role, calibrating sensitivity for each tag, and assigning INT2 or INT4 precision within a fixed memory budget. Its serving system combines calibration, quantization and memory management, and fused Triton decode kernels. In the reported OSWorld computer-use evaluation with Qwen3-VL-32B-Thinking, it matched SGLang's accuracy using a BF16 cache while supporting 4.5x KV-cache capacity and 30% higher end-to-end throughput on real GPUs.

### Source excerpt

arXiv:2605.17170v2 Announce Type: replace Abstract: Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bit-widths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache capacity and achieving 30% higher end-to-end throughput, when running on real GPU systems.

## TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

DevFeed: [TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming](<https://devfeed.tech/articles/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and-layer-wise-streaming-145202.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26619>)

Author: Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [memory pool](<https://devfeed.tech/topics/memory-pool.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [components](<https://devfeed.tech/tags/components.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context](<https://devfeed.tech/tags/context.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [execution](<https://devfeed.tech/tags/execution.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [management](<https://devfeed.tech/tags/management.md>), [serving](<https://devfeed.tech/tags/serving.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [transparent](<https://devfeed.tech/tags/transparent.md>)

### AI overview

TideCache serves hybrid LLMs by coordinating GPU and host memory for global-attention and bounded-state caches. Its Transparent Cache Pool reallocates physical pages while preserving virtual addresses and backend interfaces; its Cache Streamer uses bounded-state computation intervals to prefetch global-attention prefixes into shared GPU slots, overlapping transfers with computation while keeping generated KV resident. A Request Scheduler chooses resident or streamed execution and adjusts capacity based on request size and queue state. Implemented in SGLang, TideCache was evaluated on GB300 and GH200 across seven model/platform configurations. Compared with SGLang using HiCache, it completed up to 2.04x as many agent turns, reduced overall P50 per-token latency by 20-59%, and improved agent P50 TTFT by 1.40-3.05x.

### Source excerpt

arXiv:2611.26619v1 Announce Type: cross Abstract: Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.

## BlockJack: Cross-Tenant Redirection Fault Attacks on Paged KV Cache Address Translation in LLM Serving

DevFeed: [BlockJack: Cross-Tenant Redirection Fault Attacks on Paged KV Cache Address Translation in LLM Serving](<https://devfeed.tech/articles/blockjack-cross-tenant-redirection-fault-attacks-on-paged-kv-cache-address-translation-in-llm-serving-146051.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25945>)

Author: Qifan Wang, Jeremy Boy, Jonas Sander, Yansong Gao, Hongsheng Hu, Ye Dong, Thomas Eisenbarth, David Oswald

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [rowhammer](<https://devfeed.tech/topics/rowhammer.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [defense](<https://devfeed.tech/tags/defense.md>), [inference](<https://devfeed.tech/tags/inference.md>), [information](<https://devfeed.tech/tags/information.md>), [integrity](<https://devfeed.tech/tags/integrity.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [medical](<https://devfeed.tech/tags/medical.md>), [redirection](<https://devfeed.tech/tags/redirection.md>), [rowhammer](<https://devfeed.tech/tags/rowhammer.md>), [validation](<https://devfeed.tech/tags/validation.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

BlockJack corrupts vLLM-style paged KV cache block-table entries to redirect one tenant's cache reads to another tenant's blocks, without changing model parameters, prompts, or cached states. The authors report confidentiality and integrity attacks across seven Qwen, Llama, Gemma, and Phi models; on a 248-scene medical benchmark, up to 41.1% of responses recommended less urgent care while appearing consistent with the victim's question. They demonstrate physical feasibility using Rowhammer and evaluate assignment validation as a defense. vLLM and Sonar confirmed the vulnerability and proposed patches that the authors reviewed and validated.

### Source excerpt

arXiv:2611.25945v1 Announce Type: new Abstract: Large language model (LLM) serving frameworks accelerate inference by retaining previously computed states in a key-value (KV) cache. Paged serving frameworks such as vLLM organize this cache through block tables analogous to OS page tables, which map each request's logical KV blocks to physical KV blocks, forming a software-managed address translation layer. Although fault attacks such as Rowhammer have exploited OS page tables to redirect memory accesses, prior fault attacks on LLMs have largely targeted the data plane, i.e., model weights or KV-cache contents, leaving KV-cache mapping unexplored. We present BlockJack (i.e., Block Hijack), the first KV cache address-translation fault attack in LLM inference. By corrupting block-table entries, BlockJack redirects a request's KV cache reads to blocks assigned to another tenant's request without modifying model parameters, prompts, or cached KV states. It achieves both confidentiality violation (victim information appears in attacker responses) and integrity violation (attacker-supplied context steers victim answers). We study paged KV cache mappings in vLLM and formally characterize when redirect targets are predictable. Across seven models spanning the Qwen, Llama, Gemma, and Phi families, a faulted mapping exposes victim context. For the integrity attack, we examine how topic proximity, intra-topic divergence, and model behavior shape redirected responses. On a 248-scene medical benchmark, the integrity attack causes up to 41.1% of responses to recommend less urgent care, while still appearing consistent with the victim's question. We demonstrate physical feasibility of BlockJack with Rowhammer and subsequently evaluate assignment validation as a defense. Following responsible disclosure, vLLM and Sonar confirmed the vulnerability and proposed patches that we reviewed and validated.

## Global Importance Estimation for KV Cache Eviction

DevFeed: [Global Importance Estimation for KV Cache Eviction](<https://devfeed.tech/articles/global-importance-estimation-for-kv-cache-eviction-145448.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24969>)

Author: Seojin Kim, Noseong Park

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

PRKV is a training-free method for reducing KV-cache memory during long-context LLM inference. It uses Personalized PageRank on attention-induced transition graphs to propagate token-importance estimates across multiple hops, beyond recency or instantaneous attention, without modifying the model. The abstract reports evaluations across multiple LLM families on long-context understanding, sparse retrieval, and chain-of-thought reasoning: at compression levels up to 90%, PRKV achieves competitive performance against established eviction baselines while reducing peak memory with low end-to-end overhead. The supplied text does not specify benchmark details or individual results.

### Source excerpt

arXiv:2611.24969v1 Announce Type: cross Abstract: Memory consumption of the key-value (KV) cache remains a central bottleneck for long-context LLM inference. We introduce PRKV, a training-free KV eviction method that estimates multi-hop global token importance by applying Personalized PageRank over attention-induced transition graphs. This enables importance propagation beyond recency or instantaneous attention, without modifying the model. We benchmark PRKV on long-context understanding, sparse retrieval, and chain-of-thought reasoning tasks using multiple LLM families. Under moderate to aggressive compression ratios, including up to 90%, PRKV achieves competitive task performance against established eviction baselines while reducing peak memory usage with low end-to-end overhead. These results demonstrate that multi-hop global influence estimation yields a favorable accuracy--memory trade-off for scalable long-context inference.

## Characterizing High Bandwidth Flash for LLM Serving

DevFeed: [Characterizing High Bandwidth Flash for LLM Serving](<https://devfeed.tech/articles/characterizing-high-bandwidth-flash-for-llm-serving-144948.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39131>)

Author: Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [systems](<https://devfeed.tech/topics/systems.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [endurance](<https://devfeed.tech/tags/endurance.md>), [hbm](<https://devfeed.tech/tags/hbm.md>), [high-bandwidth-flash](<https://devfeed.tech/tags/high-bandwidth-flash.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [simulations](<https://devfeed.tech/tags/simulations.md>)

### AI overview

Trace-driven simulations of high-throughput agentic LLM serving find that adding high-bandwidth flash (HBF) to an HBM-HBF-host memory hierarchy can reduce completion time by 21.1-86.9% versus HBM-only systems in the evaluated workloads. Modeled energy savings reach 62.4%, varying with workload and weight placement. Because HBF has access costs and limited write endurance, the study pairs data placement with buffered cache-aware scheduling; in one evaluated configuration, this scheduling raises estimated HBF write lifetime from 0.99 to 13.63 years. These are simulation results, not measurements of deployed systems.

### Source excerpt

arXiv:2609.39131v3 Announce Type: replace Abstract: Large language model (LLM) serving requires substantial memory to store model weights and KV caches. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it important to retain KV cache for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for LLM serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 21.1-86.9% relative to HBM-only systems. Modeled energy savings reach 62.4%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 0.99 to 13.63 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling in serving LLM with HBF, positioning HBF as an actively utilized lower-level memory tier rather than passive storage.

## CommitKV: Commit-Aligned KV Cache Compression for Multimodal Agents

DevFeed: [CommitKV: Commit-Aligned KV Cache Compression for Multimodal Agents](<https://devfeed.tech/articles/commitkv-commit-aligned-kv-cache-compression-for-multimodal-agents-143297.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.07855>)

Author: Weizhong Huang, Jinchao Zhang, Xiawu Zheng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-costs](<https://devfeed.tech/tags/inference-costs.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [memory](<https://devfeed.tech/tags/memory.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [multimodal-models](<https://devfeed.tech/tags/multimodal-models.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [redundancy](<https://devfeed.tech/tags/redundancy.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [times](<https://devfeed.tech/tags/times.md>)

### AI overview

CommitKV reduces multimodal agents' KV cache costs by comparing the deletion effect of the same page before a tool call completes and after its observation returns. It retires pages with low influence in both windows, temporarily protects pages influential before the call, then fills the cache budget using attention importance, key redundancy, and token-level deletion effects. Across four open-weight models and eight benchmarks, it achieved the highest average accuracy among the compared compression methods on each model. For Qwen3-VL-8B, accuracy was 69.85% at 512 tokens, 0.91% below FullKV; at 256 tokens, it beat the strongest compression baseline by 8.44%. In matched deployments with SGLang and vLLM at 256 tokens, peak allocated GPU memory fell 54.37%-55.23% and end-to-end inference was 2.12-2.17x faster than FullKV.

### Source excerpt

arXiv:2608.07855v2 Announce Type: replace Abstract: Multimodal agents gather visual evidence through reasoning and tool use, extending their ability to solve tasks beyond the initial input. However, accumulating observations increases KV cache memory and inference costs. Token eviction reduces these costs by retaining useful cached tokens. Existing methods use attention, redundancy, or stage-specific queries to rank tokens, without explicitly comparing the same history across tool interactions. When observations shift attention, they may mistake temporarily reduced influence for lost usefulness and evict information needed later. We propose CommitKV, a training-free KV cache compression method that accounts for this changing evidence use. Commit-aligned measurement compares the deletion effects of the same page before a tool call completes and after its observation returns. Lifecycle-aware compression retires pages with low influence in both windows and temporarily protects pages influential before the call. Joint scoring combines attention importance, key redundancy, and token-level deletion effects to fill the remaining budget. Extensive experiments cover four open-weight multimodal models and eight benchmarks. CommitKV achieves the highest average accuracy among the compared compression methods on all four models. On Qwen3-VL-8B, it achieves 69.85% at a 512-token budget, only 0.91% below FullKV, and exceeds the strongest compression baseline by 8.44% at 256 tokens. Deploying the Qwen3-VL-8B agent with SGLang and vLLM at a 256-token budget reduces peak allocated GPU memory by 54.37%-55.23% and accelerates end-to-end inference by 2.12-2.17$\times$ over FullKV on the matched workload.

## KnowMe Paper Radar: Lightweight Discovery of Promising Papers in the Age of Autoresearch

DevFeed: [KnowMe Paper Radar: Lightweight Discovery of Promising Papers in the Age of Autoresearch](<https://devfeed.tech/articles/knowme-paper-radar-lightweight-discovery-of-promising-papers-in-the-age-of-autoresearch-142693.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23702>)

Author: Zhuo Chen, Jian Wang, Jinjie Gu, Lijun Wu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [authors](<https://devfeed.tech/tags/authors.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [autoresearch](<https://devfeed.tech/tags/autoresearch.md>), [browser](<https://devfeed.tech/tags/browser.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cli](<https://devfeed.tech/tags/cli.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [command-line](<https://devfeed.tech/tags/command-line.md>), [command-line-interface](<https://devfeed.tech/tags/command-line-interface.md>), [component](<https://devfeed.tech/tags/component.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [paper](<https://devfeed.tech/tags/paper.md>), [papers](<https://devfeed.tech/tags/papers.md>)

### AI overview

KnowMe Paper Radar ranks papers for a reader's current research questions by combining topical relevance and recency with author research histories, collaboration paths, and direction-specific institutional context. Users set topics, priorities, exclusions, and dates; exposed score components and sources help explain priorities and surface work beyond familiar authors. Automatic acquisition, version merging, and graph-cache reuse support routine use without LLM inference. On 174,640 acquired records, repeated context preparation was more than three times faster; cached abstracts raised NDCG@10 from 0.412 to 0.598 on expert-judged retrieval tasks. The abstract also reports analyses of semantic retrieval, research-history modeling, and literature-grounded planning, with browser and structured CLI interfaces for researchers and research agents.

### Source excerpt

arXiv:2611.23702v1 Announce Type: new Abstract: The growing volume of topically similar papers makes it difficult to identify new work that can offer useful research insights or methodological guidance. We present KnowMe Paper Radar, a lightweight discovery tool that combines multiple sources of evidence into personalized reading priorities. Users specify topics, priorities, exclusions and dates; the system brings together topical relevance, recency, author research histories, collaboration paths and direction-specific institutional context. By exposing each score component and its supporting sources, KnowMe helps readers select promising papers for their current questions and discover relevant work beyond familiar authors. Automatic acquisition, version merging and graph-cache reuse keep this workflow convenient without large language model (LLM) inference during ordinary use. Experiments on 174,640 acquired records show more than threefold faster repeated context preparation, while cached abstracts improve normalized discounted cumulative gain at rank 10 (NDCG@10) from 0.412 to 0.598 on expert-judged retrieval tasks. Complementary analyses examine how the same accumulated records support semantic retrieval, research-history modeling and literature-grounded planning. A browser interface and a structured command-line interface (CLI) make these capabilities available to both researchers and research agents, positioning KnowMe as a practical upstream tool for literature-informed recursive self-improvement (RSI) workflows.

## Gradle Build Cache Guide: Zero-Config CI Caching | Harness

DevFeed: [Gradle Build Cache Guide: Zero-Config CI Caching | Harness](<https://devfeed.tech/articles/gradle-build-cache-guide-zero-config-ci-caching-harness-146730.md>)

Original publisher: [Read original article](<https://www.harness.io/blog/fast-ci-master-gradle-build-caching>)

Author: Hemanth Mantri

Published: 2026-11-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Harness](<https://devfeed.tech/sources/harness-blog.md>)

Topics: [Gradle](<https://devfeed.tech/topics/gradle.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [build times](<https://devfeed.tech/topics/build-times.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Build tool](<https://devfeed.tech/topics/build-tool.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [build-times](<https://devfeed.tech/tags/build-times.md>), [build-tool](<https://devfeed.tech/tags/build-tool.md>), [buildcache](<https://devfeed.tech/tags/buildcache.md>), [built-in](<https://devfeed.tech/tags/built-in.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [can](<https://devfeed.tech/tags/can.md>), [ci](<https://devfeed.tech/tags/ci.md>), [gradle](<https://devfeed.tech/tags/gradle.md>), [guide](<https://devfeed.tech/tags/guide.md>), [harness](<https://devfeed.tech/tags/harness.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [leveraging](<https://devfeed.tech/tags/leveraging.md>), [management](<https://devfeed.tech/tags/management.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [remote](<https://devfeed.tech/tags/remote.md>), [simplify](<https://devfeed.tech/tags/simplify.md>), [speed](<https://devfeed.tech/tags/speed.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

Gradle's build cache can reuse task outputs across workspaces when the task's declared inputs and outputs produce a matching cache key; custom Java tasks can opt in with `@CacheableTask`. Reliable hits depend on declaring all files and properties read or written, keeping inputs deterministic, and using relative path sensitivity to avoid checkout-path differences. Sharing a local cache across teammates or CI requires remote-cache setup. The excerpt says Harness CI Build Intelligence offers a managed remote cache enabled at the pipeline stage, with the Harness agent detecting tools and managing configuration and secure storage. It supports Gradle, Maven 3.9+, Bazel, and Go. Adding `--profile` makes cache savings visible in Harness dashboards; caching works without it, but savings are not displayed. These product capabilities are presented in the supplied excerpt.

### Source excerpt

Speed up Gradle builds with Harness CI's Build Intelligence, leveraging zero-config remote caching to reduce build times and simplify cache management. | Blog

## Dell PowerMaxOS 10.5 Brings Back PowerPath With Up to 29% Higher IOPS and Adds Cyber Detect Snapshot Scans

DevFeed: [Dell PowerMaxOS 10.5 Brings Back PowerPath With Up to 29% Higher IOPS and Adds Cyber Detect Snapshot Scans](<https://devfeed.tech/articles/dell-powermaxos-10-5-brings-back-powerpath-with-up-to-29-higher-iops-and-adds-cyber-detect-snapshot-scans-141141.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/dell-powermaxos-10-5-powerpath-cyber-detect-opentelemetry-nvme-tcp>)

Author: Harold Fritts

Published: 2026-11-18T18:32:24Z

Content type: news

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [dell](<https://devfeed.tech/topics/dell.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [cyber resiliency](<https://devfeed.tech/topics/cyber-resiliency.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [8k](<https://devfeed.tech/tags/8k.md>), [ai](<https://devfeed.tech/tags/ai.md>), [attack](<https://devfeed.tech/tags/attack.md>), [blast-radius](<https://devfeed.tech/tags/blast-radius.md>), [broadcom](<https://devfeed.tech/tags/broadcom.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [claude](<https://devfeed.tech/tags/claude.md>), [corruption](<https://devfeed.tech/tags/corruption.md>), [dell](<https://devfeed.tech/tags/dell.md>), [dell-cyber-detect](<https://devfeed.tech/tags/dell-cyber-detect.md>), [dell-poweredge](<https://devfeed.tech/tags/dell-poweredge.md>), [detection](<https://devfeed.tech/tags/detection.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-storage](<https://devfeed.tech/tags/enterprise-storage.md>), [esg](<https://devfeed.tech/tags/esg.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [rest-apis](<https://devfeed.tech/tags/rest-apis.md>), [scanning](<https://devfeed.tech/tags/scanning.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [update](<https://devfeed.tech/tags/update.md>)

### AI overview

PowerMaxOS 10.5 adds cache-aware PowerPath multipathing, snapshot scanning with Cyber Detect, and expanded telemetry and management for Dell PowerMax arrays. Dell reports up to 29% higher IOPS than round-robin multipathing in an internal PowerMax 8500 test using 8K random reads served from cache; that result is workload-specific. Bulk REST API metrics were up to 14 times faster in Dell's comparison of one request covering 10 instances with 14 calls in version 10.4. OpenTelemetry can stream metrics to Prometheus or compatible collectors, and Brocade fabric events are exposed alongside array data. Cyber Detect analyzes immutable snapshots to identify clean copies and corruption, but requires a separate license and supported PowerEdge server. The OS update is available at no extra charge with an active Dell service contract and installs non-disruptively. PowerPath also supports PowerMax 2000 and 8000, while the 10.5 update targets the 2500 and 8500. An MCP integration is planned for later this year.

### Source excerpt

Dell announced PowerMaxOS 10.5 today, a software update for the PowerMax 2500 and 8500 that returns PowerPath multipathing to the platform, adds Dell Cyber Detect scanning of PowerMax snapshots, and opens array telemetry through OpenTelemetry and faster bulk REST APIs. Dell's headline figure is up to 29 percent higher IOPS from cache-aware PowerPath against native The post Dell PowerMaxOS 10.5 Brings Back PowerPath With Up to 29% Higher IOPS and Adds Cyber Detect Snapshot Scans appeared first on StorageReview.com.

## CodeReel: Accelerating Code-to-Video Agents with Exact Frame Reuse and Blocking-Aware Scheduling

DevFeed: [CodeReel: Accelerating Code-to-Video Agents with Exact Frame Reuse and Blocking-Aware Scheduling](<https://devfeed.tech/articles/codereel-accelerating-code-to-video-agents-with-exact-frame-reuse-and-blocking-aware-scheduling-140098.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22967>)

Author: Junyi Shen, Yao Lu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [caching](<https://devfeed.tech/tags/caching.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [code](<https://devfeed.tech/tags/code.md>), [codex-cli](<https://devfeed.tech/tags/codex-cli.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [execution](<https://devfeed.tech/tags/execution.md>), [frame](<https://devfeed.tech/tags/frame.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [pixels](<https://devfeed.tech/tags/pixels.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [time](<https://devfeed.tech/tags/time.md>), [training](<https://devfeed.tech/tags/training.md>), [video](<https://devfeed.tech/tags/video.md>), [video-agents](<https://devfeed.tech/tags/video-agents.md>), [videos](<https://devfeed.tech/tags/videos.md>)

### AI overview

CodeReel reduces coding agents' wait for video builds by reusing frames based on their drawing commands, even when frames move in time, and scheduling checks before an idle-priority render. It reuses work from failed builds while rendering, though scheduling uses CPU to return results sooner. In the EditLoop benchmark, it cuts wait time 36.3% versus Rebuild, a warm pipeline with parallel rendering, and 57.9% versus Sequential. For time-shifting edits, it renders 43.9% of frames, close to the 43.0% with new pixels; Manim-style caching renders 99.8%. Real-agent median wait per build was 0.62x Rebuild in Claude Code and 0.67x in Codex CLI. The abstract reports gains across five models and two frameworks.

### Source excerpt

arXiv:2611.22967v1 Announce Type: cross Abstract: Coding agents make videos by writing and refining programs in a build--inspect--edit loop. A finished film costs a median of $2.85 in model calls and 11 minutes, over a third of that time in builds that block the agent. We find that much of this time is avoidable, for two reasons. First, 37.5% of the frames of a session's later films were drawn before, many shifted in time or inside an edited scene. Second, half of the builds stop at a check, where the agent needs only the verdict, but which builds stop is unknown until the checks finish. We present CodeReel, a training-free execution layer behind agents' existing commands. Exact frame reuse keys each frame by its drawing commands, regardless of its position in time, and reuses each frame whose key recurs. Blocking-aware scheduling runs the checks first and starts the render alongside them, at idle priority, unless the session's recent builds make a pass unlikely. Scheduling spends CPU to answer sooner and reuse returns it; exact keys let later versions use frames rendered for a failed one as the agent thinks. We introduce EditLoop, a benchmark of replayable agent sessions whose films are checked against renders from scratch. On it, CodeReel cuts the agent's wait by 36.3% against Rebuild, a warm pipeline with parallel rendering, and by 57.9% against Sequential, today's step-by-step pipeline. On time-shifting edits its reuse renders 43.9%, close to the 43.0% whose pixels are new, while Manim-style caching renders 99.8%. With real agents, the median wait per build is 0.62x Rebuild's in Claude Code, on topics never used in development, and 0.67x in Codex CLI. The gain holds for five models and two frameworks.

## KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants

DevFeed: [KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants](<https://devfeed.tech/articles/kv-ltm-persistent-per-session-kv-caches-as-long-term-memory-for-personal-assistants-139943.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21998>)

Author: Ziv Kfir, Ido Kfir

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [axis](<https://devfeed.tech/tags/axis.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [budget](<https://devfeed.tech/tags/budget.md>), [but](<https://devfeed.tech/tags/but.md>), [cache](<https://devfeed.tech/tags/cache.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [compute](<https://devfeed.tech/tags/compute.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [content](<https://devfeed.tech/tags/content.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [memory](<https://devfeed.tech/tags/memory.md>), [personal](<https://devfeed.tech/tags/personal.md>), [personal-assistants](<https://devfeed.tech/tags/personal-assistants.md>), [session](<https://devfeed.tech/tags/session.md>)

### AI overview

KV-LTM proposes storing each conversation session as raw text plus a KV cache made by a fixed reader model. At query time, it restores cached sessions and appends the question; text remains the record, and the cache accelerates prefill while allowing fallback to text. It complements retrieval and indexing. Analytical sizing estimates 31K-328K bytes per token for the specified BF16 full-attention and latent-cached MLA configurations; the pinned KDA+MLA hybrid is about 8.1K bytes per token plus fixed per-session state. At 30K tokens daily, these imply roughly 0.34-3.6 TB per user-year, compared with about 49 MB of raw text. The authors propose logical QA evaluation using per-session prefill under a matched model and input-state contract, plus separate real-cache replay to test numerical equivalence and measure costs; results are not assumed. The design trades storage for avoided re-prefill computation and write-time fidelity within a retention budget.

### Source excerpt

arXiv:2611.21998v1 Announce Type: new Abstract: Personal assistants accumulate months of conversations, but today's long-term memory stacks put derivation or selection between a question and its evidence. In a system-level comparison on LongMemEval, two commercial assistants scored well below the same model reading the history offline [1], consistent with, but not isolating, content loss between write and read. We propose KV-LTM: a memory tier storing each session's raw text plus its KV cache under a fixed reader model, answering by restoring cached sessions and appending the question. KV-LTM is complementary to, not a replacement for, retrieval and indexing. Text is the memory of record; the cache is a recompute-free prefill accelerator, degrading gracefully to text. We specify the design: storage record, budget-derived retention, append-only read path, and bounded aggregation. We size it analytically: across pinned BF16 full-attention (GQA) and latent-cached MLA configurations, per-token state spans approximately 31K-328K bytes (the pinned KDA+MLA hybrid differs: approximately 8.1K bytes/token plus a fixed per-session recurrent term), giving approximately 0.34-3.6 TB per user-year at a declared 30K-token/day scenario, against approximately 49 MB of raw text; per-query cost is modelled as regime-conditional transfer, attention, decode, aggregation, and compute. We define an evaluation protocol needing no persistent-cache implementation for logical QA evaluation: per-session prefill emulates valid cache reuse under the same model/input-state contract and ideal arithmetic, with named arms, controlled baselines, and a preregistered reader-model gate; a separate minimal real-cache replay tests numerical equivalence and measures the cost axis, with outcomes not assumed. KV-LTM trades storage for avoided re-prefill compute and write-time fidelity, within a declared retention budget.

## Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels

DevFeed: [Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels](<https://devfeed.tech/articles/storing-the-gemma-4-k-v-cache-once-exact-global-kv-memory-savings-in-a-paged-serving-engine-with-unchanged-attention-kernels-140392.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23035>)

Author: Kushagra Bainsla

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [emulation](<https://devfeed.tech/tags/emulation.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [factor](<https://devfeed.tech/tags/factor.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [gemma-4-31b](<https://devfeed.tech/tags/gemma-4-31b.md>), [global](<https://devfeed.tech/tags/global.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [key](<https://devfeed.tech/tags/key.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [layers](<https://devfeed.tech/tags/layers.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nixl](<https://devfeed.tech/tags/nixl.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

For Gemma 4 31B global attention, the key is the value scaled by a fixed weight, so the authors store one 640-value row per token instead of separate 1,024-value key and value rows. An unchanged paged attention kernel reads the compact row through overlapping strided views. In a vLLM prototype, steady decode KV memory per request fell 1.34-1.40x at 32K-64K tokens; outputs matched an emulation of the same arithmetic, while 2 of 432 final answers differed from the stock engine within its run-to-run noise. With two A100 GPUs, NIXL transfer and CPU offload, a full 15 GiB decode pool handled 1.38x stock tokens per step for 16K prompts and 1.24x for 8K prompts; eager throughput gains were 1.30x and 1.11x. The prototype made equal-batch eager steps 4%-9% slower, and the method did not yet combine with an 8-bit KV cache at the default scale.

### Source excerpt

arXiv:2611.23035v1 Announce Type: cross Abstract: In the global attention layers of Gemma 4 31B, the value is the normalized key projection, and the key is the value scaled by a fixed weight, with rotary position encoding on one quarter of its dimensions. The Gemma 4 report states that this design reduces the global KV cache by up to 37.5%, but the open serving engines that we checked still store a full key and a full value: 1,024 values per KV head and token. We store one 640-value row instead, exact in real arithmetic, and an unchanged paged attention kernel reads it through two overlapping strided views as its key cache and its value cache. In a vLLM prototype with the 16-bit 31B model, the outputs are bit-identical to an emulation of the same arithmetic on the stock cache, 2 of 432 final answers differ from the stock engine (within its run-to-run noise), and the steady decode KV memory per request falls by a factor of 1.34 to 1.40 at 32K to 64K tokens. On a prefill/decode pair of two A100 GPUs with NIXL transfer and CPU offload, a decode engine with a full 15 GiB pool ran 1.38 times the stock decode tokens per step for 16K-token prompts (repeated to 0.06%) and 1.24 times for 8K-token prompts; a model of the vLLM block pool predicted both ratios to within 1% (the engine was 0.7% to 0.9% higher) and matched the change between them. The eager decode throughput rose by 1.30 and 1.11 times, and on this path the compact engine gave the same output tokens as the emulation for 8 prompts at batch 1. The run-time prototype makes an eager step 4% to 9% slower at equal batch, and the method does not yet stack with an 8-bit KV cache at the default scale.

## Revisiting Index Performance under High Cache-Miss Penalties

DevFeed: [Revisiting Index Performance under High Cache-Miss Penalties](<https://devfeed.tech/articles/revisiting-index-performance-under-high-cache-miss-penalties-140625.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21835>)

Author: Mathew M. Kim, Beomseok Nam

Published: 2026-11-18T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Performance Testing](<https://devfeed.tech/topics/performance-testing.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [application](<https://devfeed.tech/tags/application.md>), [applications](<https://devfeed.tech/tags/applications.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cxl](<https://devfeed.tech/tags/cxl.md>), [designs](<https://devfeed.tech/tags/designs.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [index](<https://devfeed.tech/tags/index.md>), [indexes](<https://devfeed.tech/tags/indexes.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [latency](<https://devfeed.tech/tags/latency.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-bandwidth](<https://devfeed.tech/tags/memory-bandwidth.md>), [modern-data](<https://devfeed.tech/tags/modern-data.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

AuxBench evaluates byte-addressable indexes as part of larger application workloads, showing that index-only benchmarks can mispredict performance. Across four indexes, controlled post-lookup memory activity changed their relative performance, and designs optimized for concurrency could lose their advantage when application work competed for cache, memory bandwidth, and CPU. Time-varying request arrivals also produced severe tail latency near saturation. The abstract describes the tested factors and findings but does not name the indexes or provide quantitative results.

### Source excerpt

arXiv:2611.21835v1 Announce Type: new Abstract: Modern data-intensive applications often query very large datasets, making index performance critical. Recent index structures exploit the fine-grained access capabilities of emerging storage devices such as DCPMM and CXL. However, existing benchmarks typically evaluate indexes in isolation using saturated, closed-loop workloads that emphasize maximum throughput. Such workloads favor highly concurrent designs while overlooking the application work that follows an index lookup and competes for cache, memory bandwidth, and CPU resources. We present AuxBench - a microbenchmark that evaluates indexes as auxiliary access paths within larger application workloads. Using AuxBench, we evaluate four byte-addressable indexes under controlled post-lookup memory activity, concurrency, and request arrival patterns. Our experiments show that post-lookup memory activity can change the relative performance of competing indexes, that concurrency-oriented designs may lose their advantages when substantial application work is introduced, and that time-varying request arrivals can produce severe tail latency near saturation. These findings demonstrate why index-only benchmarks may not adequately predict performance in larger application workloads

## OVAL: Output-Aware Local Page Bases for KV Cache Compression

DevFeed: [OVAL: Output-Aware Local Page Bases for KV Cache Compression](<https://devfeed.tech/articles/oval-output-aware-local-page-bases-for-kv-cache-compression-141617.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.06686>)

Author: Ashkan Shahbazi, Chayne Thrash, Soheil Kolouri

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [low-rank](<https://devfeed.tech/tags/low-rank.md>), [memory](<https://devfeed.tech/tags/memory.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>)

### AI overview

OVAL reduces KV-cache memory by storing each page as low-rank key and value bases that serve both as retrieval index and reconstruction data, allowing exact pages and extra per-page digests to be dropped. Its key basis combines key PCA with attention-output error, using a closed-form per-head mixing weight; the value basis uses PCA. At the same attended budget, OVAL is best or tied-best in eight of nine long-reasoning settings. It matches FullKV on MATH500 at 66% of its memory and is on par on LongBench v2; on AIME24 it scores 1.25 points above the best baseline at 56% of FullKV memory. On Video-MME and LVBench it is within 0.32 points of exact-page retrieval while using 53% of full-cache memory, versus 103% for that retrieval approach. With 4-bit bases, it is within 0.25 points of FullKV at 41% of its memory.

### Source excerpt

arXiv:2610.06686v2 Announce Type: replace Abstract: Long-context decoding, whether over long reasoning traces, long videos, or robot episodes, is increasingly limited by the memory of the key-value (KV) cache. Page-retrieval sparse attention makes each decoding step cheaper by attending only to the top-scoring KV pages. However, it still stores every page exactly, and adds a small digest per page for scoring, so the cache ends up slightly \emph{larger} than the full cache. We propose OVAL, which stores each page as a pair of output-aware low-rank bases that act as both the retrieval index and the storage. A key basis scores the page and rebuilds its keys when the page is selected, and a PCA basis rebuilds its values. The exact pages can then be dropped, and nothing extra is added to the cache. The key idea is that the key basis should be output-aware. A low-rank error in the keys perturbs the attention output at first order, but perturbs the page mass that retrieval ranks only at second order in expectation. OVAL therefore builds the key basis from a mix of key geometry (key PCA) and the attention-output error, with a per-head mixing weight set in closed form. At the same attended budget, OVAL is best or tied-best in eight of nine long-reasoning settings. It matches FullKV on MATH500 with 66\% of its memory, is 1.25 points above the best baseline on AIME24 with 56\%, and is on par with FullKV on LongBench~v2. On \mbox{Video-MME} and LVBench, it is within 0.32 points of exact-page retrieval at the same read, while storing 53\% of the full cache instead of 103\%. Its savings also stack with quantization and token pruning: with 4-bit bases, it is within 0.25 points of FullKV at 41\% of its memory.

## Vercel Drives now reads files up to 6x faster

DevFeed: [Vercel Drives now reads files up to 6x faster](<https://devfeed.tech/articles/vercel-drives-now-reads-files-up-to-6x-faster-141038.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/vercel-drives-now-reads-files-up-to-6x-faster>)

Author: Tom Lienard

Published: 2026-11-18T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Filesystems](<https://devfeed.tech/topics/filesystems.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Vercel](<https://devfeed.tech/topics/vercel.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [faster](<https://devfeed.tech/tags/faster.md>), [files](<https://devfeed.tech/tags/files.md>), [performance](<https://devfeed.tech/tags/performance.md>), [storage](<https://devfeed.tech/tags/storage.md>), [vercel](<https://devfeed.tech/tags/vercel.md>)

### AI overview

Vercel says its Drives now read files up to 6x faster on cold reads. A new read-ahead algorithm uses application access patterns to load data from persistent storage before it is needed, reducing time code spends waiting. The update applies automatically to all drives, requires no SDK upgrade or configuration changes, and leaves read pricing unchanged.

### Source excerpt

Vercel Drives now reads files up to 6x faster. The update improves cold-read performance. A new read-ahead algorithm uses your application's access patterns to load data from persistent storage before it's needed, reducing the time your code spends waiting. All drives get this update automatically, with no SDK upgrade or configuration changes required. Read pricing is unchanged. Learn more in the Vercel Drives documentation. Read more

## CVE-2026-130874 - Seerr before 3.5.2 Unauthenticated Disk Exhaustion via /imageproxy Cache Keys

DevFeed: [CVE-2026-130874 - Seerr before 3.5.2 Unauthenticated Disk Exhaustion via /imageproxy Cache Keys](<https://devfeed.tech/articles/cve-2026-130874-seerr-before-3-5-2-unauthenticated-disk-exhaustion-via-imageproxy-cache-keys-138490.md>)

Original publisher: [Read original article](<https://cvefeed.io/vuln/detail/CVE-2026-130874>)

Author: CVEFEED.io; vulncheck.com

Published: 2026-11-17T20:17:54Z

Content type: news

Language: en

Sources: [Latest High/Critical Vulnerabilitiy Feed](<https://devfeed.tech/sources/latest-high-critical-vulnerabilitiy-feed.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cve](<https://devfeed.tech/tags/cve.md>), [cve-2026-130874](<https://devfeed.tech/tags/cve-2026-130874.md>), [disk-space](<https://devfeed.tech/tags/disk-space.md>), [image](<https://devfeed.tech/tags/image.md>), [keys](<https://devfeed.tech/tags/keys.md>), [resource](<https://devfeed.tech/tags/resource.md>), [seerr](<https://devfeed.tech/tags/seerr.md>), [seerr-team](<https://devfeed.tech/tags/seerr-team.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>), [volume](<https://devfeed.tech/tags/volume.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

Seerr versions before 3.5.2 are affected by an unauthenticated disk exhaustion vulnerability in `/imageproxy`. Repeated requests for the same TMDB image using equivalent dot-segment path spellings can create duplicate cache entries because the cache keys are not normalized. The duplicates can fill the shared SQLite configuration volume, disrupting logins and requests. The excerpt gives a severity score of 8.7; it does not describe a fix or mitigation.

### Source excerpt

CVE ID :CVE-2026-130874 Published : Nov. 17, 2026, 8:17 p.m. | 47 minutes ago Description :Seerr before 3.5.2 contains an uncontrolled resource consumption vulnerability that allows unauthenticated attackers to exhaust disk space through unnormalized /imageproxy cache keys. Attackers can request the same TMDB image using many equivalent dot-segment path spellings, storing unlimited duplicate copies that fill the shared SQLite configuration volume and break logins and requests. Severity: 8.7 | HIGH Visit the link for more details, such as CVSS details, affected products, timeline, and more...

## Built on object storage, served from cache

DevFeed: [Built on object storage, served from cache](<https://devfeed.tech/articles/built-on-object-storage-served-from-cache-138350.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/built-object-storage-served-cache>)

Author: Carlota Soto

Published: 2026-11-17T13:54:05Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [branch](<https://devfeed.tech/tags/branch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [delta](<https://devfeed.tech/tags/delta.md>), [durability](<https://devfeed.tech/tags/durability.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [failover](<https://devfeed.tech/tags/failover.md>), [history](<https://devfeed.tech/tags/history.md>), [image](<https://devfeed.tech/tags/image.md>), [lakebase](<https://devfeed.tech/tags/lakebase.md>), [latency](<https://devfeed.tech/tags/latency.md>), [layer](<https://devfeed.tech/tags/layer.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Lakebase Postgres uses object storage for durable history without putting it on the routine query read path. Reads check the compute node's local cache first, then pageserver NVMe, and reach object storage only on a pageserver miss; each tier caches fetched data. In the reported production sample, compute caches handled about 99.46% of page accesses, pageserver NVMe about 0.54%, and object storage about 39 requests per second--roughly one in 28 million accesses. Object-store fetches mainly served old branches or restores and databases waking after long idle periods; those initial reads become cached. Pageservers reconstruct requested page versions from image and delta layers, while eagerly creating base images reduced the reported p99 replay work to around six deltas. The excerpt also describes quorum-replicated WAL, asynchronous persistence of materialized history, and branching or restores as pointers to an LSN.

### Source excerpt

The common argument against running Postgres on object storage goes like this:Postgres...

## What Production AI Demands from the Data Layer

DevFeed: [What Production AI Demands from the Data Layer](<https://devfeed.tech/articles/what-production-ai-demands-from-the-data-layer-139722.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/solutions/what-production-ai-demands-from-the-data-layer/>)

Author: Sabrinath Rao

Published: 2026-11-17T13:50:00Z

Content type: opinion

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [AI Factory](<https://devfeed.tech/topics/ai-factory.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [IO](<https://devfeed.tech/topics/io.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [ai-data-layer](<https://devfeed.tech/tags/ai-data-layer.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [ai-training-and-inference](<https://devfeed.tech/tags/ai-training-and-inference.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [cache](<https://devfeed.tech/tags/cache.md>), [data](<https://devfeed.tech/tags/data.md>), [data-layer](<https://devfeed.tech/tags/data-layer.md>), [flashblade-exa](<https://devfeed.tech/tags/flashblade-exa.md>), [flashblade-object-storage](<https://devfeed.tech/tags/flashblade-object-storage.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [production-ai](<https://devfeed.tech/tags/production-ai.md>), [solutions](<https://devfeed.tech/tags/solutions.md>)

### AI overview

The post argues that production AI benefits when storage resources scale independently and are measured by their effect on useful compute. FlashBlade//EXA separates metadata operations from payload traffic, allowing each path to scale on demand; STN reports growth from about 35 GB/s to as much as 100 GB/s. In submitted MLPerf Storage v3.0 results, it ranked first on the reported checkpointing and KV cache metrics, with checkpoint write bandwidth rising from 327.65 GiB/s on 10 data nodes to 877.52 GiB/s on 30; the author cautions these workloads are proxies and recommends testing real workload mixes. Purity//FB 4.8.4 adds S3 over RDMA through standard S3 calls, though the newer capability has early-release constraints. Pure KVA retains attention states and integrates with vLLM; company benchmarks report faster inference and improved time to first token. For multi-tenancy, file systems are assigned to separate data node groups to isolate drives, ports, and failures. The suggested evaluations include checkpoint stall time, goodput, concurrent time to first token, data copies, node-failure impact, and watts per sustained GB/s.

### Source excerpt

What Production AI Demands from the Data Layer by Everpure Blog Explore how a scalable AI data layer can accelerate checkpointing and inference while helping keep GPUs productive across AI workloads. The post What Production AI Demands from the Data Layer appeared first on Everpure Blog.

## CVE-2026-125162 - TYPO3 CMS - Cache Flooding in Page Cache Handling

DevFeed: [CVE-2026-125162 - TYPO3 CMS - Cache Flooding in Page Cache Handling](<https://devfeed.tech/articles/cve-2026-125162-typo3-cms-cache-flooding-in-page-cache-handling-136946.md>)

Original publisher: [Read original article](<https://cvefeed.io/vuln/detail/CVE-2026-125162>)

Author: CVEFEED.io

Published: 2026-11-17T11:11:15Z

Content type: news

Language: en

Sources: [Latest High/Critical Vulnerabilitiy Feed](<https://devfeed.tech/sources/latest-high-critical-vulnerabilitiy-feed.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [cve-2026-125162](<https://devfeed.tech/tags/cve-2026-125162.md>), [page-cache](<https://devfeed.tech/tags/page-cache.md>), [typo3](<https://devfeed.tech/tags/typo3.md>), [typo3-cms](<https://devfeed.tech/tags/typo3-cms.md>)

### AI overview

TYPO3's cHash handling could let unauthenticated visitors flood page cache storage and trigger uncached renders. When cHash was supplied as an array, validation treated it as missing, but the cache identifier treated it as present and included all URL arguments, allowing arbitrary values to create unlimited cache entries. The issue affects installations where `$GLOBALS['TYPO3_CONF_VARS']['FE']['cacheHash']['enforceValidation']` is disabled. That setting is enabled by default for new installations since TYPO3 v12, but may remain disabled on upgraded installations.

### Source excerpt

CVE ID :CVE-2026-125162 Published : Nov. 17, 2026, 11:11 a.m. | 12 minutes ago Description :The cHash argument, which ensures that only URL arguments originally generated by TYPO3 create separate page cache entries, was treated inconsistently when it was passed as an array. The cHash validation treated such a value as missing, while the page cache identifier treated it as present and included all URL arguments. As a result, unauthenticated visitors could create an unlimited number of page cache entries and trigger uncached page renderings by adding an array cHash with arbitrary values to any URL. This could exhaust server resources and fill the page cache storage. Only installations where the cHash validation is not enforced via $GLOBALS['TYPO3_CONF_VARS']['FE']['cacheHash']['enforceValidation'] are affected. This setting is enabled by default for new installations since TYPO3 v12, but is often still disabled in installations upgraded from older versions. Severity: 8.2 | HIGH Visit the link for more details, such as CVSS details, affected products, timeline, and more...

## KaleidoKV: Rearranging Evidence for Efficient KV Cache Reuse in Offline RAG Inference

DevFeed: [KaleidoKV: Rearranging Evidence for Efficient KV Cache Reuse in Offline RAG Inference](<https://devfeed.tech/articles/kaleidokv-rearranging-evidence-for-efficient-kv-cache-reuse-in-offline-rag-inference-138293.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19253>)

Author: Peiting Xie, Xingyu Tan, Xiangjun Zai, Chen Chen, Xiaoyang Wang, Wenjie Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [admission](<https://devfeed.tech/tags/admission.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [engine](<https://devfeed.tech/tags/engine.md>), [execution](<https://devfeed.tech/tags/execution.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [global](<https://devfeed.tech/tags/global.md>), [groups](<https://devfeed.tech/tags/groups.md>), [independent](<https://devfeed.tech/tags/independent.md>), [index](<https://devfeed.tech/tags/index.md>), [inference](<https://devfeed.tech/tags/inference.md>), [key](<https://devfeed.tech/tags/key.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [offline](<https://devfeed.tech/tags/offline.md>), [rag](<https://devfeed.tech/tags/rag.md>), [reuse](<https://devfeed.tech/tags/reuse.md>)

### AI overview

KaleidoKV targets wasted prefill in offline RAG batches where requests share evidence but prefix caching depends on evidence order. It jointly rearranges evidence across requests, retaining coordinated moves only when they reduce planned prefill, and uses a global prefix index to evaluate affected request paths incrementally. Its prefix-ready admission delays requests whose shared prefixes are still being computed while letting independent requests proceed; each request's evidence set and content are preserved. Across four QA workloads and four LLMs, it reports the highest throughput and lowest mean time to first token in all 16 settings, reaching about 1.24-1.75x the strongest baseline's throughput including planning time. For Llama-3.1-8B under the default configuration, actual prefill was within 0.5% of planned prefill across the four workloads.

### Source excerpt

arXiv:2611.19253v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) augments large language models (LLMs) on question-answering (QA) tasks with retrieved evidence at inference time. Related questions in an offline batch often retrieve overlapping evidence, but prefix caching reuses key-value (KV) states only for identical token prefixes, making reuse dependent on evidence order. Although all questions and evidence are known before execution, existing reordering methods commit to early grouping decisions and can miss sharing opportunities that require several requests to move together. Evaluating alternative layouts can incur planning costs that offset inference savings, while concurrent requests may still recompute shared prefixes before their KV states become available. To address these challenges, we propose KaleidoKV, a framework that plans evidence orders across the batch and controls when requests enter the engine while preserving each request's evidence set and content. Joint layout planning revises formed groups through coordinated request moves and keeps a move only if it lowers planned prefill, the computation required when each shared prefix is computed once. A global prefix index selects evidence orders by locating reusable prefixes and evaluates each move incrementally from only the affected request paths. Prefix-ready admission tracks nested prefix dependencies, delaying requests whose shared prefixes are still being computed while allowing independent requests to proceed. Experiments on four QA workloads and four LLMs show that KaleidoKV achieves the highest throughput and lowest mean time to first token in all 16 settings, with approximately 1.24-1.75x the throughput of the strongest baseline, including planning time. With Llama-3.1-8B under the default configuration, actual prefill stays within 0.5% of planned prefill across all four workloads.

## Where the Savings Go: The Limits of Hiding Cross-Tenant Cache Hits in LLM Serving

DevFeed: [Where the Savings Go: The Limits of Hiding Cross-Tenant Cache Hits in LLM Serving](<https://devfeed.tech/articles/where-the-savings-go-the-limits-of-hiding-cross-tenant-cache-hits-in-llm-serving-139296.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20291>)

Author: Bowen Sun, Chaowei Xiao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [random](<https://devfeed.tech/tags/random.md>), [request](<https://devfeed.tech/tags/request.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [share](<https://devfeed.tech/tags/share.md>), [trace](<https://devfeed.tech/tags/trace.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

Sharing an LLM prefix cache can leak whether another tenant used a text through saved computation that speeds later queued requests. In step-based schedulers that keep working while requests wait, the saved time shifts every queued request earlier. A rate-limited attacker can bracket a probe with two groups of one-token listeners and compare their completion steps. In a replayed production trace on one vLLM instance at up to half capacity, one probe detected cached texts of 256 tokens or more with advantage at least 0.98. Hiding only the hit request still allowed detection with one probe on vLLM and SGLang; random delays had to reach roughly ten times the shift or more to reduce advantage to 0.1. The abstract also bounds a tenant's gain from content written by k other accounts by order k times the permitted leakage: with zero leakage, only the provider retains the savings. Exact hiding is possible in principle on one instance, but pause-or-hold mechanisms under saturated prefill must pad every step that might hold a hit, with cost growing over time rather than with hits.

### Source excerpt

arXiv:2611.20291v1 Announce Type: new Abstract: LLM serving engines cache prompt prefixes; a cache hit skips computation and returns faster, so a shared cache reveals whether others have sent a given text. Per-account salting closes this channel but forgoes cross-tenant reuse and isolates only whole accounts, even where access is redistributed or trust domains are subdivided below them. Defenses that share globally but hide hits promise reuse without relying on account boundaries. Hiding a hit requires everything tenants observe afterwards to be distributed as if it had missed. The defenses we examine target the hit-miss gap; we instead follow where the saved time goes. Step-based schedulers that never idle while work waits pass it in full to every request queued behind the hit, shifting each earlier by the same amount. An attacker within ordinary rate limits brackets a probe with two groups of listeners that each return one token, sends them in one short burst, and counts in how many steps they finish. Replaying a production trace on one vLLM instance at up to half capacity, a single probe detects cached texts of 256 tokens or more with advantage at least 0.98. Hiding only the hit request falls to one probe on vLLM and SGLang; random delays must reach about ten times the shift or more to cut the advantage to 0.1; narrowing what is shared gives no guarantee against this inference. Since any observable tenant benefit is also a signal, we prove that a rate-limited tenant's gain from content written by k other accounts is at most of order k times the leakage allowed, whatever the provider does with identities; at zero leakage, only the provider keeps the savings. Exact hiding is attainable in principle on one instance, but mechanisms that only pause or hold work know skipped work's duration only approximately, so under saturated prefill they must pad every step that may hold a hit, at a price that grows with time rather than with hits.

## NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention

DevFeed: [NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention](<https://devfeed.tech/articles/nvfp4-mla-unified-fp4-kv-cache-storage-and-attention-computation-for-multi-head-latent-attention-136979.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18972>)

Author: Chao Ni, Qi Zhang, Xiao Wang, Zongfei Jing, Mingyang Hao, Binghan Chen, Kefeng Duan, Yu-Jung Chen, Mengdi Wang, Tijmen Blankevoort, Jun Yang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [mla](<https://devfeed.tech/tags/mla.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

NVFP4-MLA jointly designs FP4 quantization, the attention kernel, and cache system for multi-head latent attention. It uses selective residual compensation on quantization-sensitive channels to preserve accuracy, and two-dimensional quantization blocks so the shared latent cache supports both attention BMM reduction directions without requantization. On DeepSeek-R1 and Kimi-K3, it matches BF16 MLA average accuracy across nine benchmarks while compressing KV cache by 3.20x and 3.05x, respectively. On NVIDIA Rubin GPUs, attention kernels run up to 2.82x and 2.71x faster than FP8 MLA; p90 time to first token in agentic multi-turn serving falls by up to 3.5x and 2.1x. Disaggregated DeepSeek-R1 serving achieves up to 2.30x higher per-GPU throughput.

### Source excerpt

arXiv:2611.18972v1 Announce Type: new Abstract: Long-context reasoning and agentic workloads make key-value (KV) cache capacity and attention computation the dominant costs of serving frontier large language models. We present NVFP4-MLA, the first design to push both KV-cache storage and attention computation of multi-head latent attention (MLA) to FP4. NVFP4-MLA co-designs the quantization algorithm with the attention kernel and the cache system. Selective residual compensation preserves model accuracy by adding NVFP4 residuals only to the quantization-sensitive channels. Two-dimensional quantization blocks make the shared latent cache compatible with the different reduction directions of the two attention batched matrix multiplications (BMMs), removing requantization between them. On DeepSeek-R1 and Kimi-K3, NVFP4-MLA matches BF16 MLA average accuracy across nine benchmarks while compressing the MLA KV cache by 3.20x and 3.05x relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to 2.82x on DeepSeek-R1 and 2.71x on Kimi-K3, and lowers p90 time to first token in agentic multi-turn serving by up to 3.5x and 2.1x, respectively. In disaggregated DeepSeek-R1 serving, it also raises per-GPU throughput by up to 2.30x.

[Next page](<https://devfeed.tech/topics/cache.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgImRhMmU2ZGRhLWFkNGUtNDBlMi1hMTU2LTU2YmJhMmJjYjY2NyJd>)