# LLM optimization

LLM optimization is a set of techniques for making large language model training and inference faster or cheaper, including optimizing LLM inference and serving on GPUs.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## CertiK and Columbia University Bring Math Proofs to the GPU Code Behind Every Large AI Model

DevFeed: [CertiK and Columbia University Bring Math Proofs to the GPU Code Behind Every Large AI Model](<https://devfeed.tech/articles/certik-and-columbia-university-bring-math-proofs-to-the-gpu-code-behind-every-large-ai-model-146451.md>)

Original publisher: [Read original article](<https://hackernoon.com/certik-and-columbia-university-bring-math-proofs-to-the-gpu-code-behind-every-large-ai-model?source=rss>)

Author: Ishan Pandey

Published: 2026-11-20T14:53:35Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [kernels](<https://devfeed.tech/topics/kernels.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-model](<https://devfeed.tech/tags/ai-model.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [certik](<https://devfeed.tech/tags/certik.md>), [code](<https://devfeed.tech/tags/code.md>), [columbia-university](<https://devfeed.tech/tags/columbia-university.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [math](<https://devfeed.tech/tags/math.md>), [production](<https://devfeed.tech/tags/production.md>), [proofs](<https://devfeed.tech/tags/proofs.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [technology](<https://devfeed.tech/tags/technology.md>), [web3](<https://devfeed.tech/tags/web3.md>)

### AI overview

TensorLean uses machine-checked Lean proofs to verify GPU kernels at Nvidia's PTX instruction level, accounting for overlapping data transfers, changing layouts, and lower-precision arithmetic within stated tolerances. Applied to more than 100 production kernels from DeepSeek and FlashAttention, it found two bugs in DeepSeek kernels and 23 in FlashAttention; failed proofs identify breaking cases, and repaired code can be checked again. Unlike tests on selected inputs, formal proofs establish requirements across a defined input range. The article describes a workflow in which teams reverify each optimization and retain proofs alongside kernels, with reusable specifications intended to ease application across related versions. The excerpt presents this as a way to reduce risks from silent computational errors and costly reruns, while noting that adoption by labs and library maintainers remains the next test.

### Source excerpt

CertiK and Columbia's TensorLean uses machine-checked proofs to verify 100+ production AI kernels from DeepSeek and FlashAttention, finding 25 bugs.

## PostHog puts 444.6x cheaper on decision models

DevFeed: [PostHog puts 444.6x cheaper on decision models](<https://devfeed.tech/articles/posthog-puts-444-6x-cheaper-on-decision-models-146091.md>)

Original publisher: [Read original article](<https://jevdaily.com/issues/2026-11-20>)

Author: Jev Daily

Published: 2026-11-20T14:11:27.545612Z

Content type: news

Language: en

Sources: [Jev Daily](<https://devfeed.tech/sources/jev-daily.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [antigravity](<https://devfeed.tech/tags/antigravity.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gemini-4-neon](<https://devfeed.tech/tags/gemini-4-neon.md>), [gpt-6-2-astra](<https://devfeed.tech/tags/gpt-6-2-astra.md>), [jev](<https://devfeed.tech/tags/jev.md>), [kitchen](<https://devfeed.tech/tags/kitchen.md>), [llm](<https://devfeed.tech/tags/llm.md>), [opus-5-5](<https://devfeed.tech/tags/opus-5-5.md>), [prompt-caching](<https://devfeed.tech/tags/prompt-caching.md>), [run](<https://devfeed.tech/tags/run.md>), [someone-measured-it](<https://devfeed.tech/tags/someone-measured-it.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [the-case-against](<https://devfeed.tech/tags/the-case-against.md>), [theo](<https://devfeed.tech/tags/theo.md>), [waiting](<https://devfeed.tech/tags/waiting.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

PostHog reports that Jev, a decision model, handles some work up to 444.6 times cheaper and 193.6 times faster than constant LLM calls. The excerpt does not identify the workload, comparison LLM, or measurement method, so the figures cannot be assessed further from this evidence. Separately, Theo uses a waiter analogy to criticize an agent that pauses a task to ask an unnecessary question; he provides no run, model settings, or task, so this is not a measured result.

### Source excerpt

PostHog says decision models like Jev do some of its work up to 444.6x cheaper and 193.6x faster than constant LLM calls, without saying which work or against...

## Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

DevFeed: [Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter](<https://devfeed.tech/articles/tetra-serving-leech-lattice-quantized-llms-at-2-7-bits-per-parameter-144940.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.35465>)

Author: Pier-Jean Malandrino

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [awq](<https://devfeed.tech/tags/awq.md>), [block](<https://devfeed.tech/tags/block.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [figure](<https://devfeed.tech/tags/figure.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [full](<https://devfeed.tech/tags/full.md>), [generate](<https://devfeed.tech/tags/generate.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [lattice](<https://devfeed.tech/tags/lattice.md>), [llms](<https://devfeed.tech/tags/llms.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [serving](<https://devfeed.tech/tags/serving.md>), [tetra](<https://devfeed.tech/tags/tetra.md>)

### AI overview

Tetra makes Leech-lattice quantization practical for GPU inference by encoding each 24-weight block in 48 bits and decoding it during matrix-vector multiplication with six table loads and two small lookups. Its shared 16 KiB table avoids the enormous full codebook, and the kernel reads 2.148 bits per weight, compared with 4.804 bits per weight for the authors' earlier load-time expansion approach. Full Qwen3-4B, 8B and 14B models occupy 2.70-2.73 bits per parameter, using row-specific scales, 4-bit storage for the matrices that lose most quality, and 4-bit embedding tables. On MMLU they trail 4-bit AWQ models (5.3-6.0 bits per parameter) by 2.46-4.76 points; on GSM8K they trail FP16 by 3.26-9.63 points. Reported generation rates are 57.2-113.8 tokens per second. At 4B, Tetra scores 23.6 points above llama.cpp IQ2_XXS. All reported measurements used one NVIDIA L40S GPU.

### Source excerpt

arXiv:2609.35465v2 Announce Type: replace Abstract: Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above llama.cpp's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.

## The ROI of Small Examples: Optimizing In-Context Learning Under Token Budgets

DevFeed: [The ROI of Small Examples: Optimizing In-Context Learning Under Token Budgets](<https://devfeed.tech/articles/the-roi-of-small-examples-optimizing-in-context-learning-under-token-budgets-145450.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24971>)

Author: Anxo Perez, Jorge Gab\\'in, Javier Parapar

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [few-shot](<https://devfeed.tech/topics/few-shot.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [examples](<https://devfeed.tech/tags/examples.md>), [in-context-learning](<https://devfeed.tech/tags/in-context-learning.md>), [inference](<https://devfeed.tech/tags/inference.md>), [model-performance](<https://devfeed.tech/tags/model-performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [qa](<https://devfeed.tech/tags/qa.md>)

### AI overview

Under a fixed prompt token budget, the study finds that using more short demonstrations generally improves in-context learning for question answering compared with spending the budget on a few long demonstrations. The authors grouped examples by length and tested six instruction-tuned models on Beat the AI and Natural Questions. The effect was strongest on Beat the AI; it also appeared on Natural Questions, where gains were smaller and saturated earlier with long contexts. In an ANLI R3 inference experiment, one of the two shorter length groups performed best for five of six models, though longer examples remained competitive. The results suggest demonstration length is a relevant prompt design choice alongside example count.

### Source excerpt

arXiv:2611.24971v1 Announce Type: cross Abstract: In-context learning (ICL) improves large language model performance at inference time without updating model parameters. However, these gains are not free: in practical deployments, longer prompts directly increase inference cost and latency. Under a fixed token budget, prompt construction therefore becomes a resource-allocation problem. While prior work has often focused on how many examples to include in ICL, we study the complementary question of whether the available budget is better spent on a few long demonstrations or on a larger number of shorter ones. We study this aspect in question answering (QA), by organizing candidate demonstrations into length-based bins and constructing prompts from examples within the same length range under fixed budgets. Experiments on two QA benchmarks (\textit{Beat the AI} and \textit{Natural Questions}), using six instruction-tuned LLMs, show that shorter demonstrations are generally the most effective use of prompt space. This effect is especially strong on \textit{Beat the AI}, and remains visible on NQ, although gains are smaller and saturate earlier in the long-context setting. An additional natural language inference experiment on ANLI R3 finds that one of the two shorter bins performs best for five of six models, although longer demonstrations remain competitive. Overall, our results highlight demonstration granularity as an important factor in prompt design.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimization

DevFeed: [KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimization](<https://devfeed.tech/articles/kernelopt-dispatch-aware-agentic-search-for-gpu-kernel-optimization-144934.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.30059>)

Author: Aheli Poddar, Sanskar Prasad, Arindam Samanta, Subha Chakraborty, Vishal Goyal, Rohit Singh Rathaur

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Compilers and Linkers](<https://devfeed.tech/topics/compilers-and-linkers.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [changes](<https://devfeed.tech/tags/changes.md>), [code](<https://devfeed.tech/tags/code.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [compilers](<https://devfeed.tech/tags/compilers.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

KernelOPT uses profiling-guided agents to inspect compiler artifacts, including generated Triton kernels and external library calls, then synthesize and refine kernels and evaluate the assembled model. Its four validation stages combine execution checks, multi-seed numerical comparisons, model-level checks with a float64 fallback, and an end-to-end performance gate of 1.03x. On 250 KernelBench problems run on NVIDIA H200, it reports geometric mean speedups over torch.compile of 1.40x for single operators, 1.15x for composite operators, and 1.07x for model architectures. The authors note that gains include precision changes and algebraic simplifications within evaluated tolerances, and that isolated kernel improvements can differ from end-to-end performance.

### Source excerpt

arXiv:2609.30059v3 Announce Type: replace-cross Abstract: Deep learning inference and training performance depends critically on GPU kernel efficiency. Modern compilers such as PyTorch Inductor automatically generate GPU kernels from high-level model code, but frequently underperform expert-written implementations by wide margins. Recent LLM-assisted optimizers explore these opportunities through kernel generation and refinement; yet translating isolated kernel improvements into model-level gains remains a challenge. We present KernelOPT, a profiling-guided multi-agent system for GPU kernel optimization. It inspects generated Triton kernels and external library calls in compiler artifacts to guide synthesis and refinement, then assembles and evaluates the resulting model. Four stages of validation combine execution checks, multi-seed numerical comparison, model-level checking with a float64 fallback and end-to-end performance gating ($\gamma{=}1.03$). The system supports PyTorch nn.Modules, standalone Triton kernels, and Helion kernels. It is evaluated on all 250 KernelBench problems on NVIDIA H200 which includes 100 single-operator problems (Level1), 100 composite operators (Level2), and 50 model architectures (Level3). KernelOPT achieves geometric mean speedups over torch.compile of 1.40$\times$, 1.15$\times$ and 1.07$\times$ respectively across L1, L2 and L3. Among problems exceeding 1.03$\times$ speedup, the geomeans are 2.54$\times$ (36 L1), 1.84$\times$ (23 L2), and 1.37$\times$ (11 L3). These gains include arithmetic-precision changes and task-specific algebraic simplifications under the evaluated numerical tolerances. The evaluation also reveals discrepancies between isolated kernel improvements and end-to-end performance, motivating model-level measurement alongside kernel-level optimization. Code is available at https://github.com/TorchedHat/KernelOPT.

## FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

DevFeed: [FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models](<https://devfeed.tech/articles/frontierreplay-split-free-exact-prefix-caching-for-hybrid-language-models-145128.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26020>)

Author: Zekun Wang, Yizhuo Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [attention](<https://devfeed.tech/tags/attention.md>), [branch](<https://devfeed.tech/tags/branch.md>), [caching](<https://devfeed.tech/tags/caching.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [load](<https://devfeed.tech/tags/load.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [prefix-caching](<https://devfeed.tech/tags/prefix-caching.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [question](<https://devfeed.tech/tags/question.md>)

### AI overview

FrontierReplay targets exact prefix caching in hybrid language models, where recurrent state updates limit reuse to saved positions. It identifies predictable structural boundaries, such as turn ends and known branch points, and creates exact checkpoints there without interrupting prefill: it logs a short window of recurrent-layer inputs and uses a cross-layer state-only replay kernel, reported to cost 6-37x less per token than full recomputation without host synchronization. Its retention policy prioritizes active anchors and proven branches and lends unused checkpoint slots to the attention KV cache. Across the evaluated architectures, scales, and workloads, the abstract reports 16-89% lower median time to first token, up to 1.73x higher capacity at an 80% SLO, and up to 4.26x higher goodput (3.67x at larger scales) versus existing exact caches. Versus approximate replay, it reports up to 1.54x higher goodput while preserving exact task accuracy; approximation suffered up to a 17% accuracy drop on long contexts.

### Source excerpt

arXiv:2611.26020v1 Announce Type: new Abstract: Hybrid language models combine self-attention with recurrent layers whose in-place state updates restrict prefix reuse to exact saved positions. Saving states on a coarse grid wastes reusable prefixes, whereas existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load. We observe that across multi-turn chat, agent loops, document question answering, and tree search, prefix reuse almost always ends at structural boundaries predictable in advance, such as turn ends and known branch points. We present FrontierReplay, a prefix caching system that materializes exact checkpoints at these boundaries without cutting the prefill. While the forward pass runs uncut, FrontierReplay logs a short window of recurrent-layer inputs and invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization. A frontier-aware retention policy prioritizes active anchors and proven branches while lending dead checkpoint slots to the attention KV cache. Across diverse hybrid architectures, model scales, and workloads, FrontierReplay delivers clear gains over existing exact caches, lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up to 4.26x (3.67x at larger scales). Compared with approximate replay methods, FrontierReplay delivers up to 1.54x higher goodput while preserving exact task accuracy, avoiding the up to 17% accuracy drop suffered by approximation on long contexts.

## GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers

DevFeed: [GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers](<https://devfeed.tech/articles/glide-a-fast-calibrated-decision-model-distilled-from-reasoning-teachers-145678.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26509>)

Author: Sahibzada Allahyar, Julia White, Dhruv Atreja, George Hurn-Maloney, Urchade Zaratiana

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [calibrated-decision-model](<https://devfeed.tech/tags/calibrated-decision-model.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [di](<https://devfeed.tech/tags/di.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [fast](<https://devfeed.tech/tags/fast.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model](<https://devfeed.tech/tags/model.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-model](<https://devfeed.tech/tags/reasoning-model.md>)

### AI overview

GLiDE transfers reasoning capability into single-pass decision models that score candidate answers without generating text or intermediate reasoning. The 0.4B-27B family uses Qwen hybrid language models with bidirectional interactions among context, question, and candidates, plus soft-target distillation from multiple teachers--including a reasoning model--and progressive post-training on varied decision tasks. Parameter-free recurrent depth adds inference-time computation without token generation. The abstract reports that each of five sizes outperforms evaluated alternatives of equal or smaller size on the Hugging Face benchmark. The 27B model scores 71.52 on a hidden set with 2.0% expected calibration error and 92.9 ms median latency on one GPU; checkpoint averaging and recurrent depth raise the score to 73.37, with 2.1% calibration error.

### Source excerpt

arXiv:2611.26509v1 Announce Type: new Abstract: Decision models enable fast, confidence-aware decisions by scoring candidate answers in a single forward pass, without generating text or intermediate reasoning. Yet reasoning can substantially improve decision quality, raising the question of whether its benefits can be retained without the cost of generating it. We investigate this question with \textbf{GLiDE}, a family of decision models spanning 0.4B to 27B parameters. Building on Qwen hybrid language models, GLiDE enables bidirectional interactions between the context, question, and candidate answers by removing causal masking from full-attention layers. To transfer reasoning capabilities into single-pass inference, we combine soft-target distillation from multiple teachers, including a reasoning model, with progressive post-training on diverse decision tasks. We further explore parameter-free recurrent depth to increase inference-time computation without generating tokens. Across five model sizes, GLiDE lies on the performance--parameter Pareto frontier of the Hugging Face \DI{}, with each model outperforming evaluated alternatives of equal or smaller size. \dmtwo{} (27B) achieves a hidden-set score of 71.52, with an expected calibration error of 2.0\% and a median latency of 92.9\,ms on a single GPU. Further checkpoint averaging and parameter-free recurrent depth yield GLiDE~27B, which reaches 73.37 with an expected calibration error of 2.1\%.

## One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion

DevFeed: [One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion](<https://devfeed.tech/articles/one-latent-many-tokens-jointly-learning-compressed-embeddings-for-efficient-language-diffusion-145381.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.33698>)

Author: Yulin Yuan, Ying Zhang, Xiangming Meng

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [flow](<https://devfeed.tech/tags/flow.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [generation](<https://devfeed.tech/tags/generation.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [structured](<https://devfeed.tech/tags/structured.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

JPEG-DLM reduces diffusion language generation cost by jointly training a compressor, flow matching model, and decoder, so compressed embeddings are designed to be easier to model and decode. This contrasts with two-stage methods that fix the compressed embedding space before training the diffusion model, which the abstract says can hinder modeling and token recovery. The authors report the highest throughput among recent diffusion and flow models on LM1B and OWT, with Gen-PPL lower than most. On OWT at a 0.5 compression rate, JPEG-DLM reports Gen-PPL 34.52 and about 2.3x ELF's throughput.

### Source excerpt

arXiv:2609.33698v2 Announce Type: replace Abstract: Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens.JPEG-DLM achieves the highest throughput among recent diffusion and flow models on LM1B and OWT, with a Gen-PPL lower than most of them. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling.Code is available at https://github.com/infiniteYuanyl/JPEG-DLM.

## TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

DevFeed: [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](<https://devfeed.tech/articles/triaxialkv-toward-extreme-low-precision-kv-cache-quantization-for-agentic-inference-tasks-144895.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.17170>)

Author: Hanzhang Shen, Haoran Wu, Yiren Zhao, Robert Mullins

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [compression](<https://devfeed.tech/tags/compression.md>), [computer](<https://devfeed.tech/tags/computer.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom](<https://devfeed.tech/tags/custom.md>), [decode](<https://devfeed.tech/tags/decode.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [image](<https://devfeed.tech/tags/image.md>), [implement](<https://devfeed.tech/tags/implement.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

TriAxialKV reduces KV-cache memory for agentic inference by tagging tokens across recency, modality, and semantic role, calibrating sensitivity for each tag, and assigning INT2 or INT4 precision within a fixed memory budget. Its serving system combines calibration, quantization and memory management, and fused Triton decode kernels. In the reported OSWorld computer-use evaluation with Qwen3-VL-32B-Thinking, it matched SGLang's accuracy using a BF16 cache while supporting 4.5x KV-cache capacity and 30% higher end-to-end throughput on real GPUs.

### Source excerpt

arXiv:2605.17170v2 Announce Type: replace Abstract: Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bit-widths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache capacity and achieving 30% higher end-to-end throughput, when running on real GPU systems.

## Closing Performance Gap for Diffusion Language Models under Low Inference Budgets

DevFeed: [Closing Performance Gap for Diffusion Language Models under Low Inference Budgets](<https://devfeed.tech/articles/closing-performance-gap-for-diffusion-language-models-under-low-inference-budgets-145100.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25763>)

Author: Hao Wu, Sipeng Chen, Jingwen Fu, Yuwang Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [budget](<https://devfeed.tech/tags/budget.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [mathematical-reasoning](<https://devfeed.tech/tags/mathematical-reasoning.md>), [mbpp](<https://devfeed.tech/tags/mbpp.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CaRR improves submission quality for editable diffusion language models under tight inference budgets without adding backbone forwards. Its Token Commitment Remasking (TCR) remasks selected native pure replacements and refills them on the next scheduled edit using updated context; Submission Commitment Recovery (SCR) uses the refinement trajectory to recover supported candidates when the final response fails task-specific output requirements. Both work within the existing mask/edit schedule. Across four budgets, experiments in code generation and mathematical reasoning report consistent gains in average accuracy over native ME-DLM, largest at tight budgets. At one-eighth of the full scheduled forward budget, CaRR raises six-metric average accuracy by 7.44 percentage points, with gains of 11.37 points on MBPP and 10.05 on MBPP+.

### Source excerpt

arXiv:2611.25763v1 Announce Type: cross Abstract: Diffusion language models offer an architectural route to low-budget generation by updating multiple tokens in parallel, which can reduce the number of sequential decoding steps relative to token-by-token autoregressive decoding. However, realizing this advantage at low budgets is challenging, as generation quality often declines when the number of denoising and refinement steps is reduced. Editable diffusion models address this quality loss through post-draft refinement, but useful intermediate results may still be lost before final submission. We introduce CaRR (Commitment-aware Refinement and Recovery), a training-free framework which improves submission quality from a fixed editable checkpoint through updated-context refill and trajectory-based recovery within its existing forward schedule. Token Commitment Remasking (TCR) remasks selected native pure replacements and refills them on the next scheduled edit forward using the updated context. Submission Commitment Recovery (SCR) uses the resulting refinement trajectory to recover supported candidates when the final response does not satisfy task-specific output requirements. Both modules operate within the native mask/edit schedule without additional backbone forwards. Experiments on code generation and mathematical reasoning show consistent gains in average accuracy over native ME-DLM across four inference budgets, with the largest gains under tight budgets. At one-eighth of the full scheduled forward budget, CaRR improves the six-metric average accuracy by 7.44 pp, including gains of 11.37 and 10.05 on MBPP and MBPP+.

## Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts

DevFeed: [Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts](<https://devfeed.tech/articles/relax-within-balance-across-load-balancing-under-composition-shifts-in-vision-language-mixture-of-experts-145359.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.00574>)

Author: Ziang Wu, Peng Jin, Qishen Yin, Munan Ning, Hao Li, Peizhen Zhang, Huihui Zhou, Li Yuan

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [balance](<https://devfeed.tech/tags/balance.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [cancellation](<https://devfeed.tech/tags/cancellation.md>), [changes](<https://devfeed.tech/tags/changes.md>), [code](<https://devfeed.tech/tags/code.md>), [composition](<https://devfeed.tech/tags/composition.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fast](<https://devfeed.tech/tags/fast.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [flat](<https://devfeed.tech/tags/flat.md>), [gap](<https://devfeed.tech/tags/gap.md>), [image](<https://devfeed.tech/tags/image.md>), [language](<https://devfeed.tech/tags/language.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

On a fixed vision-language MoE router, balancing image and text tokens in one pooled auxiliary loss can conceal opposing modality load errors: they cancel near the training image-to-text ratio, but load skews as resolution, tiling, or image count changes that mix. The paper derives a quadratic response law under fixed modality profiles, with the image-text load gap governing sensitivity. It also reports that patches from an image route similarly as patch counts vary, allowing large images to dominate flat pooling. ReBA separates modality losses by token fraction and weights images equally, allowing per-image specialization while balancing visual load overall. Across four split backbones, the authors report stable balance across resolutions, tilings, and image counts without hurting accuracy; large modality gaps in released native MoEs suggest potential for native post-training.

### Source excerpt

arXiv:2608.00574v2 Announce Type: replace-cross Abstract: Vision-language mixture-of-experts (MoE) models such as Qwen3-VL need balanced loads for fast parallel serving. Standard training pools image and text tokens into one auxiliary loss, assuming training balance holds at inference. Yet we find a surprising mismatch: on a fixed router, balance shifts sharply whenever resolution, dynamic tiling, or image count changes the image-to-text token ratio. Splitting load by modality reveals cross-modal load cancellation: opposing image and text errors cancel near the training ratio, hiding strong skew until the mix changes. To quantify this shift, we derive an exact quadratic law: under fixed modality profiles, one reference run predicts the full response curve, and the image-text gap governs sensitivity. Meanwhile, patches within an image route similarly while patch counts vary, letting large images dominate flat pooling. We propose ReBA (Relax Within, Balance Across) to separate modality losses by token fraction and weight images equally, letting each image specialize while keeping overall visual load even. Across four split backbones, ReBA turns fragile single-ratio balance into stable balance across resolutions, tilings, and image counts without hurting accuracy. Large modality gaps in released native MoEs also show its promise for native post-training. Code is available at https://github.com/ZiangWu-77/ReBA.

## One Schedule Does Not Fit All: Adaptive Inference Schedules for Diffusion Language Models

DevFeed: [One Schedule Does Not Fit All: Adaptive Inference Schedules for Diffusion Language Models](<https://devfeed.tech/articles/one-schedule-does-not-fit-all-adaptive-inference-schedules-for-diffusion-language-models-145114.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25895>)

Author: Rabimba Karanjai, Linsey Pang, Hemanth Hegadehalli Madhavarao, Richard Williamson, Uma Kona, Prakhar Mehrotra

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [full](<https://devfeed.tech/tags/full.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grid](<https://devfeed.tech/tags/grid.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [schedules](<https://devfeed.tech/tags/schedules.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

The cheapest quality-preserving denoising schedule varies by task, model, and sampler. Task-Calibrated Denoising Allocation (TCDA) uses paired calibration data, a utility tolerance, and degeneration guardrails with simultaneous one-sided bootstrap bounds to certify candidate schedules. It chooses the cheapest certified option or falls back to the full schedule; its probability of deploying an infeasible schedule is asymptotically bounded by α. Candidates include early prefix decoding, state-reusing continuation, reduced time grids, and confidence-threshold decoding. Across seven datasets and two model families, reported denoiser-evaluation reductions range from 1.33x to 8x, varying by task and model. Fresh-seed validation confirmed each selection, and across 500 calibration splits false certification stayed below 2% at a nominal 5% level. The abstract also reports that decoded-token commitment appears more informative for schedule selection than remaining latent discretization error. For reproducible unconditional-generation comparisons with the ELF reference sampler, its randomly drawn time grid must be fixed.

### Source excerpt

arXiv:2611.25895v1 Announce Type: cross Abstract: Diffusion language models typically use a fixed denoising schedule across workloads. We show that the cheapest schedule that preserves quality depends on the task, model, and sampler. We propose Task-Calibrated Denoising Allocation (TCDA), a deployment rule with an explicit error guarantee. Given candidate samplers, paired calibration data, a utility tolerance, and guardrails against degeneration, TCDA uses simultaneous one-sided bootstrap bounds to certify which candidates meet both the utility and guardrail criteria. It deploys the cheapest certified candidate or falls back to the full schedule. The probability of deploying an infeasible schedule is asymptotically at most alpha. Candidates include early prefix decoding, state-reusing continuation, reduced time grids, and confidence-threshold decoding. Across seven datasets and two model families, TCDA certifies reductions in denoiser evaluations by factors of 3.4 on XSum and 2.0 on WMT14 for ELF-B; 8.0 on WMT14 and 4.1 on GSM8K for LLaDA-8B; and 1.33 on OpenWebText for both ELF-M and ELF-L. A validation protocol fixed before generation confirmed every selection on fresh seeds. Across 500 calibration splits, false certification remained below 2% in every setting at a nominal level of 5%. Certified schedules vary across tasks and model scales. Trajectory measurements suggest that when decoded tokens commit is more informative for schedule selection than the remaining latent discretization error. We also find that the ELF reference sampler draws its time grid at random, so reproducible schedule comparisons for unconditional generation require fixing the grid.

## TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

DevFeed: [TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming](<https://devfeed.tech/articles/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and-layer-wise-streaming-145202.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26619>)

Author: Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [memory pool](<https://devfeed.tech/topics/memory-pool.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [components](<https://devfeed.tech/tags/components.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context](<https://devfeed.tech/tags/context.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [execution](<https://devfeed.tech/tags/execution.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [management](<https://devfeed.tech/tags/management.md>), [serving](<https://devfeed.tech/tags/serving.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [transparent](<https://devfeed.tech/tags/transparent.md>)

### AI overview

TideCache serves hybrid LLMs by coordinating GPU and host memory for global-attention and bounded-state caches. Its Transparent Cache Pool reallocates physical pages while preserving virtual addresses and backend interfaces; its Cache Streamer uses bounded-state computation intervals to prefetch global-attention prefixes into shared GPU slots, overlapping transfers with computation while keeping generated KV resident. A Request Scheduler chooses resident or streamed execution and adjusts capacity based on request size and queue state. Implemented in SGLang, TideCache was evaluated on GB300 and GH200 across seven model/platform configurations. Compared with SGLang using HiCache, it completed up to 2.04x as many agent turns, reduced overall P50 per-token latency by 20-59%, and improved agent P50 TTFT by 1.40-3.05x.

### Source excerpt

arXiv:2611.26619v1 Announce Type: cross Abstract: Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.

## DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention

DevFeed: [DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention](<https://devfeed.tech/articles/dts-delta-based-tile-scoring-for-efficient-long-context-sparse-attention-145120.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25938>)

Author: Emre U\\u{g}ur, Jiawen Qi, Qinyu Chen

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [concurrently](<https://devfeed.tech/tags/concurrently.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [delta](<https://devfeed.tech/tags/delta.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [free](<https://devfeed.tech/tags/free.md>), [full](<https://devfeed.tech/tags/full.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [h100](<https://devfeed.tech/tags/h100.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>)

### AI overview

DTS reduces the cost of scoring blocks for sparse long-context attention by choosing query and key anchors based on local vector changes: it uses fewer anchors in redundant regions and more in heterogeneous ones. Parallel delta packing speeds selection by scanning tile-aligned chunks concurrently; selected tiles are computed with the original query, key, and value vectors. The method is training-free. On LLaMA-3.1-8B-Instruct, average RULER and LongBench scores were within 0.10 percentage points of full attention, with RULER tested up to 128K tokens. At 128K, scoring-pipeline latency was 38% lower than XAttention; at 512K, prefill was 3.3x faster than a PyTorch full-attention baseline using FlashAttention-2 on four NVIDIA H100 GPUs.

### Source excerpt

arXiv:2611.25938v1 Announce Type: new Abstract: Large language models (LLMs) increasingly need to process long documents and codebases, making long-context attention a major prefill bottleneck. Block sparse attention reduces computation by skipping unimportant blocks of the attention matrix, but scoring blocks accurately can itself be expensive. We propose Delta-Based Tile Scoring (DTS), a training-free, content-adaptive tile scoring method for sparse block attention. DTS selects query and key anchors according to local vector changes, using fewer representatives in redundant regions while retaining more detail in heterogeneous regions. To accelerate anchor selection, parallel delta packing partitions the query and key sequences into tile-aligned chunks and performs bounded local scans over these chunks concurrently. The selected tiles are then computed using the original query, key, and value vectors. On LLaMA-3.1-8B-Instruct, DTS keeps average RULER and LongBench scores within 0.10 percentage points of full attention, with RULER evaluated up to 128K tokens. At 128K tokens, DTS reduces measured scoring-pipeline latency by 38% compared with XAttention. On four NVIDIA H100 GPUs, it achieves a 3.3$\times$ prefill speedup over a PyTorch full-attention baseline using FlashAttention-2 at 512K tokens.

## Cautious Context Steering for Language Model Personalization

DevFeed: [Cautious Context Steering for Language Model Personalization](<https://devfeed.tech/articles/cautious-context-steering-for-language-model-personalization-145360.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.05813>)

Author: Gihoon Kim, Jeyoung Lee, Suhan Woo, Jangwon Oh, Sekwon Oh, Minsu Jeon, Euntai Kim

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [personalization](<https://devfeed.tech/topics/personalization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [ccs](<https://devfeed.tech/tags/ccs.md>), [context](<https://devfeed.tech/tags/context.md>), [cos](<https://devfeed.tech/tags/cos.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [format](<https://devfeed.tech/tags/format.md>), [generation](<https://devfeed.tech/tags/generation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [pass](<https://devfeed.tech/tags/pass.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [steering](<https://devfeed.tech/tags/steering.md>)

### AI overview

Cautious Context Steering (CCS) uses a lightweight adapter to decide when and how strongly a frozen language model should use user context. Trained through cautious distillation, it follows context-steered predictions where context helps predict a preferred token and the context-free distribution elsewhere. The abstract reports stronger preference alignment than training-free context methods and trained personalization baselines. On the introduced GSM8K-P benchmark, CCS maintains task accuracy while improving adherence to task-specific formats. Its adapter adds less than 0.4% of the backbone's parameters; at inference, one context-conditioned model pass plus adapter evaluation per decoding step reduces generation time on PRISM by about 35-36% compared with Context Steering, which uses two passes.

### Source excerpt

arXiv:2608.05813v2 Announce Type: replace Abstract: User context can personalize a language model without per-user fine-tuning, but its influence is not equally helpful at every token. Context Steering (CoS) uses a fixed steering strength throughout generation and requires two LM forward passes per step. We propose Cautious Context Steering (CCS), a lightweight adapter that learns when and how strongly to steer a frozen LM using user context. We train the adapter through cautious distillation, using oracle targets built from predictions with and without context. It learns to match the steered distribution where context helps predict the user's preferred token and the context-free base distribution elsewhere. The CCS adapter outperforms both training-free context-based methods and trained personalization baselines in preference alignment across benchmarks. We also introduce GSM8K-P to evaluate task accuracy and task-specific format preferences together. CCS achieves high format adherence while maintaining task accuracy, offering a better balance than the baselines. The CCS adapter adds less than 0.4% of the backbone's parameters. At inference, each decoding step uses one context-conditioned LM forward pass and a lightweight adapter evaluation, reducing generation time on PRISM by approximately 35-36% compared with CoS.

## Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression

DevFeed: [Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression](<https://devfeed.tech/articles/complete-svd-a-functional-norm-approach-to-low-rank-llm-compression-145134.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26059>)

Author: Alper Kalle, Theo Rudkiewicz, Mohamed-Oumar Ouerfelli, Mohamed Tamaazousti

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [low-rank](<https://devfeed.tech/tags/low-rank.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [svd](<https://devfeed.tech/tags/svd.md>)

### AI overview

Complete-SVD compresses LLMs by minimizing the L2 distance between original and compressed model logits, giving equal weight to every logit change. Linearization and a Kronecker-factored approximation turn this objective into a weighted SVD with a closed-form solution. Because its output-side factor is intractable to compute directly, the method estimates it without bias using random probes, requiring K backward passes per sequence regardless of vocabulary size or sequence length. With uniform ranks and WikiText-2 calibration, it reports the lowest C4 and PTB perplexity at every matched setting, and at compression ratio 0.7 with LEMS rank allocation. The related KL-SVD variant lowers calibration perplexity in most settings and raises downstream accuracy, but also raises cross-corpus perplexity. Complete-SVD improves IFEval accuracy over the strongest prior output-aware methods throughout the reported comparisons.

### Source excerpt

arXiv:2611.26059v1 Announce Type: new Abstract: Post-training low-rank compression of neural networks must not only approximate the original weights but also preserve the model's input-output behavior. Recent output-aware methods measure the compression error beyond the compressed layer, at the model output, weighting logit changes by the observed next token or by the model's predicted probabilities. We propose Complete-SVD, a method for large language models that minimizes the $L_2$ distance between the logits of the original and compressed models, weighting all logit changes equally. Linearizing the network around the original weights, together with a Kronecker-factored approximation, reduces this objective to a weighted SVD with a closed-form solution. Its output-side factor is intractable to compute; we estimate it without bias from random probes, at a cost of $K$ backward passes per sequence, independent of vocabulary size and sequence length. At matched settings with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every setting, and at ratio 0.7 under LEMS rank allocation. Replacing the $L_2$ distance by the Kullback-Leibler divergence (KL-SVD), with the same estimation framework, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity. Complete-SVD also improves IFEval instruction-following accuracy over the strongest prior output-aware methods throughout.

## Global Importance Estimation for KV Cache Eviction

DevFeed: [Global Importance Estimation for KV Cache Eviction](<https://devfeed.tech/articles/global-importance-estimation-for-kv-cache-eviction-145448.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24969>)

Author: Seojin Kim, Noseong Park

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

PRKV is a training-free method for reducing KV-cache memory during long-context LLM inference. It uses Personalized PageRank on attention-induced transition graphs to propagate token-importance estimates across multiple hops, beyond recency or instantaneous attention, without modifying the model. The abstract reports evaluations across multiple LLM families on long-context understanding, sparse retrieval, and chain-of-thought reasoning: at compression levels up to 90%, PRKV achieves competitive performance against established eviction baselines while reducing peak memory with low end-to-end overhead. The supplied text does not specify benchmark details or individual results.

### Source excerpt

arXiv:2611.24969v1 Announce Type: cross Abstract: Memory consumption of the key-value (KV) cache remains a central bottleneck for long-context LLM inference. We introduce PRKV, a training-free KV eviction method that estimates multi-hop global token importance by applying Personalized PageRank over attention-induced transition graphs. This enables importance propagation beyond recency or instantaneous attention, without modifying the model. We benchmark PRKV on long-context understanding, sparse retrieval, and chain-of-thought reasoning tasks using multiple LLM families. Under moderate to aggressive compression ratios, including up to 90%, PRKV achieves competitive task performance against established eviction baselines while reducing peak memory usage with low end-to-end overhead. These results demonstrate that multi-hop global influence estimation yields a favorable accuracy--memory trade-off for scalable long-context inference.

## Aggron: Aggressive Compression of Agentic KV Caches during Head-of-Line Stalls in the Request Queue

DevFeed: [Aggron: Aggressive Compression of Agentic KV Caches during Head-of-Line Stalls in the Request Queue](<https://devfeed.tech/articles/aggron-aggressive-compression-of-agentic-kv-caches-during-head-of-line-stalls-in-the-request-queue-146134.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26435>)

Author: Rishabh Ravi, Tony Hong, Chen Wang, Yue Zhu, Mengmei Ye, Hubertus Franke, Fan Lai, Josep Torrellas

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [queueing theory](<https://devfeed.tech/topics/queueing-theory.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [compression](<https://devfeed.tech/tags/compression.md>), [context-compression](<https://devfeed.tech/tags/context-compression.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load](<https://devfeed.tech/tags/load.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [queue](<https://devfeed.tech/tags/queue.md>), [request](<https://devfeed.tech/tags/request.md>), [serving](<https://devfeed.tech/tags/serving.md>), [stalled](<https://devfeed.tech/tags/stalled.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Aggron addresses KV-cache limits in long-running LLM agents by summarizing selected cache content at every turn, keeping each summarization request lightweight. It runs these requests in a separate queue while the corresponding agent request is stalled behind a head-of-line block, using capacity left idle in the main queue; it launches summarization only when it predicts completion before that agent request executes. The abstract says this approach retains accuracy and, versus state-of-the-art asynchronous summarization at high load, reduces P99 program latency by 27-31% and increases throughput by 1.34-1.40x. The supplied text is an abstract, so it does not provide evaluation details or broader comparisons.

### Source excerpt

arXiv:2611.26435v1 Announce Type: new Abstract: LLM agents run long, multi-turn programs whose contexts grow at every step, and serving them is limited by KV cache capacity. To alleviate this limitation, systems use context compression techniques, which typically wait until the KV cache crosses a large threshold and then summarize large spans of history at once, either in the background while the agent progresses or on the critical path. These summarizations can be done using the same model running on the same engine, or using the same or a different model running on extra hardware. When running on the same engine, summarization competes with the requests being served for both compute and memory capacity. To maximize the cost-effectiveness of summarization, this paper presents a new serving system called {\em Aggron}. Aggron summarizes agentic requests at every turn and selectively summarizes only certain parts of the KV cache. As a result, Aggron retains accuracy and keeps summarization requests lightweight. Moreover, Aggron launches summarization requests from a separate {\em Summarization Queue} while the corresponding agentic request whose KV cache we want to compress is stalled in the main request queue due to head-of-line stalls. The Summarization queue utilizes compute and memory capacity left over by the head-of-line stalls in the main queue. Further, to maximize efficiency, the server issues summarizations only when it predicts that they can complete before the corresponding agentic request will finally execute. In our evaluation, we compare Aggron to state-of-the-art asynchronous summarization. At high load, Aggron reduces P99 program latency by $27$--$31\%$, and raises throughput by $1.34$--$1.40\times$.

## Characterizing High Bandwidth Flash for LLM Serving

DevFeed: [Characterizing High Bandwidth Flash for LLM Serving](<https://devfeed.tech/articles/characterizing-high-bandwidth-flash-for-llm-serving-144948.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39131>)

Author: Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [systems](<https://devfeed.tech/topics/systems.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [endurance](<https://devfeed.tech/tags/endurance.md>), [hbm](<https://devfeed.tech/tags/hbm.md>), [high-bandwidth-flash](<https://devfeed.tech/tags/high-bandwidth-flash.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [simulations](<https://devfeed.tech/tags/simulations.md>)

### AI overview

Trace-driven simulations of high-throughput agentic LLM serving find that adding high-bandwidth flash (HBF) to an HBM-HBF-host memory hierarchy can reduce completion time by 21.1-86.9% versus HBM-only systems in the evaluated workloads. Modeled energy savings reach 62.4%, varying with workload and weight placement. Because HBF has access costs and limited write endurance, the study pairs data placement with buffered cache-aware scheduling; in one evaluated configuration, this scheduling raises estimated HBF write lifetime from 0.99 to 13.63 years. These are simulation results, not measurements of deployed systems.

### Source excerpt

arXiv:2609.39131v3 Announce Type: replace Abstract: Large language model (LLM) serving requires substantial memory to store model weights and KV caches. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it important to retain KV cache for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for LLM serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 21.1-86.9% relative to HBM-only systems. Modeled energy savings reach 62.4%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 0.99 to 13.63 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling in serving LLM with HBF, positioning HBF as an actively utilized lower-level memory tier rather than passive storage.

## Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling

DevFeed: [Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling](<https://devfeed.tech/articles/ask-in-the-crowd-differentially-private-llm-inference-via-dummy-augmented-shuffling-145566.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25669>)

Author: Zhihao Liu, Zixiong Guo, Shuo Shao, Yu He, Wenli Wang, Meihui Chen, Di Wang, Yuke Hu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [balance](<https://devfeed.tech/tags/balance.md>), [cache](<https://devfeed.tech/tags/cache.md>), [classification](<https://devfeed.tech/tags/classification.md>), [client](<https://devfeed.tech/tags/client.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-efficiency](<https://devfeed.tech/tags/inference-efficiency.md>), [inference-framework](<https://devfeed.tech/tags/inference-framework.md>), [input](<https://devfeed.tech/tags/input.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [private](<https://devfeed.tech/tags/private.md>)

### AI overview

DP-SPD hides a user's real query among client-generated dummy queries, shuffles the queries before sending them to the LLM server, and reuses transformer key-value caches for queries with shared prefixes during decoding. The paper analyzes its privacy guarantee and evaluates classification and generation utility and protection against QIA, EIA, and GPT-based attacks. The abstract reports strong inference-time privacy, near-plaintext utility, and improved efficiency, but gives no quantitative results or implementation details.

### Source excerpt

arXiv:2611.25669v1 Announce Type: new Abstract: Large language models are increasingly deployed as LLM-as-a-Service (LLMaaS), where user queries are sent to remote LLMs for inference, raising severe privacy concerns about the exposure of sensitive user inputs. Differentially private inference mitigates these risks by injecting noise into input embeddings. However, existing methods struggle to balance privacy and utility, as stronger privacy protection often leads to substantial accuracy degradation in LLM inference. In this work, we propose DP-SPD, a differentially private LLM inference framework that protects user queries by hiding the real query among client-generated dummy queries. All queries are randomly shuffled before being sent to the server, preventing direct identification of the user's true intent. To improve inference efficiency, we exploit the shared-prefix structure among queries and reuse transformer key--value (KV) caches during decoding. We analyze the privacy guarantee of DP-SPD and evaluate its utility on classification and generation tasks, as well as its privacy protection under QIA, EIA, and GPT-based attacks. Experimental results show that our method achieves strong inference-time privacy protection and near-plaintext utility while improving inference efficiency through shared-prefix KV-cache reuse.

## HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

DevFeed: [HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs](<https://devfeed.tech/articles/hisa-sparsifying-the-hidden-dimension-of-rope-attention-in-llms-145170.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26363>)

Author: Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [kernels](<https://devfeed.tech/topics/kernels.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rope](<https://devfeed.tech/tags/rope.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>)

### AI overview

HiSA makes attention sparse in the hidden feature dimension: it keeps query and key projections intact but computes only part of their dot product, using a static, parameter-free rule based on token positions and RoPE structure. The authors report that models trained from scratch with HiSA outperform those using the full dot product while computing about one third as many query-key products. At inference, they report faster decoding and lower memory use, and describe FlashAttention-3 kernels for both training and inference. They also report that dense attention learns to suppress the score components HiSA omits. These findings are from the supplied abstract.

### Source excerpt

arXiv:2611.26363v1 Announce Type: new Abstract: Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.

## Supportive Token Revealing for Fast Diffusion Language Model Decoding

DevFeed: [Supportive Token Revealing for Fast Diffusion Language Model Decoding](<https://devfeed.tech/articles/supportive-token-revealing-for-fast-diffusion-language-model-decoding-144901.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.04236>)

Author: Giries Abu Ayoub, Mario Barbara, Llu\\'is Pastor-P\\'erez, Tanja Bien, Aneesh Barthakur, Alaa Maalouf, Loay Mualem

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [axon](<https://devfeed.tech/tags/axon.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fast](<https://devfeed.tech/tags/fast.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

AXON is a training-free add-on for parallel diffusion language model decoders that aims to make remaining uncertain tokens easier to denoise. It monitors those tokens and, when they appear to need more context, uses attention, uncertainty, and confidence signals to select confident masked tokens as anchors for later denoising. This shifts the reveal criterion from which token is safest to reveal to which reveal would best support other tokens, while leaving the base decoder in place. On reasoning and code-generation benchmarks across multiple diffusion models, the abstract reports improved quality-latency trade-offs, often with fewer function evaluations while maintaining or improving accuracy.

### Source excerpt

arXiv:2606.04236v2 Announce Type: replace-cross Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

## Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

DevFeed: [Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization](<https://devfeed.tech/articles/multi-objective-structured-pruning-of-llms-for-latency-and-model-size-optimization-145356.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.22583>)

Author: Muhammad Junaid Ali, Smail Niar, El-Ghazali Talbi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [size](<https://devfeed.tech/tags/size.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

The paper proposes a hardware-aware, two-stage structured pruning framework to balance LLM latency and model size for edge deployment. First, multi-objective depth pruning removes entire attention and MLP blocks; then Parallel Bayesian Optimization selects layer-wise pruning ratios under latency constraints, while importance-based strategies choose components within each layer's budget. The abstract reports that across multiple LLMs pruned by 37.5% and 50%, the approach outperformed existing methods on commonsense reasoning while reducing inference cost, with minimal reported impact on commonsense reasoning and zero-shot performance. The excerpt gives no specific model names, latency measurements, or detailed evaluation results.

### Source excerpt

arXiv:2607.22583v4 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.

[Next page](<https://devfeed.tech/topics/llm-optimization.md?cursor=WyIyMDI2LTExLTIwVDA1OjAwOjAwKzAwOjAwIiwgIjJkYjIxZjlmLTgyYjAtNDVlYy04YTNlLTJhNmRjNDhiM2UxOSJd>)