# NVFP4

NVFP4 is a 4-bit data format and quantization recipe in NVIDIA Transformer Engine, using hierarchical block and global scaling for low-precision training.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling

DevFeed: [AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling](<https://devfeed.tech/articles/ascmla-unlocking-fp4-efficiency-for-long-context-mla-decoding-through-asymmetric-scaling-146129.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26106>)

Author: Zechun Zhou, Zhiyong Zheng, Xiaomin Zhang, Lin Ye, Han Li, Jingwei Sun, Guangzhong Sun, Jing Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mla](<https://devfeed.tech/tags/mla.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

AscMLA addresses the mismatch between K/V scaling requirements when FP4 is applied to MLA's shared latent. It combines asymmetric K/V scaling with pipelined scale delivery to process one FP4 payload directly, avoiding the dequantization or separate encodings used by existing approaches, which add execution or storage overhead. The abstract reports 38.2% less cache memory than FP8 and geometric-mean decoding speedups of 1.35x over TRT-LLM's FP8 kernels and 1.26x over CAKE's NVFP4 MLA kernels.

### Source excerpt

arXiv:2611.26106v1 Announce Type: new Abstract: Long-context decoding repeatedly reads a growing KV cache. FP4 reduces cache storage and traffic but needs block scaling to handle its limited dynamic range. Applying FP4 to multi-head latent attention (MLA) requires reconciling different K/V scaling requirements on a shared latent. Existing approaches address this mismatch through dequantization or separate encodings, adding execution or storage overhead. We present AscMLA, which combines asymmetric K/V scaling with pipelined scale delivery to process a single FP4 payload directly. AscMLA uses 38.2% less cache memory than FP8 and achieves geometric-mean speedups of 1.35x over TRT-LLM's FP8 and 1.26x over CAKE's NVFP4 MLA kernels.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## NVIDIA Vera Rubin NVL72 on CoreWeave: 10x More Tokens Per Megawatt Than Blackwell

DevFeed: [NVIDIA Vera Rubin NVL72 on CoreWeave: 10x More Tokens Per Megawatt Than Blackwell](<https://devfeed.tech/articles/nvidia-vera-rubin-nvl72-on-coreweave-10x-more-tokens-per-megawatt-than-blackwell-144651.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/nvidia-vera-rubin-nvl72-on-coreweave-10x-more-tokens-per-megawatt-than-blackwell>)

Author: CoreWeave

Published: 2026-11-19T22:40:29Z

Content type: news

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-security](<https://devfeed.tech/tags/agentic-security.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [blog](<https://devfeed.tech/tags/blog.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [first](<https://devfeed.tech/tags/first.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [megawatt](<https://devfeed.tech/tags/megawatt.md>), [milestone](<https://devfeed.tech/tags/milestone.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [performance](<https://devfeed.tech/tags/performance.md>), [silicon](<https://devfeed.tech/tags/silicon.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [vera-rubin-nvl72](<https://devfeed.tech/tags/vera-rubin-nvl72.md>)

### AI overview

In CoreWeave's DeepSeek R1 evaluation, Vera Rubin NVL72 delivered 10x the tokens per second per megawatt of NVIDIA GB200 NVL72 at a matched interactivity target (TPS/user). Both systems used inference optimizations including expert parallelism, NVFP4, multi-token prediction, and disaggregated prefill and decode with TensorRT-LLM and Dynamo. CoreWeave attributes the result's relevance to reasoning workloads' high token demand and describes Vera Rubin's 72 Rubin GPUs, 36 Vera CPUs, 260 TB/s NVLink 6 fabric, and native NVFP4 support. The measurement is an initial result; CoreWeave expects further gains from tuning and says it will publish updated numbers. Customer benefits in the excerpt are expectations, not reported measured outcomes.

### Source excerpt

CoreWeave publishes first-ever measured silicon performance of NVIDIA Vera Rubin NVL72, marking a new milestone in AI infrastructure.

## SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

DevFeed: [SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference](<https://devfeed.tech/articles/sharq-bridging-activation-sparsity-and-fp4-quantization-for-llm-inference-142535.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.26587>)

Author: Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [activation](<https://devfeed.tech/tags/activation.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [block](<https://devfeed.tech/tags/block.md>), [bridging](<https://devfeed.tech/tags/bridging.md>), [code](<https://devfeed.tech/tags/code.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [error](<https://devfeed.tech/tags/error.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [formats](<https://devfeed.tech/tags/formats.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>)

### AI overview

SharQ combines activation sparsity with FP4 quantization using an online sparse-dense decomposition. It selects an input-adaptive N:M mask for an outlier-heavy sparse backbone, quantizes that backbone to FP4, then builds a dense residual against the quantized backbone. Sparse and dense FP4 GEMMs process the two paths, with the dense path compensating for mask loss and sparse-path quantization error. The paths share one FP4 weight payload with separate scale views; a fused preparation kernel combines mask generation, residual construction, and layer normalization. The method requires no calibration, retraining, or model-specific tuning. Across four listed language and vision-language models, it reportedly recovers 43-63% of the NVFP4-to-FP16 accuracy gap and generalizes across NVFP4, HiF4, and MXFP4. On an RTX 5090, reported gains are 2.2-2.4x lower latency than FP16 and 1.2-1.4x higher throughput than FP8 for language serving; with SageAttention, it reaches up to 1.58x speedup on Wan2.2 video generation.

### Source excerpt

arXiv:2606.26587v2 Announce Type: replace-cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error. We introduce SharQ, a training-free inference method that bridges activation sparsity and FP4 quantization through an online sparse--dense decomposition. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for both mask-induced activation loss and sparse-path quantization error. The two paths share a single FP4 weight payload with path-specific scale views, and a fused preparation kernel absorbs mask generation, residual construction, and layer normalization into one operator. SharQ requires no calibration data, retraining, or model-specific tuning. Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, SharQ recovers 43--63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, and generalizes across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, SharQ delivers 2.2--2.4$\times$ latency reduction over FP16 and 1.2--1.4$\times$ throughput improvement over FP8 in language model serving, and up to 1.58$\times$ speedup on Wan2.2-T2V-A14B video generation when combined with SageAttention. Our code is available at https://github.com/actypedef/SharQ.

## Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell

DevFeed: [Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell](<https://devfeed.tech/articles/beyond-bitwidth-deployment-constrained-mixed-precision-quantization-for-multilingual-llm-inference-on-nvidia-blackwell-141712.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21974>)

Author: Hyojung Han

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [english](<https://devfeed.tech/tags/english.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gptq](<https://devfeed.tech/tags/gptq.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [int4](<https://devfeed.tech/tags/int4.md>), [int8](<https://devfeed.tech/tags/int8.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [multilingual-llm](<https://devfeed.tech/tags/multilingual-llm.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

On Qwen3-8B running with vLLM on an NVIDIA B200, the tested mixed-precision setup met the study's preregistered quality, memory, and latency gates: 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head used 5.917 GiB against a 5.98 GiB limit, retained 0.904 Korean teacher-correct accuracy against a 0.90 gate, and reached a 0.939 TPOT ratio relative to BF16. On the untouched 5,168-item FINAL-81 set, it improved paired accuracy over NVFP4 W4A4 by 1.4 percentage points (adjusted interval +0.2 to +2.6), less than its 3.4-point selection-set gain. Packed INT2 and INT3 executed, but all seven INT2 grids degenerated during generation. Restoration effects were non-additive, and Korean and reasoning sensitivity rankings differed. Because the comparison changes both activation precision and INT8 protection, the isolated allocation effect is unclear. Final checkpoints were not retained, so bitwise rebuild reproducibility is unverified.

### Source excerpt

arXiv:2611.21974v1 Announce Type: cross Abstract: Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.

## NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention

DevFeed: [NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention](<https://devfeed.tech/articles/nvfp4-mla-unified-fp4-kv-cache-storage-and-attention-computation-for-multi-head-latent-attention-136979.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18972>)

Author: Chao Ni, Qi Zhang, Xiao Wang, Zongfei Jing, Mingyang Hao, Binghan Chen, Kefeng Duan, Yu-Jung Chen, Mengdi Wang, Tijmen Blankevoort, Jun Yang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [mla](<https://devfeed.tech/tags/mla.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

NVFP4-MLA jointly designs FP4 quantization, the attention kernel, and cache system for multi-head latent attention. It uses selective residual compensation on quantization-sensitive channels to preserve accuracy, and two-dimensional quantization blocks so the shared latent cache supports both attention BMM reduction directions without requantization. On DeepSeek-R1 and Kimi-K3, it matches BF16 MLA average accuracy across nine benchmarks while compressing KV cache by 3.20x and 3.05x, respectively. On NVIDIA Rubin GPUs, attention kernels run up to 2.82x and 2.71x faster than FP8 MLA; p90 time to first token in agentic multi-turn serving falls by up to 3.5x and 2.1x. Disaggregated DeepSeek-R1 serving achieves up to 2.30x higher per-GPU throughput.

### Source excerpt

arXiv:2611.18972v1 Announce Type: new Abstract: Long-context reasoning and agentic workloads make key-value (KV) cache capacity and attention computation the dominant costs of serving frontier large language models. We present NVFP4-MLA, the first design to push both KV-cache storage and attention computation of multi-head latent attention (MLA) to FP4. NVFP4-MLA co-designs the quantization algorithm with the attention kernel and the cache system. Selective residual compensation preserves model accuracy by adding NVFP4 residuals only to the quantization-sensitive channels. Two-dimensional quantization blocks make the shared latent cache compatible with the different reduction directions of the two attention batched matrix multiplications (BMMs), removing requantization between them. On DeepSeek-R1 and Kimi-K3, NVFP4-MLA matches BF16 MLA average accuracy across nine benchmarks while compressing the MLA KV cache by 3.20x and 3.05x relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to 2.82x on DeepSeek-R1 and 2.71x on Kimi-K3, and lowers p90 time to first token in agentic multi-turn serving by up to 3.5x and 2.1x, respectively. In disaggregated DeepSeek-R1 serving, it also raises per-GPU throughput by up to 2.30x.

## MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache Compression

DevFeed: [MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache Compression](<https://devfeed.tech/articles/mistiq-microscaling-trellis-quantization-for-extreme-kv-cache-compression-139142.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21216>)

Author: Ofir Gordon, Lior Dikstein, Paul Balanca, Idan Achituve, Douglas Orr, Hai Victor Habi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bits](<https://devfeed.tech/tags/bits.md>), [block](<https://devfeed.tech/tags/block.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llama3](<https://devfeed.tech/tags/llama3.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [trellis](<https://devfeed.tech/tags/trellis.md>)

### AI overview

MiSTiQ combines trellis-coded KV-cache storage and microscaling (MX) formats to target both extreme compression and low-precision attention computation. Trellis quantization with beam search explores large reconstruction spaces without large codebooks, while MX-Hash functions produce reconstruction values directly in the target MX format using scale-conditioned statistics of block-normalized Gaussian inputs. The abstract reports results on Llama3 and Qwen3 at about 1.25 effective bits per element: lower WikiText perplexity and higher GSM8K accuracy than evaluated baselines across several block-scale formats, and the highest LongBench average for both models with NVFP4. The excerpt does not provide numerical scores or implementation details.

### Source excerpt

arXiv:2611.21216v1 Announce Type: new Abstract: Modern large language models (LLMs) increasingly support longer contexts and serve more concurrent requests, increasing both sequence lengths and batch sizes. Both trends increase KV-cache size and memory traffic during inference. KV-cache quantization reduces these costs, with recent approaches using codebook-based vector quantization to achieve extremely low bit widths. However, hardware constraints limit codebook size, and most methods reconstruct the cache in higher-precision floating-point formats for attention, limiting the computational benefits of quantization. To address these limitations, we introduce MiSTiQ, a drop-in replacement for codebook-based vector quantizers that combines trellis-coded storage with microscaling (MX) formats for low-precision computation. We combine trellis quantization with beam search to encode the KV cache over large reconstruction spaces without large codebooks. To support MX computation, we derive MX-Hash functions that generate reconstruction values directly in the target MX element format. These functions use the scale-conditioned marginal distribution of block-normalized Gaussian inputs. On Llama3 and Qwen3, MiSTiQ achieves lower WikiText perplexity and higher GSM8K accuracy than the evaluated baselines across several block-scale formats at approximately 1.25 effective bits per element. With NVFP4, it also achieves the highest LongBench average on both models.

## Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

DevFeed: [Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters](<https://devfeed.tech/articles/benchmarking-llm-inference-on-desktop-scale-nvidia-grace-blackwell-clusters-136990.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20289>)

Author: Kevin Bhimani, Christian Escobar, Daniel Mihovch, Rudolf Eigenmann, Sunita Chandrasekaran

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [DGX Spark](<https://devfeed.tech/topics/dgx-spark.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [desktop](<https://devfeed.tech/tags/desktop.md>), [dgx-spark](<https://devfeed.tech/tags/dgx-spark.md>), [energy](<https://devfeed.tech/tags/energy.md>), [gb10](<https://devfeed.tech/tags/gb10.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grace-blackwell](<https://devfeed.tech/tags/grace-blackwell.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>)

### AI overview

Benchmarks of 17 open-weight models on one or two RDMA-connected NVIDIA DGX Spark GB10 systems find that deployment strategy changes the benefit of adding a second system. With vLLM, tensor parallelism (TP=2) delivered a median 1.59x peak-throughput gain over one Spark, compared with 1.39x for pipeline parallelism (PP=2); TP=2 also reduced unloaded inter-token latency to 0.52-0.73x of one Spark, while PP=2 offered little reduction. For models that fit on one Spark, TP=2 generally suited low concurrency, whereas two replicas achieved higher throughput at high concurrency. One Spark used less GPU energy per output token at matched concurrency. The abstract also reports that NVFP4 reduced unloaded latency more than adding a second Spark at BF16, and suggests collective communication overhead helps explain TP=2's advantage.

### Source excerpt

arXiv:2611.20289v1 Announce Type: new Abstract: Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.

## Gradient-Guided Hardware-Native MIXFP4

DevFeed: [Gradient-Guided Hardware-Native MIXFP4](<https://devfeed.tech/articles/gradient-guided-hardware-native-mixfp4-137341.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18769>)

Author: Jian-Jia Chen, Brian J. Chan, Wen-Hung Lee, Yi-Ting Li, Po-Shen Chen, Kuan-Te Kuo, Hsien-Cheng Huang, Cody Hao Yu, Chi-Chih Chang, Mohamed S. Abdelfattah, Kai-Chiang Wu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [map](<https://devfeed.tech/tags/map.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [native](<https://devfeed.tech/tags/native.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

FlipQuant reduces NVFP4's perplexity gap to BF16 by learning which of two 4-bit formats, E2M1 or E0M3, to use for each matrix-multiply weight tile. It keeps both quantized candidates fixed and learns a binary format map from end-to-end gradients of a bucketed KL loss against a BF16 teacher, re-evaluating candidates after updates. Native kernels use 16 x 64 tiles on SM120 and 256 x 64 tiles on SM100, with median GEMM overheads of 1.9% and 1.7%, respectively. Across six language models and two corpora, the 16 x 64 configuration closed an average 42.5% of NVFP4's PPL gap to BF16, versus 13.3% for FourOverSix.

### Source excerpt

arXiv:2611.18769v1 Announce Type: cross Abstract: Blackwell tensor cores support native inference with 4-bit weights and activations (W4A4), but NVFP4 leaves a perplexity (PPL) gap to BF16. Choosing a format for each 16-element block can improve accuracy, but current matrix multiply instructions cannot execute choices this fine. Applying these rules at hardware-compatible tiles loses most of their benefit. We introduce FlipQuant, which selects E2M1 or E0M3 for each MMA weight tile. It keeps both quantized weight candidates fixed and learns a binary format map from end-to-end gradients of a bucketed KL loss to a BF16 teacher, re-evaluating them after each update. Native kernels execute 16 x 64 tiles on SM120 and 256 x 64 tiles on SM100 with median GEMM overheads of 1.9% and 1.7%, respectively. Across six language models and two corpora, FlipQuant at 16 x 64 closes 42.5% of NVFP4's PPL gap to BF16 on average, compared with 13.3% for FourOverSix. Code is available at https://github.com/brian030128/flipquant.

## CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1

DevFeed: [CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1](<https://devfeed.tech/articles/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-training-v6-1-136727.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-r-training-v6-1>)

Author: CoreWeave

Published: 2026-11-16T19:52:27Z

Content type: news

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>), [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [NVIDIA Spectrum-X Ethernet](<https://devfeed.tech/topics/nvidia-spectrum-x-ethernet.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-provider](<https://devfeed.tech/tags/cloud-provider.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [first](<https://devfeed.tech/tags/first.md>), [full](<https://devfeed.tech/tags/full.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [leading](<https://devfeed.tech/tags/leading.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera](<https://devfeed.tech/tags/nvidia-vera.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [rack](<https://devfeed.tech/tags/rack.md>), [results](<https://devfeed.tech/tags/results.md>), [rubin](<https://devfeed.tech/tags/rubin.md>), [training](<https://devfeed.tech/tags/training.md>), [vera](<https://devfeed.tech/tags/vera.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>)

### AI overview

In MLPerf Training v6.1, CoreWeave reports training DeepSeek-V3 671B to target quality in 17.65 minutes on 256 Vera Rubin GPUs across four racks, versus 48.40 minutes on the same number of GB200 GPUs (2.7x faster, per CoreWeave's calculation). It also reports the round's fastest 72-GPU Llama 3.1 8B submission: 7.16 minutes on one Vera Rubin rack, compared with 13.70 minutes on GB200. The article attributes multi-rack performance to an integrated stack: Spectrum-X RoCE networking and ConnectX-9 SuperNICs, topology-aware placement and orchestration, optimized software, and fleet monitoring and recovery. On GB200, it reports Qwen3.5 397B reinforcement learning in 22.78 minutes across four racks, split evenly between training and generation using NeMo and TensorRT-LLM. These are CoreWeave's descriptions of published results; its speedup and fastest-submission claims are its own analyses and were not verified by MLCommons.

### Source excerpt

See how CoreWeave's full-stack optimizations delivered leading NVIDIA Vera Rubin performance in MLPerf® Training v6.1, from a single 72-GPU rack to multi-rack scale at 256 GPUs

## When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

DevFeed: [When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM](<https://devfeed.tech/articles/when-one-gpu-isn-t-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm-132447.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/>)

Author: Fatima Fall

Published: 2026-11-13T17:00:00Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [DGX Station](<https://devfeed.tech/topics/dgx-station.md>), [gpudirect rdma](<https://devfeed.tech/topics/gpudirect-rdma.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [dgx](<https://devfeed.tech/tags/dgx.md>), [dgx-station](<https://devfeed.tech/tags/dgx-station.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [llm-benchmarking](<https://devfeed.tech/tags/llm-benchmarking.md>), [llms](<https://devfeed.tech/tags/llms.md>), [local-ai](<https://devfeed.tech/tags/local-ai.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [serving](<https://devfeed.tech/tags/serving.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The tutorial runs the 753-billion-parameter GLM-5.3 NVFP4 model across two GB300 DGX Stations because its roughly 464 GB checkpoint exceeds one GPU's 252 GB of memory. vLLM tensor parallelism splits weights and computation across both GPUs; ConnectX-8 RDMA carries rank-to-rank traffic, and Station A exposes a single OpenAI-compatible API. The workflow covers validating the link, downloading and syncing identical checkpoint copies, launching one vLLM rank per station, checking RDMA and API health, and measuring load with AIPerf on 80 SPEED-Bench coding conversations. Prerequisites include substantial disk space on each station, SSH access, Hugging Face access to the gated dataset, and acceptance of the model and dataset terms. Reported throughput peaks at 607.8 tokens/s at concurrency 16, then drops to 385.0 tokens/s at concurrency 32, where mean latency is 453.5 seconds; mean per-user throughput declines as concurrency rises. The excerpt notes that longer prompts or more concurrent users need more KV cache, and suggests reducing context length or request concurrency if memory runs out.

### Source excerpt

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...

## The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models

DevFeed: [The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models](<https://devfeed.tech/articles/the-noise-floor-of-a-quantization-comparison-engine-restart-batch-and-kernel-drift-measured-against-8-bit-and-4-bit-effects-in-two-decision-models-133023.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15671>)

Author: Aleks Blagojevich, Trevor Sunderland

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [effects](<https://devfeed.tech/tags/effects.md>), [engine](<https://devfeed.tech/tags/engine.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb10](<https://devfeed.tech/tags/gb10.md>), [inside](<https://devfeed.tech/tags/inside.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [original](<https://devfeed.tech/tags/original.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

Across two Cloudflare decision models, 4-bit quantization changed more decisions than measured non-quantization sources such as server restarts, engine changes, batch size and kernel selection. On 2,400 public decisions, those other sources shifted 0-0.92%; quantization shifts ranged from 0.88% to 5.79%, depending on model and format. Every measured 4-bit interval exceeded the non-quantization sources, while two of five preregistered predictions failed: FP8 was not within restart variation, and changing engine moved fewer decisions than 8-bit quantization. Aggregate accuracy can mask decision changes: Clef's Q4_K_M build had a net accuracy difference of just six questions across 31,275, although 436 decisions worsened (exploratory). NVFP4 reduced accuracy by 0.95 points for Clef-flash and 0.65 for Clef. The findings are limited to these models, hardware and measured setup; the abstract says code, recipes and per-record outputs are released.

### Source excerpt

arXiv:2611.15671v1 Announce Type: new Abstract: A quantized model is usually judged by comparing it with its BF16 original, but that comparison changes more than the weights. It often changes the serving engine, it always runs on a different server instance, it may change how requests are batched, and on some stacks the library selects a different kernel depending only on which packages import. We measure each of these sources on the same 2,400 public decisions for two of Cloudflare's decision models, Clef-flash (9B) and Clef (27B), on an NVIDIA GB10, and state every quantization effect as the excess over them. Sources that are not quantization move between 0 and 0.92% of decisions. llama.cpp restarts are bit-identical on every record for both models. vLLM restarts flip 0, 5 and 20 decisions at BF16, FP8 and NVFP4 on Clef-flash and 4, 0 and 22 on Clef; a third instance of Clef's quantized builds (exploratory) leaves FP8 at 0 and puts NVFP4 at 4 to 22, depending on which two instances are compared. An engine change at BF16 flips 6 to 16, a batch-size change 8 to 15, and the Gated DeltaNet kernel transformers selects unasked 10 to 14. Quantization inside one engine moves more: for Clef-flash and Clef, 0.88% and 1.04% at Q8_0, 2.12% and 1.62% at FP8, 5.79% and 3.50% at Q4_K_M, and 5.79% and 5.67% at NVFP4. On both models, every 4-bit interval excludes every non-quantization source. Two of our five pre-registered predictions failed, and the same two fail when the tests are rerun on the 27B: FP8 is not inside restart noise, and changing engine moves fewer decisions than 8-bit quantization does. Accuracy hides most of this. Over the full 31,275 public questions, Clef's Q4_K_M build moves accuracy by a net six questions (-0.02 points [-0.20, +0.16]) while 436 of its decisions get worse (exploratory), and NVFP4 costs Clef-flash 0.95 points and Clef 0.65. We release the code, quantization recipes, per-record outputs and the pre-registration.

## SoScale: Second-Order Scale Search for FP4 Quantization

DevFeed: [SoScale: Second-Order Scale Search for FP4 Quantization](<https://devfeed.tech/articles/soscale-second-order-scale-search-for-fp4-quantization-132234.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16426>)

Author: Pengxiang Zhao, Xing Li, Hui-Ling Zhen, Weizhe Lin, Xianzhi Yu, Zhenhua Dong

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [floating-point](<https://devfeed.tech/topics/floating-point.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [gptq](<https://devfeed.tech/tags/gptq.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [order](<https://devfeed.tech/tags/order.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [scale](<https://devfeed.tech/tags/scale.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [search](<https://devfeed.tech/tags/search.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

SoScale aims to reduce accuracy loss in hierarchical FP4 quantization by choosing weight scale codes for their effect on layer outputs, rather than minimizing local element-wise reconstruction error. It searches hardware-valid codes with a Hessian-aware layer reconstruction objective and pairs the search with GPTQ-style error recovery. The abstract reports consistently lower downstream quantization degradation across Qwen3 models using MXFP4, NVFP4, and HiF4; it provides no quantitative results or detail on evaluation conditions.

### Source excerpt

arXiv:2611.16426v1 Announce Type: cross Abstract: Low-bit quantization reduces the memory footprint and inference cost of large language models (LLMs), yet accurate FP4 quantization remains challenging. Practical formats such as MXFP4, NVFP4, and HiF4 pair a 4-bit floating-point payload with block-wise or hierarchical scale codes, making accuracy highly sensitive to how each local distribution is scaled. Default maximum-absolute-value scaling preserves dynamic range, but can spend scarce FP4 levels on rarely occupied outlier regions and leave dense tensor values poorly resolved. Existing scale-search methods minimize local element-wise reconstruction error, optimizing tensor fidelity rather than layer-output fidelity. We propose SoScale, a second-order weight scale-code search method for hierarchical FP4 quantization. SoScale searches hardware-valid scale codes under a Hessian-aware layer reconstruction objective and couples this search with GPTQ-style error recovery, selecting scales by their impact on layer outputs. Across Qwen3 models, SoScale consistently reduces downstream quantization degradation under MXFP4, NVFP4, and HiF4.

## What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

DevFeed: [What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts](<https://devfeed.tech/articles/what-the-loss-curve-misses-fp4-pretraining-slows-the-acquisition-of-rare-facts-125594.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12832>)

Author: Chang Lu, Mengjiao Zhang

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiently](<https://devfeed.tech/tags/efficiently.md>), [either](<https://devfeed.tech/tags/either.md>), [error](<https://devfeed.tech/tags/error.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [formats](<https://devfeed.tech/tags/formats.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>)

### AI overview

In the reported experiments, NVFP4 pretraining required more exposures than BF16 to reach the same rare-fact recall, even when validation loss differed by only 0.5% or 2%; under the primary readout, the exposure cost at matched loss was about 1.2x. The researchers trained transformers on synthetic biographies, alone and mixed with web text. Their controls suggest update noise accounts for much of the persistent cost, while noise in Adam's second-moment estimate has a cost that fades. Raising the learning rate did not recover the lost acquisition efficiency. Gradient statistics from BF16 checkpoints ranked the tested formats and controls with Spearman 0.96. The authors recommend measuring rare-fact acquisition alongside validation loss when evaluating low-precision training.

### Source excerpt

arXiv:2611.12832v1 Announce Type: new Abstract: Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.

## Planners Exploit Deterministic Error: Dithered Low-Bit World Models

DevFeed: [Planners Exploit Deterministic Error: Dithered Low-Bit World Models](<https://devfeed.tech/articles/planners-exploit-deterministic-error-dithered-low-bit-world-models-125446.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11780>)

Author: Yong Hwi Song, Keon Myung Lee

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [failure](<https://devfeed.tech/tags/failure.md>), [faster](<https://devfeed.tech/tags/faster.md>), [planner](<https://devfeed.tech/tags/planner.md>), [planning](<https://devfeed.tech/tags/planning.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [robot](<https://devfeed.tech/tags/robot.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

Round-to-nearest activation quantization can cause sampling-based planners to lose up to 100 success points, even when latent one-step prediction error is no worse than with stochastic rounding. The abstract attributes this to persistent, candidate-shared error that biases successive planning iterations. In TD-MPC2, dither shared across candidates but redrawn each iteration removes at least half the damage in all 19 affected Meta-World task/bit-width cases (median 95%); fixed dither restores drift. W8A4 integer rollouts run 1.9-3.3x faster than FP32 and the full planning step 1.3-1.5x faster on an Ampere GPU, but not faster than BF16 with pre-converted weights; shared dither adds at most 2.3% latency. Block-scaled NVFP4 removes most of the damage and MXFP4 removes some in TD-MPC2.

### Source excerpt

arXiv:2611.11780v1 Announce Type: cross Abstract: Sampling-based planners such as MPPI and CEM query a learned world model thousands of times per control step, so low-bit world models are an obvious way to make model-based control cheaper on a robot. With round-to-nearest activation quantization and no calibration, we find that planning can collapse, by up to 100 points of success, while the world model's latent one-step prediction error stays no larger than that of a stochastic rounding scheme that plans well. At the same bit width, what decides the outcome is not the size of the rounding error but whether it persists across planner iterations and is shared across candidates. Round-to-nearest leaves an error that persists, and an iterating planner drifts toward the candidates that this error favors; in TD-MPC2, holding the same dither fixed within a step or an episode brings the drift back. Dither shared by all candidates and redrawn at every iteration removes most of the damage at the same bit width and without training: at least half of it in all 19 affected Meta-World cells (task $\times$ bit width) of TD-MPC2 (median 95%), in 4 of 5 DINO-WM cells, and in all 8 affected tasks of a 317M-parameter multi-task TD-MPC2. Captured as CUDA graphs on an Ampere GPU, a W8A4 integer TD-MPC2 world model runs its planning rollouts 1.9-3.3$\times$ and the whole planning step 1.3-1.5$\times$ faster than FP32, though not faster than BF16 with pre-converted weights; shared dither adds at most 2.3% latency. In TD-MPC2, block-scaled NVFP4 removes most of the damage and MXFP4 part of it, which ties the failure to coarse grids with a shared scale. Hypotheses, thresholds and analysis code were fixed before the data that test them, and the hypotheses that failed are reported.

## AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

DevFeed: [AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement](<https://devfeed.tech/articles/autoematic-runtime-adaptive-mixed-precision-llm-inference-via-automatic-exponent-mantissa-refinement-125889.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10165>)

Author: Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan, Charbel Sakr, Tijmen Blankevoort, Brucek Khailany, Tushar Krishna

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [1-6](<https://devfeed.tech/tags/1-6.md>), [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [base](<https://devfeed.tech/tags/base.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bits](<https://devfeed.tech/tags/bits.md>), [cache](<https://devfeed.tech/tags/cache.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decode](<https://devfeed.tech/tags/decode.md>), [design](<https://devfeed.tech/tags/design.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [precision](<https://devfeed.tech/tags/precision.md>), [runtime](<https://devfeed.tech/tags/runtime.md>)

### AI overview

AutoEMatic adapts weight, activation, and KV-cache precision at each layer during prefill and decode, using next-token entropy and hidden-state update magnitude to choose among 2-, 4-, and 6-bit AutoFP formats derived from an NVFP4 base. The system is implemented both with GPU kernels on NVFP4 Tensor Cores and on an accelerator with native AutoFP vector MAC execution. Across the reported reasoning and agentic benchmarks, it improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 on most benchmarks, at 4.9 versus 4.5 bits per weight. On GB300 GPUs it retains about 90-98% of NVFP4 decode throughput; the AutoEMatic accelerator reports 1.6-2.3x lower latency and 7-23% lower energy than an NVFP4-only accelerator at equal accuracy.

### Source excerpt

arXiv:2611.10165v1 Announce Type: new Abstract: Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.

## NVIDIA Nemotron 3.5 Super VL is now available in Amazon SageMaker JumpStart

DevFeed: [NVIDIA Nemotron 3.5 Super VL is now available in Amazon SageMaker JumpStart](<https://devfeed.tech/articles/nvidia-nemotron-3-5-super-vl-is-now-available-in-amazon-sagemaker-jumpstart-125097.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/nvidia-nemotron-3-5-super-vl-is-now-available-in-amazon-sagemaker-jumpstart/>)

Author: Venu Kanamatareddy

Published: 2026-11-09T20:01:20Z

Content type: release

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker JumpStart](<https://devfeed.tech/topics/amazon-sagemaker-jumpstart.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [agents](<https://devfeed.tech/tags/agents.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-jumpstart](<https://devfeed.tech/tags/amazon-sagemaker-jumpstart.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [document-intelligence](<https://devfeed.tech/tags/document-intelligence.md>), [image](<https://devfeed.tech/tags/image.md>), [inference](<https://devfeed.tech/tags/inference.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nemotron-3-5](<https://devfeed.tech/tags/nemotron-3-5.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-nemotron-3-5-super-vl](<https://devfeed.tech/tags/nvidia-nemotron-3-5-super-vl.md>), [text](<https://devfeed.tech/tags/text.md>), [variant](<https://devfeed.tech/tags/variant.md>), [video](<https://devfeed.tech/tags/video.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

NVIDIA Nemotron 3.5 Super VL is available through SageMaker JumpStart in NVFP4 and BF16 variants. It accepts text, images, and video and generates text; the 120B-parameter model has 12B active parameters and supports configurable reasoning. NVFP4 is the default option for G7 instances, while BF16 reference weights run on P5en with H200 GPUs and can be downloaded for customization. The reported performance measurements use different GPU systems, so they do not isolate the effect of precision. Deployment requires endpoint permissions and instance quota; new accounts start with zero G7 quota, and the SDK path requires SageMaker Python SDK v3.25 or later. For inference, media must be sent inline as base64 because endpoints have network isolation. PDFs must be rendered to images, and requests accept one image and one video. Context limits vary by serving configuration; video frame sampling is fixed at endpoint launch, and reasoning-enabled video requests need at least 8,192 max tokens. The deployed configuration includes an MTP head but does not enable speculative decoding.

### Source excerpt

NVIDIA Nemotron 3.5 Super VL, a 120B-parameter multimodal vision-language model, is now available in Amazon SageMaker JumpStart. Deploy the NVFP4 or BF16 variant in a few steps, then run text, image, and video inference for document intelligence, computer-use agents, and multimodal agentic workloads.

## ResServe: Accurate and Efficient FP4 LLM Inference with Residue Compensation

DevFeed: [ResServe: Accurate and Efficient FP4 LLM Inference with Residue Compensation](<https://devfeed.tech/articles/resserve-accurate-and-efficient-fp4-llm-inference-with-residue-compensation-123676.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08713>)

Author: Shuo Rong, Ruibo Fan, Jiawen Wen, Xiaowen Chu

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linear](<https://devfeed.tech/tags/linear.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia-b200](<https://devfeed.tech/tags/nvidia-b200.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

ResServe aims to recover some quality lost to FP4 quantization while keeping inference efficient. It adds an FP4-quantized residue to an FP4 activation base, computes the residue contribution using spare hardware capacity in narrow decoding operations, and combines results on-chip. The method covers linear layers and decode attention; selective compensation limits added work on input prompts. On NVIDIA B200, evaluated NVFP4 compensated linear kernels improved numerical precision with latency close to uncompensated kernels for operations with few rows. In evaluated NVFP4 configurations, request latency rose by at most 3.4% over W4A4 and was 22.4-53.3% lower than W4A8. Across five models, code-generation accuracy improved over uncompensated inference in the same format, with mean relative gains of 15.04% for NVFP4 and 53.57% for MXFP4.

### Source excerpt

arXiv:2611.08713v1 Announce Type: new Abstract: Block-scaled 4-bit floating-point (FP4) quantization reduces large language model inference costs but can degrade task quality. Higher-precision computation can mitigate this loss at the expense of some of the efficiency gains from FP4 inference. Narrow matrix computations common in decoding often leave spare hardware capacity for low-cost precision recovery. ResServe combines an FP4 activation base with an FP4-quantized residue representing information lost during quantization. It uses this spare capacity to compute the residue's contribution and combines results on-chip, avoiding extra result transfers. We apply this principle to linear layers and decode attention, while selective compensation limits added work when processing input prompts. On NVIDIA B200, compensated linear kernels using the NVFP4 format improve numerical precision while retaining latency close to uncompensated execution for the evaluated operations with few rows. In the evaluated NVFP4 inference configurations, ResServe adds at most 3.4% request-latency overhead over W4A4 and reduces request latency by 22.4-53.3% relative to W4A8. Across five models, code-generation accuracy improves over same-format uncompensated inference, with mean relative gains of 15.04% for NVFP4 and 53.57% for MXFP4.

## BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

DevFeed: [BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design](<https://devfeed.tech/articles/bitdecoding-fp4-native-fp4-kv-cache-decoding-via-algorithm-system-co-design-124732.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09825>)

Author: Dayou Du, Woon Yee Ng, Guangming Sheng, Nick Lau, Jianyi Cheng, Luo Mai

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Electrical engineering and computer science (EECS)](<https://devfeed.tech/topics/electrical-engineering-and-computer-science-eecs.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [b300](<https://devfeed.tech/tags/b300.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [cache](<https://devfeed.tech/tags/cache.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [gqa](<https://devfeed.tech/tags/gqa.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [mla](<https://devfeed.tech/tags/mla.md>), [moe](<https://devfeed.tech/tags/moe.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia-b300](<https://devfeed.tech/tags/nvidia-b300.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [rl](<https://devfeed.tech/tags/rl.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

BitDecoding-FP4 combines quantization methods with system-level decoding optimizations to make FP4 KV-cache attention more accurate and efficient. It splits queries across two FP4 tensors to reduce reported query-logit error from 60% to 1%, scales probabilities to avoid long-context underflow, and uses smoothing, Hadamard rotation, and scale search to suppress key outliers. For decoding, it fuses probability quantization with transposed softmax and uses stacked query matrix multiplications on paged KV caches. Across dense and MoE models using GQA and MLA, the authors report accuracy within 0.5 points of BF16 KV with NVFP4 and within 1 point with MXFP4, and similar performance in RL training rollouts. On NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is reported at 2.4-3.0x that of BF16 KV.

### Source excerpt

arXiv:2611.09825v1 Announce Type: new Abstract: The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a fundamental gap remains between algorithmic quantization and system-aware acceleration: existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, whereas accuracy-oriented 4-bit algorithms rely on vector codebooks whose lookup overhead degrades decoding throughput. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. We mitigate three error sources: Dual-Residual Query Quantization splits queries into two FP4 tensors, cutting query logit error from 60% to 1%; Underflow-Aware Probability Scaling prevents E4M3 scales from rounding long-context probabilities to zero; and Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search to suppress key outliers under token-wise scaling. At the system level, we remove the softmax bottleneck via transposed softmax with fused probability quantization and stacked query MMA on paged KV caches. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0$\times$ higher peak serving throughput than BF16 KV. The implementation is available at https://github.com/OpenBitSys/BitDecoding-FP4.

## Speed Up Image Generation with FLUX 3 Image Klein on NVIDIA Blackwell GPUs

DevFeed: [Speed Up Image Generation with FLUX 3 Image Klein on NVIDIA Blackwell GPUs](<https://devfeed.tech/articles/speed-up-image-generation-with-flux-3-image-klein-on-nvidia-blackwell-gpus-121823.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/speed-up-image-generation-with-flux-3-image-klein-on-nvidia-blackwell-gpus/>)

Author: Olivia Stoner

Published: 2026-11-06T22:20:20Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [flux](<https://devfeed.tech/topics/flux.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [ai-image-generator](<https://devfeed.tech/topics/ai-image-generator.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [RTX PRO](<https://devfeed.tech/topics/rtx-pro.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [content-creation-rendering](<https://devfeed.tech/tags/content-creation-rendering.md>), [cuda-graphs](<https://devfeed.tech/tags/cuda-graphs.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [flux-3-image](<https://devfeed.tech/tags/flux-3-image.md>), [image](<https://devfeed.tech/tags/image.md>), [image-generation](<https://devfeed.tech/tags/image-generation.md>), [latency](<https://devfeed.tech/tags/latency.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>)

### AI overview

TensorRT LLM VisualGen combines compilation, NVFP4, fused GEMM kernels, attention optimizations, and caching to accelerate FLUX 3 Image Klein on NVIDIA Blackwell GPUs. In 2048x2048 tests on one B200, the combined setup reduced Base latency from 22.36 to 6.46 seconds for text-to-image and from 54.64 to 12.28 seconds for image-to-image, up to 3.46x and 4.45x faster than uncompiled BF16. NVFP4 alone was up to 1.62x faster for text-to-image and 1.58x for image-to-image. Further options trade output similarity against speed: VC-Attention FP8 improved generation speed by 1.16-1.18x with reported LPIPS differences, while Sol-Attention and Cache-DiT reduced denoising work or reused intermediate outputs. Cache-DiT nearly halved Base text-to-image latency at tested resolutions, with smaller gains for the 16-step distilled model. Eight-GPU execution and CUDA Graphs provide additional gains, though non-denoising work limits some end-to-end scaling. The tutorial supplies YAML settings and commands for offline generation and serving; reported results focus on B200 and RTX PRO 6000 Blackwell GPUs.

### Source excerpt

When deploying diffusion models in design tools, image editors, and visual production workflows, developers must account for inference latency. Generating an...

## Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory

DevFeed: [Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory](<https://devfeed.tech/articles/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai-factory-119477.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai-factory/>)

Author: Shara Tibken

Published: 2026-11-05T16:00:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [AI Factory](<https://devfeed.tech/topics/ai-factory.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai-factory](<https://devfeed.tech/tags/ai-factory.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [better](<https://devfeed.tech/tags/better.md>), [bits](<https://devfeed.tech/tags/bits.md>), [block](<https://devfeed.tech/tags/block.md>), [bluefield-dpu](<https://devfeed.tech/tags/bluefield-dpu.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cached-context](<https://devfeed.tech/tags/cached-context.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [doca](<https://devfeed.tech/tags/doca.md>), [dynamo](<https://devfeed.tech/tags/dynamo.md>), [ethernet](<https://devfeed.tech/tags/ethernet.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gb200](<https://devfeed.tech/tags/gb200.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [gb300-nvl72](<https://devfeed.tech/tags/gb300-nvl72.md>), [gpudirect](<https://devfeed.tech/tags/gpudirect.md>), [h100](<https://devfeed.tech/tags/h100.md>), [hbm](<https://devfeed.tech/tags/hbm.md>), [high-bandwidth-memory](<https://devfeed.tech/tags/high-bandwidth-memory.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>), [rubin-gpu](<https://devfeed.tech/tags/rubin-gpu.md>), [spectrum-x](<https://devfeed.tech/tags/spectrum-x.md>), [tensorrt-llm](<https://devfeed.tech/tags/tensorrt-llm.md>), [vera-cpu](<https://devfeed.tech/tags/vera-cpu.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>), [vera-rubin-nvl72](<https://devfeed.tech/tags/vera-rubin-nvl72.md>)

### AI overview

The excerpt describes five ways NVIDIA's AI factory design aims to increase useful work per memory byte: NVFP4 shrinks model weights and KV cache; TensorRT-LLM Wide-EP spreads MoE experts across GPUs to free per-GPU memory and improve weight reuse; Dynamo routes requests to cached prefixes and manages KV cache across GPU, CPU, and storage; Vera CPU memory provides a place to stage context during tool-call gaps; and CMX offers pod-level shared flash for context reuse across nodes. The article reports benchmark-specific gains, including up to 4x throughput over an FP8 baseline for NVFP4 and up to 30x throughput per megawatt for Vera Rubin NVL72 versus GB300 NVL72 at the same 288 GB per GPU. It also notes that KV offloading helps when retrieval is faster than recomputation; storage performance examples are vendor validation tests.

### Source excerpt

In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an...

## NVIDIA's DeepSeek-V4.1-Flash NVFP4 Model: Everything You Need to Know

DevFeed: [NVIDIA's DeepSeek-V4.1-Flash NVFP4 Model: Everything You Need to Know](<https://devfeed.tech/articles/nvidia-s-deepseek-v4-1-flash-nvfp4-model-everything-you-need-to-know-116243.md>)

Original publisher: [Read original article](<https://hackernoon.com/nvidias-deepseek-v41-flash-nvfp4-model-everything-you-need-to-know?source=rss>)

Author: aimodels44

Published: 2026-11-04T08:00:21Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-tasks](<https://devfeed.tech/tags/agent-tasks.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-model](<https://devfeed.tech/tags/ai-model.md>), [applications](<https://devfeed.tech/tags/applications.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [customer-success](<https://devfeed.tech/tags/customer-success.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4](<https://devfeed.tech/tags/deepseek-v4.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvfp4-quantization](<https://devfeed.tech/tags/nvfp4-quantization.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [programming](<https://devfeed.tech/tags/programming.md>)

### AI overview

DeepSeek-V4.1-Flash-1017-NVFP4 is a quantized multimodal MoE model for text and RGB image input, with a stated context window of up to 1 million tokens. It has 552B backbone parameters (8B active during prefill and 16B during decode) plus 196B Engram conditional memory. The documented deployment uses vLLM with MXFP4 Engram support on NVIDIA Blackwell; the example runs on four GB300 GPUs. Stock vLLM images cannot load its Engram tables, SGLang is unvalidated, and speculative decoding was not tested. Reported scores versus the MXFP4 source baseline are mixed: this checkpoint scores higher on GPQA Diamond, IFBench and MMMU-Pro, and lower on AA-LCR, SciCode and Terminal-Bench 2.1, so the figures do not establish an overall winner. The stated license is MIT; training data details are undisclosed.

### Source excerpt

Discover NVIDIA's quantized DeepSeek-V4.1-Flash NVFP4 model, featuring 552B parameters, multimodal inputs, and a 1M-token context window.

## FourSight: Stable FP4 RL Post-Training via Sequence Filtering and Sparse Channel Compensation

DevFeed: [FourSight: Stable FP4 RL Post-Training via Sequence Filtering and Sparse Channel Compensation](<https://devfeed.tech/articles/foursight-stable-fp4-rl-post-training-via-sequence-filtering-and-sparse-channel-compensation-116992.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04887>)

Author: Yining Wei, Tianyuan Wu, Gu Gong, Yuechen Tao, Yinghao Yu, Xiaowen Chu, Wei Wang

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [activation](<https://devfeed.tech/tags/activation.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [channel](<https://devfeed.tech/tags/channel.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [engine](<https://devfeed.tech/tags/engine.md>), [engines](<https://devfeed.tech/tags/engines.md>), [error](<https://devfeed.tech/tags/error.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [post](<https://devfeed.tech/tags/post.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [stable](<https://devfeed.tech/tags/stable.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

FourSight aims to stabilize NVFP4 W4A4 reinforcement-learning post-training by addressing two concentrated error sources: it filters drifted trajectories from policy updates to reduce training-inference mismatch, and keeps high-variance activation channels in BF16 while using NVFP4 elsewhere. A fused SpliceGEMM kernel with specialized prefill and decode datapaths is designed to limit overhead. The abstract reports near-BF16 accuracy, 1.76-2.60x better step-to-accuracy, and step time within 2.3% of FP4 through models up to 120B parameters. These claims are from the supplied abstract.

### Source excerpt

arXiv:2611.04887v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training increasingly targets low-bit deployment, with NVFP4 weight-and activation quantization (W4A4) becoming the state-of-the-practice format. However, integrating NVFP4 into RL degrades rollout quality and amplifies cross-engine training-inference mismatch (TIM), destabilizing training. We show that these failures are concentrated rather than uniform - severely mismatched tail tokens concentrate in a few outlier sequences, while policy weights remain largely intact and quantization error concentrates in a few high-variance activation channels. Guided by this, we present FourSight, an algorithm-system co-design with two targeted solutions: TIM-aware sequence filtering that excludes drifted trajectories from policy updates, and sparse precision compensation that keeps only top-variance channels in BF16 while retaining native NVFP4 elsewhere. We further design a fused SpliceGEMM kernel with specialized prefill and decode datapaths to minimize overhead. FourSight sustains near-BF16 accuracy and strong FP4 deployment quality across models and engines, while improving step-to-accuracy by 1.76-2.60x and staying within 2.3% of FP4 step time up to 120B-scale models.

## TriTile: Three-Tier Attention for Video Diffusion Transformers

DevFeed: [TriTile: Three-Tier Attention for Video Diffusion Transformers](<https://devfeed.tech/articles/tritile-three-tier-attention-for-video-diffusion-transformers-116793.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.35706>)

Author: Yeonjae Kim, Omin Kwon, Seong Hoon Seo, Jae W. Lee

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [error](<https://devfeed.tech/tags/error.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [generation](<https://devfeed.tech/tags/generation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [ltx](<https://devfeed.tech/tags/ltx.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [router](<https://devfeed.tech/tags/router.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

TriTile reduces video diffusion transformer attention cost by routing each tile to one of three training-free approximation tiers: skip it using block means, compute it in NVFP4, or refine it with NVFP4 residual correction. A lightweight router estimates each tier's error from block statistics and spends extra compute when it is predicted to reduce error; a single compute-price knob controls the speed-quality trade-off. A fused kernel runs the tiers in one pass on FP4 tensor cores. Against dense FlashAttention, the abstract reports generation speedups of 1.38-1.83x on Wan2.2 at 23.0-27.6 dB PSNR, 2.28-2.32x on HunyuanVideo at 34.5-36.0 dB, and 1.84x on LTX-2.5 at 34.1 dB. It claims Pareto dominance over every baseline in speed and PSNR in all reported settings.

### Source excerpt

arXiv:2610.35706v2 Announce Type: replace Abstract: Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present TriTile, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and spends extra compute on a tile only where it removes enough error; a single knob, a price on compute, controls the whole speed-quality trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores. Over dense FlashAttention, TriTile speeds up generation by 1.38-1.83x on Wan2.2 at 23.0-27.6 dB PSNR, 2.28-2.32x on HunyuanVideo at 34.5-36.0 dB, and 1.84x on LTX-2.5 at 34.1 dB, and in every setting it Pareto-dominates every baseline in speed and PSNR.

[Next page](<https://devfeed.tech/topics/nvfp4.md?cursor=WyIyMDI2LTExLTA0VDA1OjAwOjAwKzAwOjAwIiwgIjE0YzJiZjQ5LWZkYjQtNDAwOC1hNjg0LThiMTA2N2ZhNjM3ZCJd>)