# ResServe: Accurate and Efficient FP4 LLM Inference with Residue Compensation

DevFeed: [ResServe: Accurate and Efficient FP4 LLM Inference with Residue Compensation](<https://devfeed.tech/articles/resserve-accurate-and-efficient-fp4-llm-inference-with-residue-compensation-123676.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08713>)

Author: Shuo Rong, Ruibo Fan, Jiawen Wen, Xiaowen Chu

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linear](<https://devfeed.tech/tags/linear.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia-b200](<https://devfeed.tech/tags/nvidia-b200.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

## AI overview

ResServe aims to recover some quality lost to FP4 quantization while keeping inference efficient. It adds an FP4-quantized residue to an FP4 activation base, computes the residue contribution using spare hardware capacity in narrow decoding operations, and combines results on-chip. The method covers linear layers and decode attention; selective compensation limits added work on input prompts. On NVIDIA B200, evaluated NVFP4 compensated linear kernels improved numerical precision with latency close to uncompensated kernels for operations with few rows. In evaluated NVFP4 configurations, request latency rose by at most 3.4% over W4A4 and was 22.4-53.3% lower than W4A8. Across five models, code-generation accuracy improved over uncompensated inference in the same format, with mean relative gains of 15.04% for NVFP4 and 53.57% for MXFP4.

## Source excerpt

arXiv:2611.08713v1 Announce Type: new Abstract: Block-scaled 4-bit floating-point (FP4) quantization reduces large language model inference costs but can degrade task quality. Higher-precision computation can mitigate this loss at the expense of some of the efficiency gains from FP4 inference. Narrow matrix computations common in decoding often leave spare hardware capacity for low-cost precision recovery. ResServe combines an FP4 activation base with an FP4-quantized residue representing information lost during quantization. It uses this spare capacity to compute the residue's contribution and combines results on-chip, avoiding extra result transfers. We apply this principle to linear layers and decode attention, while selective compensation limits added work when processing input prompts. On NVIDIA B200, compensated linear kernels using the NVFP4 format improve numerical precision while retaining latency close to uncompensated execution for the evaluated operations with few rows. In the evaluated NVFP4 inference configurations, ResServe adds at most 3.4% request-latency overhead over W4A4 and reduces request latency by 22.4-53.3% relative to W4A8. Across five models, code-generation accuracy improves over same-format uncompensated inference, with mean relative gains of 15.04% for NVFP4 and 53.57% for MXFP4.