# vLLM Blog

Technical articles, release announcements, model guides, and community updates from the vLLM project.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM

DevFeed: [Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM](<https://devfeed.tech/articles/keeping-agents-fast-high-interactivity-decode-with-flydsl-monokernels-in-vllm-140940.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-18-flydsl-monokernels>)

Author: Felix Li, Lingpeng Jin, Hattie Wu, Carlus Huang, Fangzhou Ai, Zaidy Ali, Amanzhol Salykov, Hyukjoon Lee, Ziming Huang, Jun Chen, Yifeng Huang, Bin Ding, Xiaobing Zhang, Yi Gan, Vinayak Gokhale, Hongxi

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [rocm](<https://devfeed.tech/topics/rocm.md>), [mlir](<https://devfeed.tech/topics/mlir.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [decode](<https://devfeed.tech/tags/decode.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [handoffs](<https://devfeed.tech/tags/handoffs.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [integration](<https://devfeed.tech/tags/integration.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model](<https://devfeed.tech/tags/model.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

FlyDSL MonoKernels aim to reduce low-concurrency decode latency in vLLM on AMD Instinct GPUs by combining dependent stages into a few persistent kernel launches. Authors choose model-specific boundaries and schedules; FlyDSL compiles them, while vLLM selects the opt-in path for supported steps. Thread blocks can reuse work across stages, prefetch weights, and hand off ready tiles through tagged mailboxes, though dependencies and memory traffic remain. Reported evaluations show gains across several models, with results varying by model, GPU, concurrency, and benchmark; some comparisons use historical baselines. The paths preserve vLLM's scheduler and KV cache, fall back to stock layers for unsupported workloads, and require accuracy checks per configuration. Limits include reduced gains at higher concurrency, maximum step widths, and extra weight copies on MI300X/MI325X.

### Source excerpt

How FlyDSL MonoKernels keep agents responsive with high-interactivity decode in vLLM on AMD Instinct GPUs, using model-specific fusion, data-ready handoffs, and specialized-decode integration.

## DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/s

DevFeed: [DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/s](<https://devfeed.tech/articles/deepseek-v4-1-flash-on-vera-rubin-nvl72-58x-throughput-vs-gb300-nvl72-at-387-tokens-s-132530.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-12-deepseek-v41-flash-vera-rubin>)

Author: Inferact and the vLLM Team

Published: 2026-11-12T00:00:00Z

Content type: news

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [gb300-nvl72](<https://devfeed.tech/tags/gb300-nvl72.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [large-scale-serving](<https://devfeed.tech/tags/large-scale-serving.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera](<https://devfeed.tech/tags/nvidia-vera.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rack](<https://devfeed.tech/tags/rack.md>), [rubin](<https://devfeed.tech/tags/rubin.md>), [semianalysis](<https://devfeed.tech/tags/semianalysis.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [v4-1-flash](<https://devfeed.tech/tags/v4-1-flash.md>), [vera](<https://devfeed.tech/tags/vera.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>), [vera-rubin-nvl72](<https://devfeed.tech/tags/vera-rubin-nvl72.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

On a Vera Rubin NVL72 rack, vLLM reports 1.7M tokens/s per GPU on DeepSeek-V4.1-Flash at 387 tokens/s per user, described as 58x GB300 NVL72 at the same interactivity. The gains come from kernels adapted to Rubin's separate memory domains and wider FP4-capable MMA instructions, a specialized TP8 decode path, and shared rack memory pools. The excerpt reports roughly 28% faster DEP2 prefill steps and 25% faster DEP32 decode steps; TP8 reaches 1,810 tokens/s at batch size 1 in the TL;DR. Mooncake pools KV data across DRAM and spare HBM, while Engram tables are split across GPU HBM and accessed over NVLink. At concurrency 4,800, pools plus workload-aware routing on three DEP2 instances serve 68% more requests, reduce TTFT P50 from 29.8 to 4.2 seconds, and raise prefix hit rate from 91.8% to 96.1%. The HBM pool is described as experimental.

### Source excerpt

On one NVIDIA Vera Rubin NVL72 rack, vLLM serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 1.7M tokens/s per GPU at a P90 of 387 tokens/s per user, 58x GB300 NVL72 at the same interactivity.

## Reinforcement Learning: Optimizing 0.6B Dense Models on TPU v7 via vLLM

DevFeed: [Reinforcement Learning: Optimizing 0.6B Dense Models on TPU v7 via vLLM](<https://devfeed.tech/articles/reinforcement-learning-optimizing-0-6b-dense-models-on-tpu-v7-via-vllm-138119.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-12-rl-small-dense-tpu-v7>)

Author: Loki Chen, Yang Song, Hongye Xie, Ming Lei, Greg Rehm, Chongyang Bai, Dre Olgiati, Bob Kamma, Tej Choudhary, Hang Cui, Imed Zitouni, Hamed Firooz (Meta ASA); Mingfu Liang, Kavosh Asadi, Yufei Li, Fran

Published: 2026-11-12T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [8k](<https://devfeed.tech/tags/8k.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-tpu](<https://devfeed.tech/tags/cloud-tpu.md>), [designing](<https://devfeed.tech/tags/designing.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [minutes](<https://devfeed.tech/tags/minutes.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [models](<https://devfeed.tech/tags/models.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [tpu](<https://devfeed.tech/tags/tpu.md>), [true](<https://devfeed.tech/tags/true.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

On a 64-chip Google Cloud TPU v7 slice, the team cut a Qwen3-0.6B GRPO rollout-to-update cycle from roughly 100 minutes to 56 seconds while retaining the 8,192-token response cap. The work shows that small-model RL can be limited by dispatch, synchronization, cache handling, and kernel geometry as well as compute. Key changes fused decode steps into an XLA device loop, tuned Pallas RPA v3 fetch and compute blocks for long contexts, adjusted EOS-check cadence, used continuous batching and vocab-sharded sampling, parallelized reward scoring, and improved cache persistence, weight sync, and training attention. Decode throughput reached about 183K tokens/s averaged over the full rollout; the final cycle had a 57.9-second median, with 85% of time in rollout. Results use a public math-prompt proxy and a 4,096-token prompt cap for the final runs; earlier milestones used a 16,384-token cap, so the full ladder does not hold prompt capacity constant. Several changes in the 299-to-88-second stage were not isolated. Quantization and shared-prefix attention showed little or no net gain against production-scale kernels. The published recipe and flags are reported to reproduce the result.

### Source excerpt

Co-designing RL for a Qwen3-0.6B dense policy on Google Cloud TPU v7 across vLLM, Tunix, MaxText, and Pallas, with a 107x end-to-end speedup from ~100 minutes to ~56 seconds per true-8K RL cycle.

## H3-Image Day-0 Support: 4K in 0.87s with Vera Rubin

DevFeed: [H3-Image Day-0 Support: 4K in 0.87s with Vera Rubin](<https://devfeed.tech/articles/h3-image-day-0-support-4k-in-0-87s-with-vera-rubin-130634.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-12-minimax-h3-image-rubin>)

Author: vLLM-Omni team

Published: 2026-11-12T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Image](<https://devfeed.tech/topics/image.md>), [ai-image-generator](<https://devfeed.tech/topics/ai-image-generator.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [4k](<https://devfeed.tech/tags/4k.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [api](<https://devfeed.tech/tags/api.md>), [api-pricing](<https://devfeed.tech/tags/api-pricing.md>), [arc](<https://devfeed.tech/tags/arc.md>), [ascend](<https://devfeed.tech/tags/ascend.md>), [atlas](<https://devfeed.tech/tags/atlas.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [data-center](<https://devfeed.tech/tags/data-center.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [image](<https://devfeed.tech/tags/image.md>), [image-generation](<https://devfeed.tech/tags/image-generation.md>), [images](<https://devfeed.tech/tags/images.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [revenue](<https://devfeed.tech/tags/revenue.md>), [rubin](<https://devfeed.tech/tags/rubin.md>), [support](<https://devfeed.tech/tags/support.md>), [vera](<https://devfeed.tech/tags/vera.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [vllm-omni](<https://devfeed.tech/tags/vllm-omni.md>)

### AI overview

H3-Image served native 4096x4096 images on four NVIDIA Vera Rubin GPUs at a median 0.866 seconds in a short-prompt SP4 benchmark; a separate four-replica setup reached 6,115 images/hour with 4.411-second median latency. These are distinct operating points. The model handles text-to-image and editing in six sampling steps, supports up to 16 reference images, and uses MXFP8 for eligible DiT layers on Rubin while the encoder and VAE retain native precision. VAE tile distribution with CUDA Graph replay and streams reduced decoding from 2.635 to 0.213 seconds in a four-GB300 control. Smaller GPUs can run through layerwise CPU offload, trading transfer time for memory. Other hardware recipes cover AMD, Intel, Huawei and Moore Threads, with separate configurations and workloads, so their results are not directly matched to the NVIDIA comparisons. Revenue projections use an assumed API price and full or 75% billable utilization; they exclude operating costs, and daily capacity extrapolates measured throughput rather than a 24-hour test.

### Source excerpt

H3-Image on Vera Rubin: 0.87-second 4K generation or 6,115 images/hour on four GPUs, with projected API revenue and support across consumer and data-center hardware.

## Inkling 2 Small on vLLM: Day-0 Support

DevFeed: [Inkling 2 Small on vLLM: Day-0 Support](<https://devfeed.tech/articles/inkling-2-small-on-vllm-day-0-support-132531.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-12-inkling2-small>)

Author: Inferact and Thinking Machines

Published: 2026-11-12T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [attention](<https://devfeed.tech/tags/attention.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decode](<https://devfeed.tech/tags/decode.md>), [dspark](<https://devfeed.tech/tags/dspark.md>), [inkling-2-small](<https://devfeed.tech/tags/inkling-2-small.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [model-support](<https://devfeed.tech/tags/model-support.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM supports Thinking Machines' multimodal Inkling 2 Small from day one: a 594B-parameter MoE with 17B active parameters and up to 1M-token context. Its integration adds fused all-reduce and memory-bound operations, fused expert activation quantization, and sequence parallelism for prefill; the latter cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8 while freeing 1.3-1.9 GiB per GPU. A warp-specialized Gluon relative-attention kernel reduces batch-1 decode at 100K context from 15.1 ms to 4.8 ms; prefill retains the FlashAttention-4 path. The model includes a DSpark RNN speculator that drafts seven tokens per step. Reported single-user TP4 decode reaches about 630 tokens/s with acceptance length 3.5. vLLM also supports LoRA and RL via SkyRL. Accuracy was checked on NIAH up to 1M tokens, MMMU-Pro, and HLE; reported scores matched runs without speculative decoding.

### Source excerpt

vLLM supports Thinking Machines' Inkling 2 Small from day 0: attention residuals, a new decode kernel for relative-bias attention, all-reduce fused with the ops around it, and DSpark speculative decoding.

## Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD

DevFeed: [Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD](<https://devfeed.tech/articles/encoder-disaggregation-in-vllm-scaling-multimodal-serving-with-epd-129261.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-11-encoder-disaggregation-epd>)

Author: Inferact, InferMatrix, and IBM

Published: 2026-11-11T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Deployment & Scaling](<https://devfeed.tech/topics/deployment-scaling.md>), [KServe](<https://devfeed.tech/topics/kserve.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cross-encoder](<https://devfeed.tech/tags/cross-encoder.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [execution](<https://devfeed.tech/tags/execution.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [remote](<https://devfeed.tech/tags/remote.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [serving](<https://devfeed.tech/tags/serving.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

EPD separates multimodal encoding from prefill and decode so operators can scale those stages independently and move media processing off language-model GPUs. Recent vLLM work adds metadata-only handoff for supported image, video, and audio paths, a dedicated encoder runner, NIXL and Mooncake embedding transfer, cross-encoder caching, and routing integrations. In one audio evaluation, metadata-only forwarding raised throughput from 62.21 to 73.13 requests/s and reduced mean latency from 980 to 853 ms, with accuracy shifting from 75.90% to 75.65%; the accuracy difference remains under investigation. DeepSeek V4.1 Flash tests showed higher per-GPU throughput and lower P90 TTFT at concurrency 64-128, but lower throughput at 16 and near break-even at 32. Shared cache reuse eliminated repeated encodings in a controlled reassignment test, described as a best case; real savings depend on reuse rates and Store costs. Support and results vary by model, modality, backend, and deployment.

### Source excerpt

Recent advances in vLLM encoder disaggregation: metadata-only multimodal requests, dedicated encoder execution, remote embedding transfer, cross-encoder caching, and lower-overhead EPD routing.

## Migrating vLLM to the LibTorch Stable ABI

DevFeed: [Migrating vLLM to the LibTorch Stable ABI](<https://devfeed.tech/articles/migrating-vllm-to-the-libtorch-stable-abi-122090.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-06-stable-abi-cuda>)

Author: vLLM Team, Red Hat PyTorch Team, and Meta PyTorch Team

Published: 2026-11-06T00:00:00Z

Content type: tutorial

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [CMake](<https://devfeed.tech/topics/cmake.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abi](<https://devfeed.tech/tags/abi.md>), [adapter](<https://devfeed.tech/tags/adapter.md>), [added](<https://devfeed.tech/tags/added.md>), [adopting](<https://devfeed.tech/tags/adopting.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [also](<https://devfeed.tech/tags/also.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [build](<https://devfeed.tech/tags/build.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [stable](<https://devfeed.tech/tags/stable.md>), [upgrades](<https://devfeed.tech/tags/upgrades.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

As of vLLM 0.33.0, its CUDA wheels use the LibTorch Stable ABI, allowing the same wheel to work across compatible PyTorch releases without rebuilding vLLM. The migration moved C++ calls to stable wrappers, a C shim, or header-only utilities; shifted unsupported meta functions and autograd functions to Python; registered operators through `STABLE_TORCH_LIBRARY`; and used `torch-abi-audit` to detect unstable symbols. Compiled dependencies also had to migrate for the whole wheel to qualify: the excerpt's audit changes from Torch ABI UNSTABLE before dependency migration to STABLE afterward, with a reported minimum PyTorch version of 2.13.0. Compatibility still depends on Python APIs, CUDA and GPU requirements, and third-party dependencies. The stable API is a subset of LibTorch, and shim conversions or dispatcher calls add host overhead that may matter for repeated fast kernels in eager mode; the migration reports no noticeable runtime regression. CPU and ROCm builds remain future work.

### Source excerpt

How vLLM migrated its CUDA build and compiled dependencies to the LibTorch Stable ABI, enabling compatible PyTorch upgrades without rebuilding vLLM.

## Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

DevFeed: [Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1](<https://devfeed.tech/articles/zero-mismatch-simple-async-rl-reinforce-zero-deepseek-v4-1-and-kimi-k3-1-122091.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-06-zero-mismatch-async-rl>)

Author: Ao Shen (Inferact), Jian Hu (NVIDIA Research), Yan Bai (NVIDIA DevTech), Shekhar Pandey (AMD), and the REINFORCE Zero authors

Published: 2026-11-06T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [async](<https://devfeed.tech/topics/async.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [7-4](<https://devfeed.tech/tags/7-4.md>), [8k](<https://devfeed.tech/tags/8k.md>), [about](<https://devfeed.tech/tags/about.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [align](<https://devfeed.tech/tags/align.md>), [amd](<https://devfeed.tech/tags/amd.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [flash](<https://devfeed.tech/tags/flash.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [lag](<https://devfeed.tech/tags/lag.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [training](<https://devfeed.tech/tags/training.md>), [trains](<https://devfeed.tech/tags/trains.md>), [v4-1-flash](<https://devfeed.tech/tags/v4-1-flash.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

REINFORCE Zero argues that asynchronous RL can train stably on stale samples when rollout and learner kernels are aligned, leaving unclipped token level importance sampling to correct weight lag. The paper reports stable training through 20,000 updates and learning at policy lag 64; in one comparison, importance weight deviation peaked at 1.50 times its median with aligned kernels versus 442.54 times with standard kernels. Without alignment, training drifted and a sequence gate rejected 24.48% of sequences. Alignment has a speed cost: one OSWorld setup took 6.8% longer per step than standard kernel IS with a gate. The excerpt also reports zero-mismatch deployments for DeepSeek-V4.1-Flash and Kimi K3.1, including AMD MI355X runs, while noting ongoing performance work and that long asynchronous runs on the flagship models remain future work.

### Source excerpt

Asynchronous RL trains on stale weights. Align rollout and training kernels bit for bit, and unclipped importance sampling corrects that lag. Plus zero-mismatch RL for DeepSeek-V4.1-Flash, Kimi K3.1 and AMD MI355X.

## VeRL-Omni v0.3.0: Async Training and Broader Multimodal Post-Training

DevFeed: [VeRL-Omni v0.3.0: Async Training and Broader Multimodal Post-Training](<https://devfeed.tech/articles/verl-omni-v0-3-0-async-training-and-broader-multimodal-post-training-126407.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-06-verl-omni-v0-3-0>)

Author: VeRL-Omni Team

Published: 2026-11-06T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [generative-ai-tools](<https://devfeed.tech/topics/generative-ai-tools.md>), [trl](<https://devfeed.tech/topics/trl.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [async](<https://devfeed.tech/tags/async.md>), [audio](<https://devfeed.tech/tags/audio.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [generation](<https://devfeed.tech/tags/generation.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rl](<https://devfeed.tech/tags/rl.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [trainer](<https://devfeed.tech/tags/trainer.md>), [training](<https://devfeed.tech/tags/training.md>), [v1](<https://devfeed.tech/tags/v1.md>), [video](<https://devfeed.tech/tags/video.md>), [video-generation](<https://devfeed.tech/tags/video-generation.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

VeRL-Omni v0.3.0 expands asynchronous multimodal RL and makes its V1 trainer the default. It adds separate-async execution paths for Omni and diffusion workloads, replay-buffer coordination, and new audio-video and multimodal training recipes. The MiniMax H3 DiffusionNFT case study optimizes selected audio and video reward signals, while noting those rewards alone do not establish overall quality; its launchers still use legacy V0. In MiniCPM-o 4.5 GSPO runs, colocated sync and separate-async show similar early learning, but placement efficiency depends on response length: sync is faster per prompt for short answers, the approaches are roughly tied at 512 tokens, and separate-async is faster at 1024 tokens in the reported single-run experiment. Upgrading requires checking recipe-specific trainer support and updated dependency pins; offline DPO remains on V0, and custom diffusion integrations must use named media artifacts.

### Source excerpt

VeRL-Omni v0.3.0 expands async RL, makes the V1 trainer the default, and adds reproducible recipes for audio-video generation and multimodal understanding.

## vLLM Delivers Day-0 Support for Kimi K3.1 Pro

DevFeed: [vLLM Delivers Day-0 Support for Kimi K3.1 Pro](<https://devfeed.tech/articles/vllm-delivers-day-0-support-for-kimi-k3-1-pro-118946.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-04-kimi-k31-pro>)

Author: Inferact and the vLLM Team

Published: 2026-11-04T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [lora](<https://devfeed.tech/topics/lora.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [context](<https://devfeed.tech/tags/context.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decode](<https://devfeed.tech/tags/decode.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kimi-k3-1-pro](<https://devfeed.tech/tags/kimi-k3-1-pro.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [models](<https://devfeed.tech/tags/models.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>), [token](<https://devfeed.tech/tags/token.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM's day-zero Kimi K3.1 Pro support combines sparse attention with its KDA/MLA hybrid and a packed cache for MLA latents, index keys, and KDA state. DSA scores the full context, selects 2,048 tokens, then runs exact attention over those tokens; at long context it reads about 23% of dense MLA's bytes per layer. The cache supports FP8 or NVFP4 latents and optional MXFP4 index keys, while token-sharded prefill reduces repeated indexer work. Decode context parallelism increases KV capacity but can raise per-step latency when the context fits; a fused decode path improves latency. Support spans NVIDIA, AMD, TPU, and Alibaba T-Head hardware, with different backends and configurations described for each. The article also reports bit-identical vLLM/Megatron log-probs in a 100-step RL run, while noting batch-invariant path optimization is ongoing and the DSpark checkpoint is a preview.

### Source excerpt

vLLM supports Kimi K3.1 Pro on day 0: sparse attention on a KDA hybrid, a packed KV cache with FP8 or NVFP4 latents and MXFP4 index keys, token-sharded prefill, decode context parallelism, and fused low-latency decode.

## Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes

DevFeed: [Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes](<https://devfeed.tech/articles/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and-kubernetes-118945.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-04-introducing-kinference>)

Author: the Kinference Team

Published: 2026-11-04T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [7-4](<https://devfeed.tech/tags/7-4.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [control](<https://devfeed.tech/tags/control.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-control-plane](<https://devfeed.tech/tags/inference-control-plane.md>), [kinference](<https://devfeed.tech/tags/kinference.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [policies](<https://devfeed.tech/tags/policies.md>), [rate-limiting](<https://devfeed.tech/tags/rate-limiting.md>), [scalable-platforms](<https://devfeed.tech/tags/scalable-platforms.md>), [token](<https://devfeed.tech/tags/token.md>), [tracking](<https://devfeed.tech/tags/tracking.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

Kinference is an open source control plane that uses vLLM for inference and exposes an HTTP endpoint for OpenAI- or Anthropic-compatible traffic. It adds token rate limiting, tenant usage and cost tracking, inference policies, and load balancing based on GPU queue sizes, historical performance, and KV-cache contents. The source reports that on one benchmark workload it cut median time-to-first-token by 95% and end-to-end latency by 45% versus standard Kubernetes load balancing; capacity benchmarks reported 7.4x goodput on shared-prefix and about 3.5x on multiturn workloads, with results tied to the stated benchmark setup and SLO. Policies can route or reject requests using factors including tenant, requested model, GPU state, and prompt classification, and can burst traffic to cloud providers when GPUs are saturated. Metrics include token rates, latency, cache usage, model use, and GPU saturation, with most available per tenant. The installation outline requires Gateway API CRDs and separate Kinference gateway and inference-handler components; the text directs readers to current documentation because steps may change.

### Source excerpt

Kinference is a new open source inference control plane for building scalable inference platforms with vLLM and Kubernetes, providing token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.

## Day 0 support for Mistral Large 4 on vLLM

DevFeed: [Day 0 support for Mistral Large 4 on vLLM](<https://devfeed.tech/articles/day-0-support-for-mistral-large-4-on-vllm-107223.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-29-mistral-large-4>)

Author: Mistral and the vLLM Team

Published: 2026-10-29T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [mixtral](<https://devfeed.tech/topics/mixtral.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [disaggregated-serving](<https://devfeed.tech/tags/disaggregated-serving.md>), [dspark](<https://devfeed.tech/tags/dspark.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [mistral-large-4](<https://devfeed.tech/tags/mistral-large-4.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [model-support](<https://devfeed.tech/tags/model-support.md>), [nixl](<https://devfeed.tech/tags/nixl.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM added day-zero Mistral Large 4 support across NVIDIA Blackwell, AMD MI350 with ROCm, and TPU v7x, with platform-specific kernels and quantization paths. On NVIDIA, the 72-layer MoE model uses sparse MLA and a top-2048 DSA indexer, shares KV across nine layers, and supports built-in DSpark speculative decoding, DCP/PCP long-context serving, NIXL prefill/decode disaggregation, and HiSparse host-memory offload. DSpark roughly doubles output throughput in reported TP8 tests; DCP8 increases KV capacity but can slow decode, so it pays off when cache capacity limits concurrency or context length. NVFP4 was reported close to FP8 quality while decoding faster and holding more KV cache. AMD supports image inputs, DSpark, expert parallelism, and MXFP4 experts; specialized decode lowers latency but requires TP4 and specific activation settings and supports only defined decode workloads. TPU v7x support uses sparse MLA and DSA Pallas kernels. HiSparse expands reported TP4 GB300 capacity from 1.97M GPU-only KV tokens to 8.77M admitted at startup with a 440 GiB host pool; single-node TP deployments require a stated FlashInfer all-reduce setting to avoid a known crash.

### Source excerpt

Day-0 vLLM support for Mistral Large 4 with sparse MLA, cross-layer KV sharing, DSpark speculative decoding, DCP/PCP long context and NIXL P/D on NVIDIA Blackwell.

## vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash

DevFeed: [vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash](<https://devfeed.tech/articles/vllm-delivers-day-0-support-for-minimax-m3-1-flash-96723.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-23-minimax-m31-flash>)

Author: Inferact and the vLLM Team

Published: 2026-10-23T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [rocm](<https://devfeed.tech/topics/rocm.md>), [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [dspark](<https://devfeed.tech/tags/dspark.md>), [flash](<https://devfeed.tech/tags/flash.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [m3](<https://devfeed.tech/tags/m3.md>), [m3-1-flash](<https://devfeed.tech/tags/m3-1-flash.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [minimax-m3-1-flash](<https://devfeed.tech/tags/minimax-m3-1-flash.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>), [nvidia-rubin](<https://devfeed.tech/tags/nvidia-rubin.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rl](<https://devfeed.tech/tags/rl.md>), [series](<https://devfeed.tech/tags/series.md>), [support](<https://devfeed.tech/tags/support.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM provides day-0 MiniMax-M3.1-Flash support on NVIDIA Blackwell and Rubin and AMD MI350-series GPUs, with NVFP4 KV cache and MoE support, DSpark speculative decoding, and RL training from the NVFP4 checkpoint. The model's sparse attention selects at most 16 blocks of 128 tokens per query, limiting attention to 2,048 keys; the article estimates long-context decode HBM traffic at about 12.5% of dense attention. On four B300s, NVFP4 reduces checkpoint size 44.6% and KV-cache bytes per token 43.75%; at 256K context, the stated deployment fits 76 sequences versus 28 for the prior model's FP8 cache. Hardware-specific optimizations include CUDA 13.4 QMUL4 kernels on Blackwell, distinct MSA kernels on Rubin, and table-based NVFP4 conversion, context parallelism, and fused low-latency decoding on AMD. Reported AgentX results vary by platform and metric: Rubin reaches 724.9 tokens/s per user at the latency-sensitive frontier, while AMD MI355X reaches 133.3M total tokens per dollar at the cost-efficient frontier. Day-0 RL uses veRL, FSDP2, rank-32 LoRA, and token-level importance sampling; a 16-GB300, 200-step run raises AIME-2024 accuracy from 36.3% to 67.5%. Prebuilt NVIDIA and AMD vLLM-

### Source excerpt

Day-0 vLLM support for MiniMax-M3.1-Flash on NVIDIA Blackwell, NVIDIA Rubin, and AMD MI350-series GPUs, with NVFP4 MSA kernels, NVFP4 MegaMoE, DSpark speculative decoding, AgentX results, and day-0 RL.

## GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s

DevFeed: [GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s](<https://devfeed.tech/articles/glm-5-3-optimizations-part-2-pcp-dcp-and-hybrid-hisparse-on-16-h200s-98534.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-23-glm53-part2-pcp-dcp-hisparse>)

Author: vLLM Team

Published: 2026-10-23T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-benchmark](<https://devfeed.tech/tags/agentic-benchmark.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cache](<https://devfeed.tech/tags/cache.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [context](<https://devfeed.tech/tags/context.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [glm](<https://devfeed.tech/tags/glm.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [performance](<https://devfeed.tech/tags/performance.md>), [speed](<https://devfeed.tech/tags/speed.md>), [token](<https://devfeed.tech/tags/token.md>), [user](<https://devfeed.tech/tags/user.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

On two 8x H200 nodes, vLLM combines PCP8 x DCP8 prefill with Hybrid HiSparse decode to serve GLM-5.3 agentic workloads with long contexts. PCP distributes prompt queries by token across GPUs, reducing replicated indexer work and improving long-prompt latency; DCP shards the persistent KV cache eight ways, allowing a 1M-token request to fit at 6.7 GiB per GPU. DCP adds all-gather overhead during prefill--about 6% per step through 64K cached tokens, rising to 15% at 248K--but enables contexts beyond PCP8's cache capacity. Hybrid HiSparse keeps decode KV in host memory, supporting higher long-context concurrency. On AgentX, the combined setup achieved 1.7-3.2x the per-user speed of DP8 prefill at equal throughput and cut median TTFT from 1.5-5.2 s to 0.8-0.9 s; at concurrency 64, HiSparse nearly doubled throughput to 8,340 logical tokens/s/GPU. The 1M-context result required both methods: the tested DP8 prefill and offload-only decode configurations could not hold such a request.

### Source excerpt

vLLM serves GLM-5.3 on 16 H200s with PCP and DCP prefill and Hybrid HiSparse decode, reaching 1M-token contexts and 1.7-3.2x the per-user speed of data-parallel P/D on the AgentX agentic benchmark.

## Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72

DevFeed: [Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72](<https://devfeed.tech/articles/serving-deepseek-v4-1-flash-at-scale-vllm-matches-deepinfer-and-scales-to-gb300-nvl72-98978.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-23-deepseek-v41-flash-pd-wide-ep>)

Author: Inferact and the vLLM Team

Published: 2026-10-23T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [GB300 NVL72](<https://devfeed.tech/topics/gb300-nvl72.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [async](<https://devfeed.tech/tags/async.md>), [attention](<https://devfeed.tech/tags/attention.md>), [b300](<https://devfeed.tech/tags/b300.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [deepinfer](<https://devfeed.tech/tags/deepinfer.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [gb300-nvl72](<https://devfeed.tech/tags/gb300-nvl72.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [large-scale-serving](<https://devfeed.tech/tags/large-scale-serving.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rack](<https://devfeed.tech/tags/rack.md>), [scale](<https://devfeed.tech/tags/scale.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [user](<https://devfeed.tech/tags/user.md>), [v4-1-flash](<https://devfeed.tech/tags/v4-1-flash.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [wide](<https://devfeed.tech/tags/wide.md>)

### AI overview

On one GB300 NVL72 rack, vLLM's P/D-disaggregated, Wide-EP serving of DeepSeek-V4.1-Flash reaches 954K tokens/s per GPU on SemiAnalysis AgentX at a P90 of 196 tokens/s per user, reported as 7.9x the authors' earlier single-node result. Decode throughput on GB200 is within 6% of DeepSeek's DeepInfer across batch sizes 8-256; the prefill curves track, though differing setups prevent a direct speedup comparison. The reported gains come from fitting prefill to DEP2, extending bounded replay to layer 20, using larger GPU pages for shared Engram tables and DeepSelect for prefill ranking, plus Wide-EP with EPLB for decode. At DEP32, EPLB substantially improves expert-load balance, while CPU overhead can erase GPU-side scaling gains. A specialized decode path reaches 826 P90 tokens/s for one user on four B300 GPUs, 44% above the standard path. Results depend on the stated hardware, configurations and workloads.

### Source excerpt

With P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU at a P90 of 196 tokens/s per user, 7.9x our aggregated single-node result.

## Announcing Humming: Fast, Flexible Kernels for Quantized Inference

DevFeed: [Announcing Humming: Fast, Flexible Kernels for Quantized Inference](<https://devfeed.tech/articles/announcing-humming-fast-flexible-kernels-for-quantized-inference-91851.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-21-humming>)

Author: Ant Group and the vLLM Community

Published: 2026-10-21T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [1-6](<https://devfeed.tech/tags/1-6.md>), [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [community](<https://devfeed.tech/tags/community.md>), [fast](<https://devfeed.tech/tags/fast.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [integration](<https://devfeed.tech/tags/integration.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [library](<https://devfeed.tech/tags/library.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [moe](<https://devfeed.tech/tags/moe.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [results](<https://devfeed.tech/tags/results.md>), [support](<https://devfeed.tech/tags/support.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Humming is an open-source quantized inference kernel library from Ant Group and the vLLM community. It standardizes quantization descriptions from different model exports, then selects device and workload specific configurations and JIT compiles kernels from reusable CUDA components. It supports NVIDIA GPUs from Turing onward and T-Head's Zhenwu 810E, with broad weight and activation format combinations for dense and MoE GEMMs. The article reports gains over vLLM Marlin, FlashInfer and DeepGEMM in specified benchmarks; in one DeepSeek-V4.1-Flash test on eight H20 GPUs, changing MoE execution from Marlin W4A16 to Humming W4A8 raised throughput about 14-43% and lowered mean TTFT about 25-32%. Humming is integrated into vLLM from 0.29.0; users select it separately for dense and MoE backends. Activation formats default to checkpoint settings, with fallback to supported higher precision when needed. Autotuning is planned, not yet described as available.

### Source excerpt

Introducing Humming, a quantized inference kernel library from Ant Group and the vLLM community, with broad GPU and quantization support, dense and MoE performance results, and vLLM integration.

## Day-0 Support for Flint on vLLM

DevFeed: [Day-0 Support for Flint on vLLM](<https://devfeed.tech/articles/day-0-support-for-flint-on-vllm-93046.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-21-flint>)

Author: Inferact and Recursive

Published: 2026-10-21T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [attention](<https://devfeed.tech/tags/attention.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [context](<https://devfeed.tech/tags/context.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [docker](<https://devfeed.tech/tags/docker.md>), [dsl](<https://devfeed.tech/tags/dsl.md>), [experts](<https://devfeed.tech/tags/experts.md>), [flint](<https://devfeed.tech/tags/flint.md>), [gb200](<https://devfeed.tech/tags/gb200.md>), [gguf](<https://devfeed.tech/tags/gguf.md>), [glm](<https://devfeed.tech/tags/glm.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [one](<https://devfeed.tech/tags/one.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [request](<https://devfeed.tech/tags/request.md>), [support](<https://devfeed.tech/tags/support.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM supports both Recursive Flint checkpoints, NVFP4 and BF16, on a single Blackwell GPU. Flint is a 40B-parameter MoE with about 2.5B active per token and a 262K context; most layers use sliding-window attention, limiting full-sequence KV storage to 14 layers. For batch-1 decoding at 100K context on one GB200, end-to-end speed is about 485 tokens/s. Flint-specific kernels reduced NVFP4 decode time from 4.81 to 1.78 ms and BF16 from 3.53 to 2.34 ms; an 8K prefill fell from 207 to 145 ms. NVFP4 runs weight-only with BF16 activations and KV cache, repacked losslessly at load time. The optimized path targets up to eight requests; kernels target TP=1 on Blackwell, with other configurations falling back to standard vLLM layers. Usage notes specify GLM-4.7 parsers and that reasoning counts toward max_tokens.

### Source excerpt

vLLM supports both Flint checkpoints (40B total, 2.5B active) from day 0, decoding a single request at about 500 tokens/s on one GPU.

## Introducing vLLM TorchTPU

DevFeed: [Introducing vLLM TorchTPU](<https://devfeed.tech/articles/introducing-vllm-torchtpu-91152.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-20-vllm-torchtpu>)

Author: Inferact and Google Teams

Published: 2026-10-20T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [backend](<https://devfeed.tech/tags/backend.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [offloading](<https://devfeed.tech/tags/offloading.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [torchtpu](<https://devfeed.tech/tags/torchtpu.md>), [tpus](<https://devfeed.tech/tags/tpus.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM TorchTPU is an out-of-tree vLLM backend for Google Cloud TPUs that reuses vLLM model code and its scheduler, workers, KV cache manager, and API server. It compiles models through `torch.compile(backend="tpu")` to XLA and adds TPU-specific JAX/Pallas kernels for attention, MoE, and quantized matrix operations. The excerpt reports support for all six vLLM parallelism dimensions across single- and multi-host TPU slices, plus KV cache offloading to host DRAM and disaggregated prefill/decode with cache resharding. Installation requires Python 3.12 on a Cloud TPU VM; first startup compiles graphs and may take a few minutes, while `VLLM_XLA_CACHE_PATH` can retain compiled artifacts across restarts. TPU v7x is recommended and tested in CI; v6e, v5e, and v5p are experimental.

### Source excerpt

vllm-torchtpu is a new vLLM backend for Google Cloud TPUs built on TorchTPU, with Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving.

## Achieving near-zero-downtime scaling in Elastic Expert Parallelism

DevFeed: [Achieving near-zero-downtime scaling in Elastic Expert Parallelism](<https://devfeed.tech/articles/achieving-near-zero-downtime-scaling-in-elastic-expert-parallelism-90281.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime>)

Author: Itay Alroy (NVIDIA)

Published: 2026-10-19T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Deployment & Scaling](<https://devfeed.tech/topics/deployment-scaling.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>), [Heroku AI](<https://devfeed.tech/topics/heroku-ai.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [autoscalers](<https://devfeed.tech/tags/autoscalers.md>), [batch](<https://devfeed.tech/tags/batch.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cuda-graphs](<https://devfeed.tech/tags/cuda-graphs.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [elastic-ep](<https://devfeed.tech/tags/elastic-ep.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [large-scale-serving](<https://devfeed.tech/tags/large-scale-serving.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nixl](<https://devfeed.tech/tags/nixl.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article describes how vLLM uses asynchronous preparation and NIXL EP to scale Elastic Expert Parallelism while keeping existing workers serving requests. In a DeepSeek V4 Pro benchmark, this approach reduced serving interruptions to 0.82 seconds when scaling from 16 to 20 GPUs and 1.96 seconds when scaling back. It explains how NIXL EP reuses communication resources and CUDA graphs, and how joining workers prepare independently. In a load test, the deployment completed all 11,391 requests while remaining below the stated p95 time-to-first-token objective.

### Source excerpt

How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM.

## Serving Reflection Beam on vLLM from Day 0

DevFeed: [Serving Reflection Beam on vLLM from Day 0](<https://devfeed.tech/articles/serving-reflection-beam-on-vllm-from-day-0-88770.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-16-reflection-beam>)

Author: Inferact, Reflection, and Nvidia

Published: 2026-10-16T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reflection](<https://devfeed.tech/tags/reflection.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM describes serving Reflection's Beam-1.0, a 501-billion-parameter mixture-of-experts model, and details its support for reasoning, tool calls, and structured outputs. The article explains model-specific kernel and serving optimizations, reporting 2.71x faster batch-one decoding and 1.87x faster prefill than the initial implementation. It also covers DFlash2 speculative decoding, adaptive verification, performance benchmarks, and setup options.

### Source excerpt

vLLM supports Reflection's Beam-1.0 (501B total, 23B active) from day 0, with agent-driven optimization that cut decode latency 2.71x and DFlash2 speculative decoding with adaptive verification.

## How vLLM Optimized Kimi K3 Inference on AMD Instinct MI355X

DevFeed: [How vLLM Optimized Kimi K3 Inference on AMD Instinct MI355X](<https://devfeed.tech/articles/scaling-kimi-k3-on-amd-instinct-mi355x-matching-nvidia-hgx-b300-throughput-per-gpu-on-agentx-88394.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-16-kimi-k3-mi355x>)

Author: AMD and Embedded LLM

Published: 2026-10-16T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [rocm](<https://devfeed.tech/topics/rocm.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [queueing theory](<https://devfeed.tech/topics/queueing-theory.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [models](<https://devfeed.tech/tags/models.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rocm](<https://devfeed.tech/tags/rocm.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article explains engineering changes to vLLM serving Kimi K3 on AMD Instinct MI355X GPUs, including cache reuse and offload, avoiding duplicated expert projection work, reducing kernel handoffs and copies, and parallelizing long-context attention and recurrent workloads. It reports that the AgentX benchmark measured 24.4% more total tokens per second per GPU than NVIDIA HGX B300 at a 50 tokens/s/user P90 target, and 87.4% more tokens per dollar under the stated ownership-cost model. The article also describes limitations and separate test conditions for individual optimizations.

### Source excerpt

Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster

## Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly

DevFeed: [Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly](<https://devfeed.tech/articles/hyper-optimizing-kimi-k3-in-vllm-moving-fast-and-responsibly-88988.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel>)

Author: Benjamin Chislett (NVIDIA)

Published: 2026-10-16T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>)

Tags: [403](<https://devfeed.tech/tags/403.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [code-review](<https://devfeed.tech/tags/code-review.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article describes agent-assisted CuTe DSL kernel optimizations for Kimi K3 inference on NVIDIA GB300 GPUs, including attention, MoE scheduling, communication overlap, and weight prefetching. It reports throughput and interactivity gains and outlines a validation approach based on scoped changes, correctness testing, workload coverage, and reproducible benchmarks.

### Source excerpt

How agent-assisted CuTe DSL optimizations accelerate Kimi K3 decode on NVIDIA GB300, and how scoped changes, correctness tests, and reproducible benchmarks help ship them responsibly.

## Day 0 support for Muse Spark Open on vLLM

DevFeed: [Day 0 support for Muse Spark Open on vLLM](<https://devfeed.tech/articles/day-0-support-for-muse-spark-open-on-vllm-88076.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-15-muse-spark-open>)

Author: vLLM Team

Published: 2026-10-15T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [model-support](<https://devfeed.tech/tags/model-support.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [tpu](<https://devfeed.tech/tags/tpu.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM adds day-zero support for Meta's Muse Spark Open model, an 816B-parameter mixture-of-experts model with a 1M-token context and image, video, and audio inputs. The article describes NVIDIA and AMD inference support, initial TPU support, DSpark speculative decoding, and LoRA reinforcement-learning workflows.

### Source excerpt

Day-0 vLLM support for Muse Spark Open: 816B MoE, 1M context, multimodal inputs, DSpark speculative decoding, NVIDIA and AMD serving, initial TPU support, and LoRA RL.

## Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM

DevFeed: [Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM](<https://devfeed.tech/articles/announcing-day-0-support-for-nvidia-nemotron-3-5-super-vl-on-vllm-88077.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-15-nemotron-3-5-super-vl-vllm>)

Author: NVIDIA Nemotron Team

Published: 2026-10-15T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [batching](<https://devfeed.tech/tags/batching.md>), [caching](<https://devfeed.tech/tags/caching.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [image](<https://devfeed.tech/tags/image.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [model-support](<https://devfeed.tech/tags/model-support.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvidia-nemotron](<https://devfeed.tech/tags/nvidia-nemotron.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [token](<https://devfeed.tech/tags/token.md>), [video](<https://devfeed.tech/tags/video.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

NVIDIA announces support for Nemotron 3.5 Super VL in vLLM, describing a multimodal model for long-running agent workflows that can process text, images, documents, and video. The article explains serving options, including an OpenAI-compatible API, continuous batching, prefix caching, speculative decoding, and NVFP4 inference, and outlines customization with NVIDIA NeMo. It also reports performance comparisons for agentic tasks and points readers to model weights and a getting-started cookbook.

### Source excerpt

Run NVIDIA Nemotron 3.5 Super VL on vLLM for multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.

[Next page](<https://devfeed.tech/sources/vllm-blog.md?cursor=WyIyMDI2LTEwLTE1VDAwOjAwOjAwKzAwOjAwIiwgIjdjMDRjZTMzLWFmY2UtNDAzNS1hYmI5LTQ0YjNkOGQxY2M5NSJd>)