# Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

DevFeed: [Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1](<https://devfeed.tech/articles/zero-mismatch-simple-async-rl-reinforce-zero-deepseek-v4-1-and-kimi-k3-1-122091.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-06-zero-mismatch-async-rl>)

Author: Ao Shen (Inferact), Jian Hu (NVIDIA Research), Yan Bai (NVIDIA DevTech), Shekhar Pandey (AMD), and the REINFORCE Zero authors

Published: 2026-11-06T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [async](<https://devfeed.tech/topics/async.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [7-4](<https://devfeed.tech/tags/7-4.md>), [8k](<https://devfeed.tech/tags/8k.md>), [about](<https://devfeed.tech/tags/about.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [align](<https://devfeed.tech/tags/align.md>), [amd](<https://devfeed.tech/tags/amd.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [flash](<https://devfeed.tech/tags/flash.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [lag](<https://devfeed.tech/tags/lag.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [training](<https://devfeed.tech/tags/training.md>), [trains](<https://devfeed.tech/tags/trains.md>), [v4-1-flash](<https://devfeed.tech/tags/v4-1-flash.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>), [zero](<https://devfeed.tech/tags/zero.md>)

## AI overview

REINFORCE Zero argues that asynchronous RL can train stably on stale samples when rollout and learner kernels are aligned, leaving unclipped token level importance sampling to correct weight lag. The paper reports stable training through 20,000 updates and learning at policy lag 64; in one comparison, importance weight deviation peaked at 1.50 times its median with aligned kernels versus 442.54 times with standard kernels. Without alignment, training drifted and a sequence gate rejected 24.48% of sequences. Alignment has a speed cost: one OSWorld setup took 6.8% longer per step than standard kernel IS with a gate. The excerpt also reports zero-mismatch deployments for DeepSeek-V4.1-Flash and Kimi K3.1, including AMD MI355X runs, while noting ongoing performance work and that long asynchronous runs on the flagship models remain future work.

## Source excerpt

Asynchronous RL trains on stale weights. Align rollout and training kernels bit for bit, and unclipped importance sampling corrects that lag. Plus zero-mismatch RL for DeepSeek-V4.1-Flash, Kimi K3.1 and AMD MI355X.