# Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

DevFeed: [Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries](<https://devfeed.tech/articles/keep-the-tokens-flowing-lessons-from-16-open-source-rl-libraries-7109.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/async-rl-training-landscape>)

Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Lewis Tunstall; Edward Beeching; Albert Villanova del Moral; Nouamane Tazi; Leandro von Werra; Sergio Paniego

Published: 2026-03-10T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [llm](<https://devfeed.tech/tags/llm.md>), [lora](<https://devfeed.tech/tags/lora.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rl](<https://devfeed.tech/tags/rl.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

## AI overview

This article surveys 16 open-source libraries for asynchronous reinforcement-learning training. It explains how separating inference and training across GPU pools, using rollout buffers, and synchronizing weights asynchronously can reduce training-GPU idle time. The comparison covers orchestration, buffering, weight synchronization, staleness management, partial rollouts, LoRA, and distributed-training backends, highlighting Ray, NCCL broadcasts, limited LoRA support, and distributed MoE as an emerging differentiator.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.