# PipelineRL

DevFeed: [PipelineRL](<https://devfeed.tech/articles/pipelinerl-7056.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ServiceNow/pipelinerl>)

Author: Alex Piche; Rafael Pardinas; Ehsan Kamalloo; Bahdanau

Published: 2025-04-25T22:37:16Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [gradient\_accumulation](<https://devfeed.tech/topics/gradient-accumulation.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [math](<https://devfeed.tech/topics/math.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rl](<https://devfeed.tech/tags/rl.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

This blog post presents PipelineRL, a modular reinforcement-learning architecture for training language models. It uses inflight weight updates to preserve efficient inference batching while keeping collected data on-policy or near on-policy. The article reports competitive results against Open-Reasoner-Zero on AIME 2024 and MATH 500 using a simpler GRPO-based implementation.

## Source excerpt

In this blog post, we show that 1) inflight weight updates do not harm the training process and 2) PipelineRL achieves competitive results compared to Open-Reasoner-Zero, while using a simpler RL algorithm. We also present the modular PipelineRL architecture that facilitates trying new inference / trainer combinations. In conventional RL approaches (Figure 1a), there is a trade-off between high throughput inference and on-policy data collection.