# Putting RL back in RLHF

DevFeed: [Putting RL back in RLHF](<https://devfeed.tech/articles/putting-rl-back-in-rlhf-7446.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/putting_rl_back_in_rlhf_with_rloo>)

Author: Shengyi Costa Huang; Arash Ahmadian

Published: 2024-06-12T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [cohere](<https://devfeed.tech/topics/cohere.md>)

Tags: [cohere](<https://devfeed.tech/tags/cohere.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [rl](<https://devfeed.tech/tags/rl.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [trl](<https://devfeed.tech/tags/trl.md>)

## AI overview

This article introduces the RLOO Trainer in TRL, an online reinforcement learning algorithm for RLHF designed as a more accessible alternative to PPO. It explains that RLOO uses less GPU memory, converges faster, performs competitively with PPO, and outperforms offline methods such as DPO in the reported comparisons.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.