# trl Published articles for trl. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL DevFeed: [Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Sergio Paniego Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [lora](), [grpo](), [jobs](), [vllm](), [NCCL](), [async](), [hugging face]() Tags: [async](), [grpo](), [hub](), [hugging-face](), [jobs](), [llm](), [lora](), [nccl](), [open-source](), [rl](), [storage](), [trl](), [vllm]() ### AI overview This article describes asynchronous GRPO training with a LoRA adapter across separate Hugging Face Jobs. The adapter is synchronized to vLLM replicas through a shared Storage Bucket, while a proxy handles authentication, rollout routing, and adapter-load broadcasts. Five runs reduced the time for 500 steps from 3 hours 27 minutes to 53 minutes. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Training a coding model to paint watercolours with TRL and OpenEnv DevFeed: [Training a coding model to paint watercolours with TRL and OpenEnv]() Original publisher: [Read original article]() Author: Sergio Paniego Published: 2026-09-03T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [openenv](), [coding](), [Language models](), [hugging face](), [AI Development](), [jobs](), [dataset]() Tags: [ai-art](), [coding](), [grpo](), [guide](), [hugging-face](), [javascript](), [jobs](), [open-source](), [openenv](), [rl](), [spaces](), [training](), [trl]() ### AI overview A tutorial describing an open reproduction of a reinforcement-learning pipeline that trains a coding model to create watercolor-like paintings by writing JavaScript with p5.brush. It uses TRL and OpenEnv, with datasets, environments, training scripts, models, and other artifacts published on Hugging Face. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps DevFeed: [Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps]() Original publisher: [Read original article]() Author: Leonie Monigatti; ben burtenshaw; Sergio Paniego Published: 2026-09-03T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [AI Chat]() Tags: [artificial-intelligence](), [benchmark](), [community](), [evaluation](), [fine-tuning](), [gpu](), [grpo](), [guide](), [json](), [llama-cpp](), [llm](), [model](), [nlp](), [open-source](), [python](), [rl](), [training](), [trl]() ### AI overview A tutorial on fine-tuning a 350M language model with GRPO to improve structured-output and JSON Schema compliance, then evaluating it on the IFStruct benchmark. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL DevFeed: [Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Lewis Tunstall; Edward Beeching; Albert Villanova del Moral; Leandro von Werra; Sergio Paniego Published: 2026-05-27T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [AI, ML & Data Engineering](), [NCCL]() Tags: [artificial-intelligence](), [compute](), [hub](), [hugging-face](), [inference](), [llm](), [model](), [models](), [nccl](), [open-source](), [payload](), [policy](), [rl](), [space](), [storage](), [sync](), [train](), [training](), [trl](), [update](), [vllm]() ### AI overview The article describes delta weight synchronization for asynchronous reinforcement-learning training. A TRL change stores only modified model weights in sparse safetensors files and lets vLLM fetch them from a Hugging Face bucket, reducing transfer payloads and enabling disaggregated training without a shared cluster. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## TRL v1.0: Post-Training Library Built to Move with the Field DevFeed: [TRL v1.0: Post-Training Library Built to Move with the Field]() Original publisher: [Read original article]() Author: Quentin Gallouédec; Steven Liu; Pedro Cuenca; Sergio Paniego Published: 2026-03-31T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [trl](), [post-training](), [dpo](), [grpo](), [rlvr](), [Artificial Intelligence](), [Code]() Tags: [announcement](), [artificial-intelligence](), [code](), [dpo](), [grpo](), [open-source](), [post-training](), [reinforcement-learning](), [rlvr](), [software](), [trl](), [v1]() ### AI overview TRL v1.0 presents a post-training library designed to remain useful as algorithms, models, and paradigms change. The article explains the evolution from PPO to DPO-style preference optimization and RLVR methods such as GRPO, emphasizing flexible abstractions and practical comparison over a supposedly permanent design. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Ulysses Sequence Parallelism: Training with Million-Token Contexts DevFeed: [Ulysses Sequence Parallelism: Training with Million-Token Contexts]() Original publisher: [Read original article]() Author: Kashif Rasul; Stas Bekman Published: 2026-03-09T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Artificial Intelligence](), [long-context](), [GPU](), [hugging face](), [Open Source](), [trl]() Tags: [accelerate](), [ai](), [artificial-intelligence](), [distributed-training](), [ecosystem](), [gpu](), [guide](), [long-context](), [open-source](), [tokens](), [transformers](), [trl]() ### AI overview This article explains Ulysses Sequence Parallelism, a method for training transformer models with very long or million-token contexts by sharding sequences and partitioning attention heads across multiple GPUs. It describes the all-to-all communication steps and integration across the Hugging Face ecosystem, including Accelerate, Transformers Trainer, and TRL's SFTTrainer. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## DeepMath: A lightweight math reasoning Agent with smolagents DevFeed: [DeepMath: A lightweight math reasoning Agent with smolagents]() Original publisher: [Read original article]() Author: Daniel Fleischer; Moshe Berchansky; Moshe Wasserblat Published: 2025-12-04T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [smolagents](), [math](), [grpo](), [Fine-tuning](), [Python](), [Large Language Model](), [vllm](), [trl](), [datasets](), [Inference](), [Artificial Intelligence](), [Open Source]() Tags: [agents](), [artificial-intelligence](), [datasets](), [fine-tuning](), [grpo](), [inference](), [large-language-models-llms](), [llm](), [math](), [math-reasoning](), [open-source](), [python](), [reasoning](), [sandbox](), [smolagents](), [trl](), [vllm]() ### AI overview DeepMath is a lightweight math reasoning agent built on Qwen3-4B Thinking and implemented with smolagents. It uses GRPO fine-tuning to encourage concise Python snippets for deterministic computation, executes those snippets in a constrained sandbox, and reintegrates the results into the reasoning process. Evaluation on MATH500, AIME, HMMT, and HLE reports shorter outputs and often improved accuracy. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## 20x Faster TRL Fine-tuning with RapidFire AI DevFeed: [20x Faster TRL Fine-tuning with RapidFire AI]() Original publisher: [Read original article]() Author: Kamran Bigdely; Arun Kumar; Quentin Gallouédec Published: 2025-11-21T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Fine-tuning](), [trl](), [Large Language Model](), [post-training](), [Multi-GPU](), [Orchestration](), [dashboards](), [GPU](), [data]() Tags: [dashboards](), [data](), [dpo](), [experimentation](), [fine-tuning](), [gpu](), [grpo](), [llm](), [llms](), [logs](), [metrics](), [multi-gpu](), [orchestration](), [post-training](), [rapidfireai](), [trl]() ### AI overview RapidFire AI accelerates LLM fine-tuning and post-training experimentation by running multiple TRL configurations concurrently, including on a single GPU. Its adaptive chunk-based scheduling, live metrics dashboard, multi-GPU orchestration, and interactive controls help teams compare configurations sooner, stop weak runs, and clone promising ones. The article cites internal benchmarks reporting approximately 16-24x higher experimentation throughput than sequential comparison. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Vision Language Model Alignment in TRL ⚡ DevFeed: [Vision Language Model Alignment in TRL ⚡]() Original publisher: [Read original article]() Author: Sergio Paniego; merve; Quentin Gallouédec; Kashif Rasul; Aritra Roy Gosthipaty Published: 2025-08-07T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [trl](), [vlm](), [multimodal](), [Fine-tuning]() Tags: [artificial-intelligence](), [dpo](), [fine-tuning](), [grpo](), [model](), [multimodal](), [trl](), [vision](), [vlm]() ### AI overview This article presents new and extended methods in TRL for aligning Vision Language Models with human preferences. It covers Mixed Preference Optimization, Group Relative Policy Optimization, Group Sequence Policy Optimization, Reinforce Leave One Out, and Online Direct Preference Optimization, along with native supervised fine-tuning support, training scripts, and demo notebooks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL DevFeed: [No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL]() Original publisher: [Read original article]() Author: Mert Toslali; Yu Chin Fabian Lim; Quentin Gallouédec; Ed Snible; Raghu Ganti; Mudhakar Srivatsa Published: 2025-06-03T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vllm](), [trl](), [GPU](), [Inference](), [Machine learning]() Tags: [efficiency](), [gpu](), [grpo](), [hardware](), [inference](), [llm](), [performance](), [rlhf](), [train](), [training](), [trl](), [vllm]() ### AI overview This article explains how TRL integrates vLLM to accelerate GRPO training of LLMs. It describes the GPU inefficiencies caused by running training and generation on separate devices and introduces colocated vLLM, which allows both tasks to share GPUs within the same distributed process group. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## 🐯 Liger GRPO meets TRL DevFeed: [🐯 Liger GRPO meets TRL]() Original publisher: [Read original article]() Author: Shivam Sahni; Kashif Rasul; Salman Mohammadi; Shirin Yamani; Yanning Chen; Liberty Published: 2025-05-25T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [grpo](), [trl](), [Reinforcement learning](), [Fine-tuning](), [Optimization](), [GPU](), [LLMs](), [rlhf](), [coding](), [math]() Tags: [coding](), [fine-tuning](), [gpu](), [grpo](), [large-language-models-llms](), [liger](), [llm](), [optimization](), [partnership](), [performance](), [reinforcement-learning](), [rlhf](), [trl]() ### AI overview This article explains how Group Relative Policy Optimization (GRPO) can reduce the resource requirements of reinforcement learning fine-tuning for language models. It presents a TRL optimization based on chunked GRPO loss that reduces peak memory usage by 40% and discusses scaling GRPO across multiple GPUs and nodes while preserving performance and correctness. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Open-R1: Update #1 DevFeed: [Open-R1: Update #1]() Original publisher: [Read original article]() Author: Leandro von Werra; Lewis Tunstall; Quentin Gallouédec; Guilherme Penedo; Edward Beeching; Anton Lozhkov; Brigitte Tousignant; Daniel van Strien Published: 2025-02-02T00:04:28Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [deepseek](), [synthetic-data](), [datasets](), [grpo](), [trl](), [Benchmark](), [vllm](), [GPU]() Tags: [benchmark](), [deepseek](), [follow](), [gpu](), [grpo](), [leaderboard](), [synthetic-data](), [trl](), [vllm]() ### AI overview This update reports early progress on the open-r1 project to reproduce the DeepSeek-R1 training pipeline and dataset. It covers matching MATH-500 evaluation results, the unusually long model responses and their GPU-memory implications, a public evaluation leaderboard, and the integration of GRPO into TRL with DeepSpeed and vLLM support. ### Source excerpt It's been two weeks since the release of DeepSeek R1 and just a week since we started the open-r1 project to replicate the missing pieces, namely the training pipeline and the synthetic data. This post summarizes: - the progress of Open-R1 to replicate the DeepSeek-R1 pipeline and dataset - what we learned about DeepSeek-R1 and discussions around it - cool projects the community has built since the release of DeepSeek-R1 It should serve both as an update on the project and as a collection of... ## SmolVLM - small yet mighty Vision Language Model DevFeed: [SmolVLM - small yet mighty Vision Language Model]() Original publisher: [Read original article]() Author: Andres Marafioti; merve; Miquel Farré; Elie Bakouch; Pedro Cuenca Published: 2024-11-26T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [smolvlm](), [vlm](), [multimodal-ai](), [Transformers](), [Fine-tuning](), [Inference](), [Open Source Models & Datasets](), [GPU]() Tags: [apache](), [fine-tuning](), [gpu](), [inference](), [llm](), [multimodal](), [multimodal-ai](), [nlp](), [on-device](), [open](), [open-source](), [smolvlm](), [transformers](), [trl](), [vision]() ### AI overview This article introduces SmolVLM, a family of small, fast, memory-efficient 2B vision-language models released fully open source under the Apache 2.0 license. It describes the model variants, architecture, training resources, Transformers integration, demo, fine-tuning script, and efficient local or on-device deployment. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Preference Optimization for Vision Language Models DevFeed: [Preference Optimization for Vision Language Models]() Original publisher: [Read original article]() Author: Quentin Gallouédec; Shengyi Costa Huang; merve; Kashif Rasul Published: 2024-07-10T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [Language models]() Tags: [data](), [dpo](), [fine-tuning](), [gpu](), [images](), [language-models](), [multimodal](), [optimization](), [rlhf](), [training](), [trl](), [vlm](), [vlms]() ### AI overview A tutorial on training vision-language models with TRL's direct preference optimization support, covering preference data formatting and memory considerations. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Putting RL back in RLHF DevFeed: [Putting RL back in RLHF]() Original publisher: [Read original article]() Author: Shengyi Costa Huang; Arash Ahmadian Published: 2024-06-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [rlhf](), [trl](), [Reinforcement learning](), [dpo](), [cohere]() Tags: [cohere](), [dpo](), [gpu](), [reinforcement-learning](), [research](), [rl](), [rlhf](), [trl]() ### AI overview This article introduces the RLOO Trainer in TRL, an online reinforcement learning algorithm for RLHF designed as a more accessible alternative to PPO. It explains that RLOO uses less GPU memory, converges faster, performs competitively with PPO, and outperforms offline methods such as DPO in the reported comparisons. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Welcome Llama 3 - Meta's new open LLM DevFeed: [Welcome Llama 3 - Meta's new open LLM]() Original publisher: [Read original article]() Author: Philipp Schmid; Omar Sanseviero; Pedro Cuenca; Younes B; Leandro von Werra Published: 2024-04-18T00:00:00Z Content type: release Language: en Sources: [Hugging Face - Blog]() Topics: [llama](), [Meta](), [hugging face](), [Large Language Model](), [Artificial Intelligence](), [Open Source](), [Fine-tuning](), [Transformers](), [trl](), [Google AI](), [Amazon SageMaker](), [Google Cloud Platform (GCP)]() Tags: [amazon-sagemaker](), [artificial-intelligence](), [community](), [fine-tuning](), [google-cloud](), [gpu](), [hardware](), [hugging-face](), [inference](), [inference-endpoints](), [llama](), [llm](), [meta](), [nlp](), [open-source](), [research](), [tokens](), [transformers](), [trl]() ### AI overview Meta has released Llama 3, an open-access LLM family available through Hugging Face. The release includes 8B and 70B base and instruction-tuned models, plus Llama Guard 2 for classifying potentially unsafe LLM inputs and responses. It also adds integrations with Transformers, Hugging Chat, Inference Endpoints, Google Cloud, Amazon SageMaker, and TRL. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Vision Language Models Explained DevFeed: [Vision Language Models Explained]() Original publisher: [Read original article]() Author: merve; Edward Beeching Published: 2024-04-11T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [multimodal](), [hugging face](), [trl](), [Inference](), [Machine learning]() Tags: [blog-post](), [guide](), [hallucinations](), [hugging-face](), [inference](), [leaderboard](), [models](), [multimodal](), [trl](), [vision](), [vlm](), [zero-shot]() ### AI overview An introduction to vision language models that explains their architecture, capabilities, use cases, model selection, inference, and fine-tuning with trl. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.