# trl

TRL is a full-stack library for training transformer language models with reinforcement-learning and related methods.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## D-REAL: Acceptance-Length Optimization with Unbiased Gradients for Speculative Decoding

DevFeed: [D-REAL: Acceptance-Length Optimization with Unbiased Gradients for Speculative Decoding](<https://devfeed.tech/articles/d-real-acceptance-length-optimization-with-unbiased-gradients-for-speculative-decoding-125942.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10432>)

Author: Meng Wei, Yefei Chen, Antong Li, Yanan Gao, Xingliang Shi, Xin Gao, Zhiling Ye, Yun Yue, Chunxiao Guo, Peng Wei

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>)

Tags: [acceptance](<https://devfeed.tech/tags/acceptance.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [draft](<https://devfeed.tech/tags/draft.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [model](<https://devfeed.tech/tags/model.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [serving](<https://devfeed.tech/tags/serving.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>)

### AI overview

D-REAL directly trains a speculative decoding drafter to increase expected acceptance length, which token-level objectives do not directly optimize. It weights draft-target distributional overlap at each position by the probability that earlier draft tokens survive verification, and derives an unbiased gradient estimator that accounts for the proposal distribution's parameter dependence under stochastic verification. In tests with DSpark and DFlash2 using Qwen3-8B across ten benchmarks, one post-training epoch improved benchmark-averaged mean acceptance length by up to 0.23 tokens under both greedy and stochastic sampling, from baselines where that metric had plateaued. The reported end-to-end serving throughput gain was up to 9.4%.

### Source excerpt

arXiv:2611.10432v1 Announce Type: new Abstract: Speculative decoding efficiency depends on the number of consecutively accepted draft tokens, yet token-level training objectives do not directly optimize this quantity. We introduce D-REAL (Drafter Reinforcement of Expected Acceptance Length), a post-training method that directly optimizes expected acceptance length for a fixed decoding context. D-REAL weights position-wise draft-target distributional overlap by prefix survival probabilities along a draft-sampled prefix, faithfully capturing the sequential dependence of verification on earlier proposals. By accounting for the parameter dependence of the proposal distribution, we derive an unbiased gradient estimator for this single-round objective under stochastic verification. We evaluate D-REAL on DSpark and DFlash2 with Qwen3-8B as the target model across ten benchmarks. Starting from baselines whose mean acceptance length has plateaued, one epoch of D-REAL post-training consistently improves the benchmark-averaged mean acceptance length by up to 0.23 tokens across both architectures under greedy (T = 0.0) and stochastic (T = 1.0) sampling, translating into up to a 9.4% end-to-end serving throughput improvement.

## How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

DevFeed: [How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?](<https://devfeed.tech/articles/how-much-training-inference-mismatch-can-on-policy-distillation-tolerate-125381.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11450>)

Author: Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai, Youhe Jiang, Eiko Yoneki, Tuo Shi

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [trl](<https://devfeed.tech/topics/trl.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [admission](<https://devfeed.tech/tags/admission.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [budget](<https://devfeed.tech/tags/budget.md>), [but](<https://devfeed.tech/tags/but.md>), [cache](<https://devfeed.tech/tags/cache.md>), [can](<https://devfeed.tech/tags/can.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [code](<https://devfeed.tech/tags/code.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [policy](<https://devfeed.tech/tags/policy.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

On-policy distillation (OPD) can tolerate substantial mismatch between the rollout engine and the student model without meaningful training-quality loss, but higher mismatch degrades quality. The study measures quantization across three student-teacher pairs and weight staleness and injected noise on one pair. Mismatch effects and quantization costs vary by model, and some layers are more sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout settings that destabilize reinforcement learning; weight noise can also change the generation prefixes, creating errors that per-token corrections cannot fix. The authors propose a lightweight admission check against a calibrated mismatch budget. Their model-specific FP8 recipes retain training quality while supporting at least 1.93x as many KV-cache tokens as BF16; FP8 with asynchronous rollouts reaches the same quality 1.62x sooner than synchronous BF16.

### Source excerpt

arXiv:2611.11450v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least $1.93\times$ as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality $1.62\times$ sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd

## Extending FunctionGemma for Practical On-Device Mobile Function Calling

DevFeed: [Extending FunctionGemma for Practical On-Device Mobile Function Calling](<https://devfeed.tech/articles/extending-functiongemma-for-practical-on-device-mobile-function-calling-96627.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.25373>)

Author: Ali Rezagholizadeh, Soheila Samiee

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai-on-mobile](<https://devfeed.tech/topics/ai-on-mobile.md>), [function calling](<https://devfeed.tech/topics/function-calling.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Android](<https://devfeed.tech/topics/android.md>), [trl](<https://devfeed.tech/topics/trl.md>), [responsible-ai](<https://devfeed.tech/topics/responsible-ai.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [android](<https://devfeed.tech/tags/android.md>), [application-management](<https://devfeed.tech/tags/application-management.md>), [camera](<https://devfeed.tech/tags/camera.md>), [combined](<https://devfeed.tech/tags/combined.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [demo](<https://devfeed.tech/tags/demo.md>), [device](<https://devfeed.tech/tags/device.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flashlight](<https://devfeed.tech/tags/flashlight.md>), [function](<https://devfeed.tech/tags/function.md>), [function-calling](<https://devfeed.tech/tags/function-calling.md>), [google](<https://devfeed.tech/tags/google.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [on-device](<https://devfeed.tech/tags/on-device.md>)

### AI overview

The authors extend FunctionGemma 270M-it for Android actions with MOBILEACTIONSEXTENDED, a synthetic, schema-validated dataset of about 9,500 conversations across 15 device-control categories. Using TRL supervised fine-tuning with completion-only loss, their specialist model reaches 76.5% end-to-end accuracy on the new dataset, compared with 29.3% for the base model and 17.2% for Google's Mobile-Actions variant. A model jointly trained on the new dataset and MOBILEACTIONSGOOGLE retains 76.5% on the extended dataset and scores 82.3% on Google's dataset; Google's specialist scores 90.3% there. The authors describe this 8-point drop as a trade-off for doubling category coverage. They report releasing the dataset, models, training and evaluation pipeline, and an Android demo.

### Source excerpt

arXiv:2609.25373v2 Announce Type: replace-cross Abstract: On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or narrow mobile-action catalogs. We extend FunctionGemma 270M-it to practical Android workflows by introducing MOBILEACTIONSEXTENDED, a synthetic, schema-validated dataset of ~9,500 conversations covering fifteen device-control categories, including messaging, phone calls, camera/screenshot, brightness control, device-status queries, flashlight control, and application management. We fine-tune the 270M model with TRL supervised fine-tuning under completion-only loss, producing an extended specialist and a combined model trained jointly with Google's MOBILEACTIONSGOOGLE. On MOBILEACTIONSEXTENDED, end-to-end accuracy improves from 29.3% for the base model and 17.2% for Google's Mobile-Actions variant to 76.5%. The combined model retains 76.5% on MOBILEACTIONSEXTENDED and reaches 82.3% on MOBILEACTIONSGOOGLE, down from the 90.3% of Google's Mobile-Actions specialist, representing an 8.0-percentage-point trade-off in return for doubling category coverage. We release the dataset, fine-tuned models, reproducible training/evaluation pipeline, and an Android demo, highlighting compact local function calling as a practical path towards low-latency and privacy-preserving mobile assistants.

## TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL

DevFeed: [TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL](<https://devfeed.tech/articles/torchtitan-update-new-sota-models-performance-at-scale-titanrl-93882.md>)

Original publisher: [Read original article](<https://pytorch.org/blog/torchtitan-update-new-sota-models-performance-at-scale-titanrl/>)

Author: The PyTorch Team at Meta

Published: 2026-10-21T19:45:44Z

Content type: release

Language: en

Sources: [PyTorch](<https://devfeed.tech/sources/blog-pytorch.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [trl](<https://devfeed.tech/topics/trl.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-hardware](<https://devfeed.tech/tags/ai-hardware.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [blog](<https://devfeed.tech/tags/blog.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [models](<https://devfeed.tech/tags/models.md>), [native](<https://devfeed.tech/tags/native.md>), [performance](<https://devfeed.tech/tags/performance.md>), [platform](<https://devfeed.tech/tags/platform.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [scale](<https://devfeed.tech/tags/scale.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [training](<https://devfeed.tech/tags/training.md>), [update](<https://devfeed.tech/tags/update.md>)

### AI overview

TorchTitan adds native training support for several newer open-weight models across its 5D parallelism stack, with typed recipes and component overrides for adapting configurations and kernels. Its TitanRL keeps a shared model across pretraining, post-training, and vLLM inference; asynchronous rollouts support agentic training, while batch-invariant mode provides bitwise-identical trainer and generator log probabilities in supported on-policy configurations. Performance work includes selective `torch.compile` regions, fine-grained activation rematerialization, fused MoE communication and compute, and a revamped pipeline-parallel package. FlexShard adds placements such as `Owned` and `BlockShard`; reported results include up to 79% lower optimizer time for DeepSeekV3 on 256 H100s and up to 20% faster end-to-end steps for Llama3 405B on 256 H100s. The update also describes graph-mode training, a new opt-in NCCL backend, symmetric-memory support, and `torch_checkpointing`, which avoids collectives after the first save. Numerical, performance, and scale claims are reported for specific configurations in the supplied text.

### Source excerpt

TorchTitan is a PyTorch-native platform for rapid experimentation and large-scale pre- and post-training of generative AI models. It is designed as a minimal, clean-room implementation of PyTorch-native techniques, with flexible... The post TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL appeared first on PyTorch.

## Fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod

DevFeed: [Fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod](<https://devfeed.tech/articles/fine-tuning-gpt-oss-20b-for-multilingual-reasoning-on-amazon-sagemaker-hyperpod-91293.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/fine-tuning-gpt-oss-20b-for-multilingual-reasoning-on-amazon-sagemaker-hyperpod/>)

Author: Mijanur Palash

Published: 2026-10-20T17:48:27Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Amazon SageMaker HyperPod](<https://devfeed.tech/topics/amazon-sagemaker-hyperpod.md>), [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [trl](<https://devfeed.tech/topics/trl.md>), [peft](<https://devfeed.tech/topics/peft.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [amazon-elastic-container-registry](<https://devfeed.tech/tags/amazon-elastic-container-registry.md>), [amazon-elastic-kubernetes-service](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service.md>), [amazon-elastic-kubernetes-service-amazon-eks](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service-amazon-eks.md>), [amazon-fsx](<https://devfeed.tech/tags/amazon-fsx.md>), [amazon-fsx-for-lustre](<https://devfeed.tech/tags/amazon-fsx-for-lustre.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-hyperpod](<https://devfeed.tech/tags/amazon-sagemaker-hyperpod.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpt-oss](<https://devfeed.tech/tags/gpt-oss.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [headroom](<https://devfeed.tech/tags/headroom.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [sft](<https://devfeed.tech/tags/sft.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Fine-tuning GPT-OSS-20B with LoRA on Amazon SageMaker HyperPod raised reasoning in the requested language from 22% to 100% on a 50-case, five-language evaluation. SFT taught the missing behavior; final-answer language accuracy reached 62%. GRPO on the SFT checkpoint raised that metric to 70% while maintaining 100% reasoning-language accuracy. A Qwen2.5-7B control showed GRPO added 6 points on multilingual reasoning where there was headroom, but no gain on GSM8K, already at 90.5%. The results support using SFT for behaviors the model does not yet exhibit and adding GRPO to refine behaviors it sometimes produces; when sampled groups are already mostly correct, GRPO may have no useful learning signal. The article describes HyperPod training infrastructure and configuration, including TRL for GPT-OSS-20B and veRL with Ray and vLLM for the Qwen control.

### Source excerpt

Learn how we fine-tuned GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod, and how to decide between supervised fine-tuning (SFT) and reinforcement learning with GRPO. SFT took reasoning-language accuracy from 22% to 100%, and GRPO added measurable gains where headroom existed.

## TorchNeuron: running HF models on AWS Trainium is easier than ever

DevFeed: [TorchNeuron: running HF models on AWS Trainium is easier than ever](<https://devfeed.tech/articles/torchneuron-running-hf-models-on-aws-trainium-is-easier-than-ever-90237.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/hf-torchneuron>)

Author: Régis Pierrard; Simon Pagezy; Michael Benayoun; Jingya Huang; Alvaro Moran; David Corvoysier; Jeff Boudier

Published: 2026-10-19T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AWS Trainium](<https://devfeed.tech/topics/aws-trainium.md>), [AWS AI chips](<https://devfeed.tech/topics/aws-ai-chips.md>), [neuron](<https://devfeed.tech/topics/neuron.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [NeMo](<https://devfeed.tech/topics/nemo.md>), [optimum](<https://devfeed.tech/topics/optimum.md>), [peft](<https://devfeed.tech/topics/peft.md>), [trl](<https://devfeed.tech/topics/trl.md>), [lunar foundation model](<https://devfeed.tech/topics/lunar-foundation-model.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [ai-chips](<https://devfeed.tech/tags/ai-chips.md>), [apis](<https://devfeed.tech/tags/apis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [async](<https://devfeed.tech/tags/async.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-trainium](<https://devfeed.tech/tags/aws-trainium.md>), [code](<https://devfeed.tech/tags/code.md>), [cost](<https://devfeed.tech/tags/cost.md>), [diffusers](<https://devfeed.tech/tags/diffusers.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [eos](<https://devfeed.tech/tags/eos.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [torch](<https://devfeed.tech/tags/torch.md>)

### AI overview

TorchNeuron adds a native PyTorch backend for AWS Trainium, supporting eager execution, torch.compile, and distributed APIs. Hugging Face describes how its libraries can use the backend for model training and inference, with examples covering text generation, fine-tuning, and image generation. The article says TorchNeuron is planned for public beta starting October 20, 2026.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## TRL v1.0: Post-Training Library Built to Move with the Field

DevFeed: [TRL v1.0: Post-Training Library Built to Move with the Field](<https://devfeed.tech/articles/trl-v1-0-post-training-library-built-to-move-with-the-field-7541.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/trl-v1>)

Author: Quentin Gallouédec; Steven Liu; Pedro Cuenca; Sergio Paniego

Published: 2026-03-31T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [trl](<https://devfeed.tech/topics/trl.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [code](<https://devfeed.tech/tags/code.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [software](<https://devfeed.tech/tags/software.md>), [trl](<https://devfeed.tech/tags/trl.md>), [v1](<https://devfeed.tech/tags/v1.md>)

### AI overview

TRL v1.0 presents a post-training library designed to remain useful as algorithms, models, and paradigms change. The article explains the evolution from PPO to DPO-style preference optimization and RLVR methods such as GRPO, emphasizing flexible abstractions and practical comparison over a supposedly permanent design.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## DeepMath: A lightweight math reasoning Agent with smolagents

DevFeed: [DeepMath: A lightweight math reasoning Agent with smolagents](<https://devfeed.tech/articles/deepmath-a-lightweight-math-reasoning-agent-with-smolagents-7288.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/intel-deepmath>)

Author: Daniel Fleischer; Moshe Berchansky; Moshe Wasserblat

Published: 2025-12-04T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [smolagents](<https://devfeed.tech/topics/smolagents.md>), [Mathematics](<https://devfeed.tech/topics/mathematics.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Python](<https://devfeed.tech/topics/python.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [trl](<https://devfeed.tech/topics/trl.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [math](<https://devfeed.tech/tags/math.md>), [math-reasoning](<https://devfeed.tech/tags/math-reasoning.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [sandbox](<https://devfeed.tech/tags/sandbox.md>), [smolagents](<https://devfeed.tech/tags/smolagents.md>), [trl](<https://devfeed.tech/tags/trl.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

DeepMath is a lightweight math reasoning agent built on Qwen3-4B Thinking and implemented with smolagents. It uses GRPO fine-tuning to encourage concise Python snippets for deterministic computation, executes those snippets in a constrained sandbox, and reintegrates the results into the reasoning process. Evaluation on MATH500, AIME, HMMT, and HLE reports shorter outputs and often improved accuracy.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## 20x Faster TRL Fine-tuning with RapidFire AI

DevFeed: [20x Faster TRL Fine-tuning with RapidFire AI](<https://devfeed.tech/articles/20x-faster-trl-fine-tuning-with-rapidfire-ai-7452.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/rapidfireai>)

Author: Kamran Bigdely; Arun Kumar; Quentin Gallouédec

Published: 2025-11-21T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rapidfireai](<https://devfeed.tech/tags/rapidfireai.md>), [trl](<https://devfeed.tech/tags/trl.md>)

### AI overview

RapidFire AI accelerates LLM fine-tuning and post-training experimentation by running multiple TRL configurations concurrently, including on a single GPU. Its adaptive chunk-based scheduling, live metrics dashboard, multi-GPU orchestration, and interactive controls help teams compare configurations sooner, stop weak runs, and clone promising ones. The article cites internal benchmarks reporting approximately 16-24x higher experimentation throughput than sequential comparison.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Vision Language Model Alignment in TRL ⚡

DevFeed: [Vision Language Model Alignment in TRL ⚡](<https://devfeed.tech/articles/vision-language-model-alignment-in-trl-7542.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/trl-vlm-alignment>)

Author: Sergio Paniego; merve; Quentin Gallouédec; Kashif Rasul; Aritra Roy Gosthipaty

Published: 2025-08-07T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [trl](<https://devfeed.tech/topics/trl.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [model](<https://devfeed.tech/tags/model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [trl](<https://devfeed.tech/tags/trl.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

This article presents new and extended methods in TRL for aligning Vision Language Models with human preferences. It covers Mixed Preference Optimization, Group Relative Policy Optimization, Group Sequence Policy Optimization, Reinforce Leave One Out, and Online Direct Preference Optimization, along with native supervised fine-tuning support, training scripts, and demo notebooks.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

DevFeed: [No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL](<https://devfeed.tech/articles/no-gpu-left-behind-unlocking-efficiency-with-co-located-vllm-in-trl-7558.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/vllm-colocate>)

Author: Mert Toslali; Yu Chin Fabian Lim; Quentin Gallouédec; Ed Snible; Raghu Ganti; Mudhakar Srivatsa

Published: 2025-06-03T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [trl](<https://devfeed.tech/topics/trl.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [efficiency](<https://devfeed.tech/tags/efficiency.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [train](<https://devfeed.tech/tags/train.md>), [training](<https://devfeed.tech/tags/training.md>), [trl](<https://devfeed.tech/tags/trl.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This article explains how TRL integrates vLLM to accelerate GRPO training of LLMs. It describes the GPU inefficiencies caused by running training and generation on separate devices and introduces colocated vLLM, which allows both tasks to share GPUs within the same distributed process group.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## 🐯 Liger GRPO meets TRL

DevFeed: [🐯 Liger GRPO meets TRL](<https://devfeed.tech/articles/liger-grpo-meets-trl-7332.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/liger-grpo>)

Author: Shivam Sahni; Kashif Rasul; Salman Mohammadi; Shirin Yamani; Yanning Chen; Liberty

Published: 2025-05-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Mathematics](<https://devfeed.tech/topics/mathematics.md>)

Tags: [coding](<https://devfeed.tech/tags/coding.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [liger](<https://devfeed.tech/tags/liger.md>), [llm](<https://devfeed.tech/tags/llm.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [partnership](<https://devfeed.tech/tags/partnership.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [trl](<https://devfeed.tech/tags/trl.md>)

### AI overview

This article explains how Group Relative Policy Optimization (GRPO) can reduce the resource requirements of reinforcement learning fine-tuning for language models. It presents a TRL optimization based on chunked GRPO loss that reduces peak memory usage by 40% and discusses scaling GRPO across multiple GPUs and nodes while preserving performance and correctness.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Open-R1: Update #1

DevFeed: [Open-R1: Update #1](<https://devfeed.tech/articles/open-r1-update-1-7420.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/open-r1/update-1>)

Author: Leandro von Werra; Lewis Tunstall; Quentin Gallouédec; Guilherme Penedo; Edward Beeching; Anton Lozhkov; Brigitte Tousignant; Daniel van Strien

Published: 2025-02-02T00:04:28Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [deepseek](<https://devfeed.tech/topics/deepseek.md>), [synthetic-data](<https://devfeed.tech/topics/synthetic-data.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [follow](<https://devfeed.tech/tags/follow.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [trl](<https://devfeed.tech/tags/trl.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This update reports early progress on the open-r1 project to reproduce the DeepSeek-R1 training pipeline and dataset. It covers matching MATH-500 evaluation results, the unusually long model responses and their GPU-memory implications, a public evaluation leaderboard, and the integration of GRPO into TRL with DeepSpeed and vLLM support.

### Source excerpt

It's been two weeks since the release of DeepSeek R1 and just a week since we started the open-r1 project to replicate the missing pieces, namely the training pipeline and the synthetic data. This post summarizes: - the progress of Open-R1 to replicate the DeepSeek-R1 pipeline and dataset - what we learned about DeepSeek-R1 and discussions around it - cool projects the community has built since the release of DeepSeek-R1 It should serve both as an update on the project and as a collection of...

## Putting RL back in RLHF

DevFeed: [Putting RL back in RLHF](<https://devfeed.tech/articles/putting-rl-back-in-rlhf-7446.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/putting_rl_back_in_rlhf_with_rloo>)

Author: Shengyi Costa Huang; Arash Ahmadian

Published: 2024-06-12T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [cohere](<https://devfeed.tech/topics/cohere.md>)

Tags: [cohere](<https://devfeed.tech/tags/cohere.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [rl](<https://devfeed.tech/tags/rl.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [trl](<https://devfeed.tech/tags/trl.md>)

### AI overview

This article introduces the RLOO Trainer in TRL, an online reinforcement learning algorithm for RLHF designed as a more accessible alternative to PPO. It explains that RLOO uses less GPU memory, converges faster, performs competitively with PPO, and outperforms offline methods such as DPO in the reported comparisons.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Welcome Llama 3 - Meta's new open LLM

DevFeed: [Welcome Llama 3 - Meta's new open LLM](<https://devfeed.tech/articles/welcome-llama-3-meta-s-new-open-llm-7334.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/llama3>)

Author: Philipp Schmid; Omar Sanseviero; Pedro Cuenca; Younes B; Leandro von Werra

Published: 2024-04-18T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [llama](<https://devfeed.tech/topics/llama.md>), [Meta](<https://devfeed.tech/topics/meta.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Google AI](<https://devfeed.tech/topics/google-ai.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [community](<https://devfeed.tech/tags/community.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [meta](<https://devfeed.tech/tags/meta.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [research](<https://devfeed.tech/tags/research.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [trl](<https://devfeed.tech/tags/trl.md>)

### AI overview

Meta has released Llama 3, an open-access LLM family available through Hugging Face. The release includes 8B and 70B base and instruction-tuned models, plus Llama Guard 2 for classifying potentially unsafe LLM inputs and responses. It also adds integrations with Transformers, Hugging Chat, Inference Endpoints, Google Cloud, Amazon SageMaker, and TRL.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Vision Language Models Explained

DevFeed: [Vision Language Models Explained](<https://devfeed.tech/articles/vision-language-models-explained-7560.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/vlms>)

Author: merve; Edward Beeching

Published: 2024-04-11T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [trl](<https://devfeed.tech/topics/trl.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [blog-post](<https://devfeed.tech/tags/blog-post.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [trl](<https://devfeed.tech/tags/trl.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vlm](<https://devfeed.tech/tags/vlm.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

An introduction to vision language models that explains their architecture, capabilities, use cases, model selection, inference, and fine-tuning with trl.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.