# dpo Published articles for dpo. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Newer Models, Same Advantage DevFeed: [Newer Models, Same Advantage]() Original publisher: [Read original article]() Author: Erick Lachmann; Gabriel Pimenta de Freitas Cardoso; Francisco de Almeida Rocha Alves; Victor Gabriel Ferreira Barbosa Published: 2026-07-16T11:49:48Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Fine-tuning](), [Inference](), [Language models](), [Benchmark](), [datasets](), [model-deployment](), [Optimization]() Tags: [article](), [benchmark](), [cost](), [data](), [dpo](), [errors](), [fine-tuning](), [generative](), [inference](), [model](), [models](), [multimodal](), [ocr](), [open](), [optimization](), [production](), [technology](), [training]() ### AI overview DharmaOCR is presented as a Brazilian Portuguese OCR model that outperformed newer alternatives through domain specialization and targeted training. Its two-stage pipeline combines supervised fine-tuning on Portuguese-language documents with Direct Preference Optimization, improving extraction quality, stability, inference efficiency, and production reliability. ### Source excerpt Despite newer architectures, DharmaOCR outperformed Mistral OCR4 and Unlimited-OCR on Brazilian Portuguese through domain specialization and targeted training. This article presents the evidence and the mechanism behind that advantage. Three months ago, we published a paper on DharmaOCR and open-sourced one of the models. The objective was specific: optical character recognition engineered for Brazilian Portuguese. The training pipeline was built in two stages. ## Direct Preference Optimization Beyond Chatbots DevFeed: [Direct Preference Optimization Beyond Chatbots]() Original publisher: [Read original article]() Author: Erick Lachmann; Gabriel Pimenta de Freitas Cardoso; Francisco de Almeida Rocha Alves Published: 2026-06-03T12:55:11Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [dpo](), [vlm](), [Benchmark](), [Fine-tuning](), [hugging face](), [Open Source]() Tags: [benchmark](), [cost](), [dpo](), [efficiency](), [fine-tuning](), [hugging-face](), [model](), [ocr](), [open-source](), [vision]() ### AI overview The article presents Direct Preference Optimization (DPO) as a second training stage for reducing text degeneration in DharmaOCR, a structured OCR model. Applied after supervised fine-tuning, DPO reduced degeneration across every tested model family, with an average reduction of 59.4% and a best-case reduction of 87.6%. ### Source excerpt In April, we released DharmaOCR, our specialized structured OCR model (available on Hugging Face) along with a paper detailing the methodology behind it and a benchmark demonstrating its superior quality and cost efficiency. The paper benchmarked leading vision-language model families - both open-source and commercial - on a structured document extraction task: OCR on Brazilian Portuguese text. ## TRL v1.0: Post-Training Library Built to Move with the Field DevFeed: [TRL v1.0: Post-Training Library Built to Move with the Field]() Original publisher: [Read original article]() Author: Quentin Gallouédec; Steven Liu; Pedro Cuenca; Sergio Paniego Published: 2026-03-31T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [trl](), [post-training](), [dpo](), [grpo](), [rlvr](), [Artificial Intelligence](), [Code]() Tags: [announcement](), [artificial-intelligence](), [code](), [dpo](), [grpo](), [open-source](), [post-training](), [reinforcement-learning](), [rlvr](), [software](), [trl](), [v1]() ### AI overview TRL v1.0 presents a post-training library designed to remain useful as algorithms, models, and paradigms change. The article explains the evolution from PPO to DPO-style preference optimization and RLVR methods such as GRPO, emphasizing flexible abstractions and practical comparison over a supposedly permanent design. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## 20x Faster TRL Fine-tuning with RapidFire AI DevFeed: [20x Faster TRL Fine-tuning with RapidFire AI]() Original publisher: [Read original article]() Author: Kamran Bigdely; Arun Kumar; Quentin Gallouédec Published: 2025-11-21T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Fine-tuning](), [trl](), [Large Language Model](), [post-training](), [Multi-GPU](), [Orchestration](), [dashboards](), [GPU](), [data]() Tags: [dashboards](), [data](), [dpo](), [experimentation](), [fine-tuning](), [gpu](), [grpo](), [llm](), [llms](), [logs](), [metrics](), [multi-gpu](), [orchestration](), [post-training](), [rapidfireai](), [trl]() ### AI overview RapidFire AI accelerates LLM fine-tuning and post-training experimentation by running multiple TRL configurations concurrently, including on a single GPU. Its adaptive chunk-based scheduling, live metrics dashboard, multi-GPU orchestration, and interactive controls help teams compare configurations sooner, stop weak runs, and clone promising ones. The article cites internal benchmarks reporting approximately 16-24x higher experimentation throughput than sequential comparison. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## SyGra is a low-code/no-code framework for creating and transforming LLM and SLM datasets DevFeed: [SyGra is a low-code/no-code framework for creating and transforming LLM and SLM datasets]() Original publisher: [Read original article]() Author: Bidyapati Pradhan; Vipul Mittal; Amit Kumar Saha; Surajit Dasgupta Published: 2025-09-22T06:45:05Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [datasets](), [Large Language Model](), [Low code](), [dpo](), [Python]() Tags: [ai](), [datasets](), [dpo](), [llms](), [low-code](), [ollama](), [python](), [rag](), [tgi](), [vllm]() ### AI overview SyGra is presented as a low-code/no-code Python framework for creating, transforming, and aligning datasets for large and small language models. It supports use cases including Q&A generation, preference-pair creation, reasoning data, multilingual adaptation, quality filtering, and RAG-oriented datasets. ### Source excerpt You start with a simple dataset, but the model fails on advanced reasoning tasks. How do you generate more complex datasets to strengthen performance? You already have a knowledge base, but it's not in Q&A format. How can you transform it into a usable question-answering dataset? You've prepared a supervised fine-tuning (SFT) dataset. But now you want to align your model using Direct Preference Optimization (DPO). How can you generate preference pairs? ## Vision Language Model Alignment in TRL ⚡ DevFeed: [Vision Language Model Alignment in TRL ⚡]() Original publisher: [Read original article]() Author: Sergio Paniego; merve; Quentin Gallouédec; Kashif Rasul; Aritra Roy Gosthipaty Published: 2025-08-07T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [trl](), [vlm](), [multimodal](), [Fine-tuning]() Tags: [artificial-intelligence](), [dpo](), [fine-tuning](), [grpo](), [model](), [multimodal](), [trl](), [vision](), [vlm]() ### AI overview This article presents new and extended methods in TRL for aligning Vision Language Models with human preferences. It covers Mixed Preference Optimization, Group Relative Policy Optimization, Group Sequence Policy Optimization, Reinforce Leave One Out, and Online Direct Preference Optimization, along with native supervised fine-tuning support, training scripts, and demo notebooks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Privacy for the newly appointed (and already exasperated) DPO DevFeed: [Privacy for the newly appointed (and already exasperated) DPO]() Original publisher: [Read original article]() Published: 2025-06-27T16:09:35Z Content type: opinion Language: en Sources: [Latacora]() Topics: [dpo](), [data](), [trust]() Tags: [compliance](), [data-protection](), [dpo](), [gdpr](), [law](), [privacy](), [trust]() ### AI overview This article offers guidance for startup employees responsible for privacy and data protection, including newly appointed Data Protection Officers. It argues that transparency, privacy by design, and data minimization help businesses maintain compliance and build customer trust, while noting the extraterritorial reach of the EU GDPR and China's PIPL. ### Source excerpt Every other week, regulators around the world bombard their constituents with new data protection laws and acronyms. As the person who was just voluntold you're now responsible for privacy at your startup, in addition to all your other duties and without any additional resources, how can you possibly be expected to keep up--let alone contextualize that information to maintain compliance? Privacy, at its core, is an ethical issue, which means the solution to your privacy challenges is deceptively simple: do the right thing and be transparent with your customers. That's it. That's what everyone means when they say "privacy by design." ## Топовые работы на ICLR 2025 DevFeed: [Топовые работы на ICLR 2025]() Original publisher: [Read original article]() Author: redmadrobot (red\_mad\_robot) Published: 2025-05-20T17:20:00Z Content type: article Language: ru Sources: [Redmadrobot EN](), [Redmadrobot RU]() Topics: [Large Language Model](), [Language models](), [Machine Intelligence](), [Learning](), [Jailbreak]() Tags: [2025](), [ai](), [alignment](), [china](), [conference](), [dpo](), [iclr](), [language-models](), [llm](), [ml](), [safety](), [science](), [tag-fa773991fb10](), [university]() ### AI overview This Russian-language article reviews highly rated papers from ICLR 2025 on artificial intelligence and machine learning. It discusses deepening safety alignment to improve resistance to jailbreak attacks, findings about SFT and DPO fine-tuning dynamics, and AlphaEdit, a method for targeted knowledge editing in large language models. ### Source excerpt Аналитический центр red_mad_robot продолжает обозревать топовые технологические конференции. В этот раз подготовили для вас инсайты с прошедшей в Сингапуре International Conference on Learning Representations (ICLR), посвящённой искусственному интеллекту и машинному обучению. На ICLR 2025 из более 3 тыс. работ наивысшие оценки получили 36 статей, из которых три были отмечены как "outstanding papers". Разберём выдающиеся, достойные упоминания и получившие высокие оценки работы этого года. Читать далее ## Preference Optimization for Vision Language Models DevFeed: [Preference Optimization for Vision Language Models]() Original publisher: [Read original article]() Author: Quentin Gallouédec; Shengyi Costa Huang; merve; Kashif Rasul Published: 2024-07-10T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [Language models]() Tags: [data](), [dpo](), [fine-tuning](), [gpu](), [images](), [language-models](), [multimodal](), [optimization](), [rlhf](), [training](), [trl](), [vlm](), [vlms]() ### AI overview A tutorial on training vision-language models with TRL's direct preference optimization support, covering preference data formatting and memory considerations. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Putting RL back in RLHF DevFeed: [Putting RL back in RLHF]() Original publisher: [Read original article]() Author: Shengyi Costa Huang; Arash Ahmadian Published: 2024-06-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [rlhf](), [trl](), [Reinforcement learning](), [dpo](), [cohere]() Tags: [cohere](), [dpo](), [gpu](), [reinforcement-learning](), [research](), [rl](), [rlhf](), [trl]() ### AI overview This article introduces the RLOO Trainer in TRL, an online reinforcement learning algorithm for RLHF designed as a more accessible alternative to PPO. It explains that RLOO uses less GPU memory, converges faster, performs competitively with PPO, and outperforms offline methods such as DPO in the reported comparisons. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Predictive Human Preference: From Model Ranking to Model Routing DevFeed: [Predictive Human Preference: From Model Ranking to Model Routing]() Original publisher: [Read original article]() Author: Chip Huyen Published: 2024-02-28T00:00:00Z Content type: article Language: en Sources: [Chip Huyen]() Topics: [Model Routing](), [AI Models](), [Model Development](), [post-training](), [AI Chat]() Tags: [ai-models](), [anthropic](), [chatgpt](), [dpo](), [gemini](), [latency](), [model](), [model-routing](), [ranking](), [rlhf]() ### AI overview The article examines predictive human preference: predicting which AI model users will prefer for a specific prompt. It describes model routing as a use case, where prompts could be directed to a preferred model to potentially reduce cost and latency, and discusses using preference predictions to interpret model strengths and weaknesses. It also outlines evaluating predictions against Chatbot Arena and building a preference predictor. ### Source excerpt A challenge of building AI applications is choosing which model to use. What if we don't have to? What if we can predict the best model for any prompt? Predictive human preference aims to predict which model users might prefer for a specific query. Human preference has emerged to be both the Northstar and a powerful tool for AI model development. Human preference guides post-training techniques including RLHF and DPO. Human preference is also used to rank AI models, as used by LMSYS's Chatbot Arena. Chatbot Arena aims to determine which model is generally preferred. I wanted to see if it's possible to predict which model is preferred for each query. One use case of predictive human preference is model routing. For example, if we know in advance that for a prompt, users will prefer Claude Instant's response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency. Another use case of predictive human preference is interpretability. Mapping out a model's performance on different prompts can help us understand this model's strengths and weaknesses. See section Experiment results for examples. Here's what predictive human preference for different model pairs looks like for the prompt "What's the best way to cluster text embeddings?". The predictions were generated by my toy preference predictor. The bright yellow color for the (GPT-4, GPT-3.5-Turbo) cell means that my predictor thinks GPT-4's response is very likely to be preferred to that of GPT-3.5-Turbo's for this prompt. This post first discusses the correctness of Chatbot Arena, which will then be used as a baseline to evaluate the correctness of preference predictions. It then discusses how to build a preference predictor and the initial results. Ranking Models Using Human Preference Using preferential signals (comparisons) to rank models has grown in popularity in the last