# TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization

DevFeed: [TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization](<https://devfeed.tech/articles/tlrs-dpo-token-level-risk-scoring-for-better-direct-preference-optimization-125344.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11232>)

Author: Suxin Ji, Hungtao Wan

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [drop-in](<https://devfeed.tech/tags/drop-in.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [offline](<https://devfeed.tech/tags/offline.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reference](<https://devfeed.tech/tags/reference.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

TLRS-DPO adds token-level risk control to offline, reference-based DPO, addressing its sequence-level objective's inability to localize risky patterns within otherwise acceptable responses. It derives localized risk scores from policy-reference log-ratios on rejected responses, gradually increases their compensation during training, and selectively penalizes high-risk tokens rather than entire sequences. The authors report improved alignment and safety-relevant behavior with minimal impact on helpfulness versus strong DPO baselines; the method is described as a drop-in modification to standard DPO trainers.

## Source excerpt

arXiv:2611.11232v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is a simple and widely used alternative to RLHF for aligning large language models (LLMs), but its sequence-level objective cannot precisely locate or suppress the local patterns that make an otherwise acceptable response unsafe or undesirable. We propose TLRS-DPO (Token-Level Risk Scoring for Better Direct Preference Optimization), which augments DPO with fine-grained token-level control while remaining fully offline and reference-based. TLRS-DPO introduces a token-level robust compensation term that converts the policy-reference log-ratio on rejected responses into localized risk scores, a curriculum-style schedule that gradually increases the strength of this compensation during training, and a risk-adaptive screening mechanism that selectively penalizes only high-risk tokens instead of entire sequences. This design bridges sequence-level preference fitting and token-level behavior control, and can be implemented as a drop-in modification to standard DPO trainers. Empirically, TLRS-DPO improves alignment quality and safety-relevant behavior with minimal impact on helpfulness compared to strong DPO baselines.