TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Truong Nguyen * 1 Tien-Phat Nguyen * 1 Linh Ngo Van 1 Duy Minh Ho Nguyen 2 Khoa Doan 3 Trung Le 4
arXiv:2605.12288v1 [cs.CL] 12 May 2026
Abstract
tion (DPO) (Rafailov et al., 2023) avoids the engineering complexity and variance of RL-style updates while still producing visibly improved behavior. However, a fundamental mismatch remains: language models generate text by making a sequence of local decisions, one token at a time conditioned on a prefix, while most preference optimization objectives treat each completion as a single indivisible object. Although this sequence-level view is convenient, it strains the connection between where a model is optimized (a whole response) and how it actually behaves at inference time (a distribution over next tokens at every state).
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley–Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a tokenlevel Bradley–Terry preference model over nexttoken actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
A natural response to the mismatch between autoregressive generation and sequence-level preference objectives is to push preference optimization to token granularity. However, existing “token-level” methods (e.g., TDPO (Zeng et al., 2024) and TISDPO (Liu et al., 2025a)) still fundamentally optimize a sequence-level Bradley–Terry preference model, differing mainly in how the sequence-level signal is distributed across timesteps. As a result, token-wise optimality is not explicitly enforced but remains implicit. This limitation becomes increasingly problematic for long generations: preference supervision is only observed at the level of full responses, making credit assignment to individual tokens indirect and allowing small early-token errors to cascade through subsequent decisions, even as the sequence-level objective improves. What we ultimately want, but rarely formalize, is a policy that is preference-optimal at every state, i.e., one that makes locally optimal tradeoffs at each state before the remainder of the completion unfolds. Achieving this stronger notion of optimality without sacrificing the simplicity and efficiency that make DPO attractive is therefore both challenging and valuable.
1. Introduction
This paper proposes Token-level Bregman Preference Optimization (TBPO), a preference-optimization framework that explicitly models preferences at the token level via density ratio matching while relying solely on sequence-level comparison data. TBPO is built on a token-level Bradley– Terry preference model that aligns preference learning with the autoregressive decision process of language models, and is designed to recover a token-wise optimal policy instead of implicitly relying on sequence-level credit assignment distributed across timesteps. Depending on the choice of
Aligning large language models with human preferences is a central goal of modern fine-tuning, and recent work has increasingly favored lightweight and stable objectives for this purpose. In particular, Direct Preference Optimiza* Equal contribution 1 Hanoi University of Science and Technology 2 Max Planck Research School for Intelligent Systems 3 VinUniversity 4 Monash University. Correspondence to: Trung Le <[email protected]>.
Preprint. May 13, 2026.
1
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
scoring function in the token-level Bradley–Terry model, TBPO naturally yields different variants. We introduce two instantiations: TBPO-Q, which defines the token-level score via a state–action value function, and TBPO-A, which defines it via an advantage function. By casting TBPO as density-ratio matching under a Bregman-divergence objective, we even obtain a plug-and-play family of pairwise preference losses: small tweaks to the Bregman generator yield different objectives. Crucially, this flexibility does not come at the cost of theory or usability: the resulting objective still supports an optimality characterization in which the minimizer corresponds to the desired optimal token-level policy, while retaining DPO-style simplicity without fitting an explicit reward model or running on-policy RL.
rize this rapidly expanding design space (Xiao et al., 2024; Liu et al., 2025b). Token-level methods. To better regulate optimization dynamics in autoregressive models, several token-level variants of DPO have been proposed. TDPO decomposes the sequence-level DPO objective into token-wise terms with per-token KL constraints, while TIS-DPO further incorporates token-level importance sampling to improve training stability (Zeng et al., 2024; Liu et al., 2025a). However, both methods still assume a sequence-level Bradley–Terry preference model, redistributing sequence-level supervision across timesteps rather than explicitly defining token-level preferences. As a result, per-state optimality remains implicit rather than being directly enforced.
We summarize our contributions as follows:
Density ratio matching. Another line of work interprets preference optimization through density ratio matching. Bregman Preference Optimization (BPO) formalizes this view by framing preference learning as likelihood-ratio estimation under Bregman divergences, connecting DPO to classical density-ratio estimators (e.g., logistic regression (Gutmann & Hyvärinen, 2012), KLIEP (Sugiyama et al., 2007), LSIF (Huang et al., 2006)) and showing that DPO is a special case of this framework (Kim et al., 2025). BPO further introduces a scaled Basu’s power divergence to interpolate between KLIEP and LSIF while modulating gradient reweighting during optimization. However, like DPO, BPO remains sequence-level and does not explicitly model token-level preferences.
• Token-level Bregman Preference Optimization (TBPO). We propose TBPO, a token-level preferenceoptimization framework that models preferences directly at the token level and learns from sequence-level comparisons. • Theory. We provide a rigorous derivation and an optimality characterization: under the token-level preference model, minimizing the TBPO objective recovers a token-level optimal policy. • Empirical results. Across benchmarks, model backbones, and tasks, TBPO improves alignment while also enhancing training stability and preserving response diversity relative to strong baselines.
Our contribution. Motivated by these gaps, we study token-level preference modeling. By defining preferences via a token-level Bradley–Terry model and adopting a ratiomatching perspective, we learn preference-optimal decisions at each prefix using only sequence-level comparison data.
2. Related Work Preference-based alignment. Aligning large language models (LLMs) with human judgments is commonly approached via preference-based optimization. RLHF learns an explicit reward model from pairwise comparisons and optimizes the policy via reinforcement learning, but suffers from instability and high computational cost (Christiano et al., 2017; Ouyang et al., 2022). DPO provides an RL-free alternative by deriving a closed-form relationship between rewards and optimal policies under a KL-regularized objective, yielding a supervised loss equivalent to fitting a Bradley–Terry model over policy likelihood ratios (Rafailov et al., 2023). Owing to its simplicity and stability, DPO has become a standard alignment baseline.
3. Background 3.1. Preliminary and Notions When viewing text generation as a Markov decision process (Puterman, 1994), we define the state at step t as the prompt together with the response prefix produced so far, i.e., st = [x, y<t ]. The action is the next token to generate, at = yt , and the per-token reward is given by Rt := R(st , at ) = R([x, y<t ], yt ). Using these definitions, for a policy π we define the state-action value function Qπ , the state value function V π and the advantage function Aπ as: π
DPO variants. Numerous extensions modify DPO through alternative feedback forms, regularization schemes, or reference models (Azar et al., 2024; Ethayarajh et al., 2024; Xu et al., 2024; Hong et al., 2024; Meng et al., 2024; Wu et al., 2024). While improving optimization dynamics or robustness, these methods largely preserve the sequence-level preference modeling assumption. Recent surveys summa-
Q ([x, y<t ], yt ) = Eπ
"∞ X
# k
γ Rt+k | st = [x, y<t ], at = yt ,
k=0 π
V π ([x, y<t ]) = Eπ [Q ([x, y<t ], yt ) | st = [x, y<t ]] , A ([x, y<t ], yt ) = Qπ ([x, y<t ], yt ) − V π ([x, y<t ]). π
where γ denotes the discount factor. We set γ = 1 throughout this work. 2
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Following TDPO (Zeng et al., 2024), we cast next-token generation as a sequential decision problem and optimize a KL-regularized objective at the token level: h max Ex,y<t ∼D, z∼πθ (·|[x,y<t ]) Aπref ([x, y<t ], z) πθ i −β DKL πθ (· | [x, y<t ])∥πref (· | [x, y<t ]) .
a unified framework based on Bregman divergence (Bregman, 1967). This unification leads to the following general objective: Z Dh (Rdata (x)∥Rθ (x)) = pde (x) Bh (Rdata (x)∥Rθ (x)) dx.
(1) which can be expanded as:
Intuitively, each prefix state [x, y<t ] defines a local decision over the next token. The advantage term encourages increasing the probability of tokens that are estimated (under the reference) to be locally better, while the KL penalty keeps the update conservative by anchoring πθ to πref . This is aligned with our goal of token-level preference optimality: the optimization objective explicitly targets preferenceimproving decisions at the state–action level, rather than only redistributing a sequence-level preference signal across timesteps. We later prove (Theorem 4.3) that, under sufficient model capacity, our token-level density-ratio matching objective recovers the optimal policy for Eq. 1.
Z
pde (x) h(Rdata (x)) − h(Rθ (x)) − h′ (Rθ (x)) Rdata (x) − Rθ (x) dx. (4) Dh (Rdata (x)∥Rθ (x)) =
where h represents a strictly convex and twice continuously differentiable function whose derivative is denoted by h′ . The term Bh corresponds to the pointwise Bregman divergence, which quantifies the approximation error of the first-order Taylor expansion.
4. Methodology
Using the objective in Eq. (1), TDPO (Zeng et al., 2024) derives closed-form expressions for the optimal state-action Q and advantage functions under the reference policy.
A central modeling choice in preference optimization is the granularity at which preferences are defined. While standard DPO-style derivations posit a sequence-level Bradley–Terry model, generation and learning in LMs naturally operate at the token level. Our objective is therefore to formulate a token-level preference model, which enables density ratio matching at the token level and culminates in the framework Token-level Bregman Preference Optimization (TBPO).
Qπref ([x, y<t ], z) = β log
(2) πθ∗ (z | [x, y<t ]) + β log Z([x, y<t ]; β), πref (z | [x, y<t ])
π ∗ (z | [x, y<t ]) Aπref ([x, y<t ], z) = β log θ πref (z | [x, y<t ]) + βDKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) .
4.1. Token-level Bradley–Terry model For a preference triple (x, y w , y l ) and token step t, the two responses induce different prefixes
(3) 1
where Z([x, y<t ]; β) = Ez∼πref (·|[x,y<t ]) e β is the partition function.
Qπref ([x,y<t ],z)
w sw t = [x, y<t ],
l slt = [x, y<t ],
(5)
w l l and the compared actions are aw t = yt and at = yt . We posit a token-level Bradley-Terry likelihood l w l w w l l pdata (aw t ≻ at | st , st ) = σ S(st , at ) − S(st , at ) , (6)
3.2. Density Ratio Matching Consider two probability distributions pde (x) and pnu (x). The objective of likelihood ratio estimation is to construct a parametric model Rθ (x) that closely approximates (x) the true ratio Rdata (x) := ppnude (x) using independent and identically distributed samples drawn from each distribution. The predominant technique for this task is probabilistic classification through logistic regression (Gutmann & Hyvärinen, 2012). Beyond this, classical approaches including the Kullback-Leibler importance estimation procedure (KLIEP) (Sugiyama et al., 2007) and least-squares importance fitting (LSIF) (Huang et al., 2006) have gained substantial adoption in practice.
where S(s, a) is a token-level score and the sigmoid function σ(u) = 1/(1 + exp(−u)). Note that we call this preference probability as pdata because this is computed based on the preference data y w ≻ y l and we choose an appropriate score function to recover preference samw l := ples from data. Note that in DPO, pdata y ≻ p ∗ w ∗ l σ r (x, y ) − r x, y . The key idea is to lift a sequence-level preference label y w ≻ y l to supervision at each generation step. Conditioned on the prompt x and the partial responses generated so far, w l y<t and y<t , we prefer generating the next token from the winning trajectory, ytw , over the next token from the losing
A key insight from (Sugiyama et al., 2012) is that these diverse ratio estimation techniques can be understood within 3
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
trajectory, ytl . This provides a preference signal throughout w generation and encourages the extended prefixes y≤t and l y≤t to separate in quality, as measured by a token-level score S.
This likelihood assigns higher probability to the token with larger advantage. Optimizing Eq. (1) therefore encourages the learned policy to increase the advantage gap between w l the winning and losing extensions y≤t and y≤t , ensure generation quality at every state.
Different choices of the score function S yield different instantiations of the token-level Bradley–Terry model. We consider two natural options: the state-action value function Q and the advantage function A, giving two variants: (1) TBPO-Q and (2) TBPO-A.
Similarly, we can establish the following equality that links the optimal policy to the reference policy, thereby encouraging token-level density–ratio matching. Proposition 4.2. Let πθ∗ denote the optimal policy for the optimization problem in Eq. (1). We have the following equality:
4.2. TBPO-Q Here we instantiate the token-level Bradley–Terry model by using the reference-policy state–action value as the score, i.e., S(s, a) = Qπref (s, a): w l pdata (ytw ≻ ytl | x, y<t , y<t )
w l = σ Qπref ([x, y<t ], ytw ) − Qπref ([x, y<t ], ytl )
w w πref (ytw | [x, y<t ]) πθ∗ (ytw | [x, y<t ]) = l l l l πθ∗ yt | [x, y<t ] πref yt | [x, y<t ] 1 w l pdata (ytw ≻ ytl | x, y<t , y<t ) β × wt , w , yl ) pdata (ytw ≺ ytl | x, y<t <t (10)
(7)
Intuition. Recall that Qπref ([x, y<t ], yt ) is the expected cumulative reward obtained by emitting token yt at prefix [x, y<t ] and then rolling out πref for the remainder of the sequence. Eq. (7) therefore treats each preference label as a noisy comparison of two one-step lookahead utilities: taking ytw after the winning prefix versus taking ytl after the losing prefix. Equivalently, the Bradley–Terry log-odds are proportional to the difference in these expected returns, so maximizing this likelihood pushes probability mass toward tokens whose downstream continuations are expected to yield higher reward.
where we have defined per state weight l l wt = exp DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) w w − DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ])
.
See proof in Appendix A.2. 4.4. Interpretation and estimation of per state weights In both Proposition 4.1 (TBPO-Q) and Proposition 4.2 (TBPO-A), an extra multiplicative term wt appears in the policy ratio. This term is state-dependent but actionindependent: it does not depend on the compared next tow kens ytw , ytl directly, but only on the prefixes [x, y<t ] and l [x, y<t ]. Intuitively, wt corrects for the fact that we compare actions taken in two different states (different prefixes), so state-only baseline terms that would cancel in a same-state comparison no longer cancel.
We can establish the following equality that links the optimal policy to the reference policy, thereby later encouraging token-level density–ratio matching. Proposition 4.1. Let πθ∗ denote the optimal policy for the optimization problem in Eq. (1). We have the following equality: w w πθ∗ (ytw | [x, y<t ]) π (y w | [x, y<t ]) = ref tl l l l πθ∗ yt | [x, y<t ] πref yt | [x, y<t ] 1 w l pdata (ytw ≻ ytl | x, y<t , y<t ) β × wt , w , yl ) pdata (ytw ≺ ytl | x, y<t <t (8) l Z([x,y ];β) with per state weight wt = exp log Z([x,y<t w ];β) .
TBPO-Q: ratio of partition functions. The TBPO-Q weight is (Q) wt = exp
<t
See proof in Appendix A.1.
l Z([x, y<t ]; β) log w ]; β) . Z([x, y<t
therefore measures the relative continuation value of the losing prefix versus the winning prefix. If the winning prefix w is already a higher-value state (larger Z([x, y<t ]; β), then (Q) wt < 1, which downweights the token ratio update so that we do not over-attribute preference to the single nexttoken decision when the prefixes already differ substantially in quality.
4.3. TBPO-A In this variants, we define a token-level BT preference model in terms of the reference-policy advantage. w l pdata ytw ≻ ytl | x, y<t , y<t (9) w l = σ Aπref [x, y<t ], ytw − Aπref [x, y<t ], ytl
TBPO-A: difference of KL baselines. The TBPO-A 4
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
weight is (A)
wt
It is evident that we have the recursive formula: w l w l pdata y≤t ≻ y≤t | x = pdata ytw ≻ ytl | x, y<t , y<t w l pdata y≤t−1 ≻ y≤t−1 | x . (12)
l l = exp DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) w w − DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) ,
Preference optimization can be reformulated as a matching t problem between the empirical data ratio Rdata (ytw , ytl | t w l w l w l , x), y<t , y<t , x) and the model ratio Rθ (yt , yt | y<t , y<t where we have defined w l pdata (ytw ≺ ytl | x, y<t , y<t ) t Rdata = , w l w l pdata (yt ≻ yt | x, y<t , y<t ) !β l w πθ ytl | [x, y<t ] πref (ytw | [x, y<t ]) t Rθ = w . (13) l ] π (y w | [x, y w ]) t πref ytl | [x, y<t θ t <t
which can be interpreted as a relative policy-improvement cost between the two prefixes: it compares how far the l optimal policy must deviate from the reference at [x, y<t ] w versus [x, y<t ] under the KL regularizer. 4.4.1. P RACTICAL ESTIMATION . In practice Z([x, y<t ]; β) and DKL are either computationally intractable or expensive. We use plug-in estimators that preserve the key property that wt is state-only.
We define the t-th time step loss using the Bregman ratio matching framework in Section 3.2 as:
TBPO-Q (learned baseline). We parameterize a scalar baseline head bϕ ([x, y<t ]) to predict log Z([x, y<t ]; β) and set \ (Q) l w log wt = bϕ [x, y<t ] − bϕ [x, y<t ] , \ (Q) (Q) ŵt = exp log wt .
t Dht Rdata , Rθt = h t i ′ t Ep yw ≻yl |x h Rdata − h Rθt − h Rθt Rdata − Rθt . data
≤t
≤t
The final loss is the average of the component losses: Dh (Rdata , Rθ ) =
In practice, we attach a lightweight MLP head on top of the LM’s final hidden layer to predict Z([x, y<t ]; β). We train this head during optimization while stopping gradients into the LM backbone, so the LLM does not allocate capacity to baseline estimation at the expense of preference optimization; see Appendix C.3.
T 1X t D Rt , Rt , T t=1 h data θ
(14)
where T = min(|y w |, |y l |) is the length of the shorter response among the chosen and rejected sequences. In the following theorem, we show that under the sufficient model capacity, by matching the density ratio at the token level, we can recover the optimal policy in the Objective 1. Theorem 4.3. Under sufficient model capacity, we have πθ∗ = πθ̂ where πθ̂ = argminπθ Dh (Rdata , Rθ ).
TBPO-A (K3 KL estimator). We approximate π ∗ by the current policy πθ and estimate DKL (πref (· | s) ∥ πθ (· | s)) using the low-variance unbiased estimator (Schulman, 2020). See more in the Appendix C.3.
The loss function Dh (Rdata , Rθ ) is not tractable to train a policy model. We turn it to an equivalent tractable objective function in the following theorem. Theorem 4.4. We have L (θ, pdata ) = Dh (Rdata , Rθ ) + const where we define
TBPO-Q vs. TBPO-A. Both variants require a state-only correction because token comparisons are made across two different prefixes. TBPO-Q’s wt is a partition-function / softvalue ratio and is naturally handled by a learned baseline head. TBPO-A removes the partition function analytically but introduces a KL-baseline term, which can be estimated directly from πref and πθ using an unbiased estimator.
"
T
1 X ′ t t h Rθ Rθ T t=1 # t ′ t −1 − h Rθ − h Rθ
L (θ, pdata ) = Epdata (yw ≻yl |x)
(15)
The loss in Eq. (15) depends only on the ratio Rθt , hence applies identically to TBPO-Q and TBPO-A. Different ratio estimation methods differ only in their choice of function h. Following BPO (Kim et al., 2025), we employ h function as Scaled Basu’s power divergence (SBA), with following 1+λ −R formula h(R) = R sλ(λ+1) for better training flexibility and tuning flexibility. The λ parameter lets you control whether to focus on confident samples (low Rθ ) or uncertain ones (Rθ ≈ 1), which can be useful for handling noisy preference data.
4.5. Token-level Bregman ratio matching We now use density-ratio matching to derive a tractable optimization objective from the token-level BT preference model. Given a time-step t, we could naturally define the BT preference model for the sequences up to the t-th token: t Y w l w l pdata y≤t ≻ y≤t |x = pdata yiw ≻ yil | x, y<i , y<i . i=1
(11) 5
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Method
HellaSwag
ARC
MMLU
TruthfulQA
Winogrande
GSM8K
Average
Mistral 7B v0.1 SFT DPO TDPO TIS-DPO BPO
80.72±0.39 82.39±0.38 82.70±0.38 83.10±0.37 81.03±0.39
55.54±1.45 59.40±1.43 61.54±1.41 59.09±1.42 57.67±1.44
58.41±0.39 59.15±0.38 57.17±0.40 57.81±0.39 59.16±0.40
43.67±1.46 42.30±1.49 43.48±1.53 45.91±1.56 41.71±1.46
76.71±1.18 77.74±1.16 76.95±1.18 77.66±1.17 77.26±1.16
18.49±1.07 34.87±1.21 28.65±1.24 34.42±1.31 32.90±1.29
55.59 59.30 58.41 59.66 58.29
TBPO-Q (Ours) TBPO-A (Ours)
82.72±0.37 82.74±0.38
59.74±1.41 59.47±1.43
60.82±0.39 60.54±0.39
44.47±1.45 44.40±1.46
78.58±1.11 78.21±1.21
39.34±1.34 39.04±1.35
60.95 60.73
Llama3 8B SFT DPO TDPO TIS-DPO BPO
79.46±0.40 81.23±0.38 83.29±0.37 81.37±0.38 81.68±0.37
51.45±1.46 56.05±1.45 59.04±1.43 58.70±1.42 54.95±1.45
61.19±0.39 61.70±0.38 61.86±0.39 60.77±0.39 61.48±0.38
46.46±1.45 48.47±1.53 51.76±1.55 49.87±1.57 47.88±1.52
75.69±1.21 75.84±1.20 76.55±1.19 75.14±1.21 75.13±1.21
68.67±1.37 74.20±1.37 75.10±1.36 75.12±1.37 73.22±1.37
63.82 66.24 67.91 66.82 65.72
TBPO-Q (Ours) TBPO-A (Ours)
81.90±0.38 81.97±0.38
64.07±1.40 64.08±1.40
64.92±0.38 64.84±0.39
53.36±1.52 53.29±1.53
79.08±1.14 79.63±1.12
78.92±1.12 79.22±1.21
70.38 70.50
Table 1. General capabilities on the Open LLM Leaderboard suite for Mistral 7B v0.1 and Llama 3 8B. Scores are reported by lm-eval-harness (higher is better) with ± standard error. Best is in bold, second-best is underlined.
5. Experiments
analyzed the reliability and biases of LLM judges (Zhang et al., 2025; Han et al., 2025); following these findings, we adopt task-specific judge pairs. For Anthropic HH-RLHF and TL;DR, we use Llama 3 70B Instruct and DeepSeekV3, while for MT-Bench we use Llama 3 70B Instruct and Mixtral-8x22B Instruct v0.1. Additional details are provided in Appendix C.
In this section, we present an empirical study of Token-level Bregman Preference Optimization (TBPO). We compare TBPO-Q and TBPO-A to existing preference-optimization methods, with particular attention to alignment quality, training stability, and generation diversity, evaluating whether the theoretical advantages of token-level ratio matching manifest in practice.
5.2. General Capabilities A common concern in preference optimization is overspecialization to preference data, which may harm general capabilities. We evaluate general performance using the HuggingFace Open LLM Leaderboard protocol (Hugging Face, 2024) with the Language Model Evaluation Harness (Sutawika et al.) on six benchmarks spanning commonsense reasoning (ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2019)), multi-task understanding (MMLU (Hendrycks et al., 2021)), factuality (TruthfulQA (Lin et al., 2022)), and math reasoning (GSM8k (Cobbe et al., 2021)). Table 1 shows that TBPO improves the overall average across both backbones while remaining competitive on all benchmarks, indicating stronger alignment without sacrificing core capabilities.
5.1. Experiments settings Backbones and training data. We conduct experiments with two backbone models. Each setup initializes from a publicly available supervised fine-tuning (SFT) checkpoint, followed by preference optimization on the corresponding dataset. Specifically, Mistral 7B v0.1 is initialized from mistral-7b-sft-alpha and trained on UltraFeedback Binarized, while Llama 3 8B is initialized from LLaMA3-SFTv2 and trained on Llama3-UltraFeedback-ArmoRM. Baselines. We compare TBPO-Q and TBPO-A against various prior preference-optimization baselines, including SFT, DPO (Rafailov et al., 2023), TDPO (Zeng et al., 2024), TIS-DPO (Liu et al., 2025a), and BPO-SBA (Kim et al., 2025). See more details in Appendix C.2.
On Mistral 7B v0.1, sequence-level preference optimization already improves over SFT (DPO: 59.30 vs. SFT: 55.59). TBPO further improves the average: TBPO-Q is best (60.95) with TBPO-A close behind (60.73), exceeding the strongest non-TBPO baseline (TIS-DPO: 59.66). Gains
LLM judges. Some benchmarks, including Anthropic HH-RLHF, TL;DR, and MT-Bench, require pairwise evaluation with an LLM-as-a-judge framework. Prior work has 6
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
MTBench Pairwise Winrate Lose
Tie
Win
TBPOA
TBPOQ
Judge: Llama 3 70B
Judge: mixtral8x22b
BPO
6.9
32.5
60.6
7.5
26.2
66.2
DPO
8.2
36.5
55.4
9.4
27.5
63.1
TISDPO
13.1
33.8
53.1
10.6
35.0
54.4
TDPO
14.4
38.8
46.9
14.4
35.6
50.0
BPO
6.2
31.2
62.5
7.5
29.4
63.1
DPO
6.9
35.6
57.5
8.8
31.2
60.0
TISDPO
13.8
29.4
56.9
13.1
33.1
53.8
16.2
35.6
TDPO 100
11.2
80
60
40
20
40.6
0
48.1
20
40
60
80
100
100
80
60
40
20
0
48.1
20
40
60
80
100
Figure 1. MT-Bench pairwise win/tie/lose rates for TBPO-Q (top) and TBPO-A (bottom) against prior preference-optimization baselines, evaluated by two LLM judges. TBPO achieves higher win rates with low loss rates across both judges, and the advantage persists even against the strongest baseline.
are broad and strongest on reasoning-heavy tasks: TBPOQ attains the best GSM8K (39.34 vs. 34.87 for DPO and 18.49 for SFT) and also the best MMLU (60.82) and Winogrande (78.58). While DPO slightly reduces TruthfulQA vs. SFT (42.30 vs. 43.67), TBPO restores and improves it (44.47–44.40). Where other token-level methods peak (e.g., ARC for TDPO, TruthfulQA for TIS-DPO), TBPO remains highly competitive.
of decisive (non-tie) cases. This suggesting the advantage is not driven by a small number of outliers. The overall ordering and effect sizes are consistent across both judges, supporting the conclusion that TBPO improves generation quality in a robust, judge-agnostic manner, and that TBPOQ and TBPO-A are comparably strong. Harmless and Helpfulness. To evaluate preference alignment in safety-sensitive dialogue, we test on 200 held-out samples from the Anthropic HH-RLHF dataset. Using an LLM-as-a-judge protocol, we report pairwise and lengthcontrolled (LC) win rates (Li et al., 2023). Figure 2 shows LC win rate versus average response length, highlighting the verbosity–quality trade-off.
On Llama 3 8B, TBPO-A is best on average (70.50) with TBPO-Q close behind (70.38), improving substantially over TDPO (67.91). Improvements concentrate on ARC (+5 vs. TDPO; 64.08 vs. 59.04), GSM8K (+4; 79.22 vs. 75.10), and Winogrande (+3; 79.63 vs. 76.55). TBPO-A leads on ARC/GSM8K/Winogrande, while TBPO-Q attains the best MMLU (64.92) and TruthfulQA (53.36), showing robustness to the preference-model formulation.
Across both judges, TBPO variants achieve the strongest preference alignment while producing shorter responses than competing methods, ruling out verbosity as the primary driver of higher win rates. SFT is the most verbose model yet does not achieve top performance, whereas TBPO attains the best LC win rates with substantially shorter generations. Among TBPO variants, TBPO-Q produces the most concise responses while remaining at the top of the win-rate plot, and TBPO-A achieves the highest (or nearhighest) win rates with only a modest increase in length, indicating a controllable conciseness–performance trade-off. In contrast, strong sequence- and token-level baselines improve over SFT but generally require longer responses and still trail TBPO. The consistent ordering across judges and large margins relative to standard errors suggest TBPO’s gains reflect genuine improvements in helpful-and-harmless
Overall, TBPO delivers the strongest mean performance across two model families, with large gains on reasoningcentric benchmarks and no evidence of systematic trade-offs on broader language understanding. 5.3. Generation Quality MT-Bench. To evaluate generation quality, we run MTBench (Zheng et al., 2023) under the standard LLM-as-ajudge protocol and report pairwise win/tie/lose rates against prior preference-optimization baselines. Figure 1 shows that TBPO-Q and TBPO-A consistently improve win rates with low loss rates; against the strongest baseline (TDPO), many comparisons are ties, but TBPO wins the majority 7
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Anthropic HH Llama 3 8B Judge: Llama 3 70B
Judge: Deepseek V3
97.5 95.0
LC Win Rate
92.5 90.0 87.5 85.0 82.5 80.0 400
500
600
700
800
Average Response Length
900
1000
1100
400
500
600
700
800
Average Response Length
900
1000
1100
Figure 2. LC win rate vs. average response length against the dataset-preferred completion for Llama 3 8B, evaluated by two LLM judges (Llama 3 70B, DeepSeek-V3); error bars are ±1 s.e. over 200 prompts. TBPO leads with shorter outputs, indicating gains beyond verbosity and consistent across judges.
Generation Diversity SFT
30.0 27.5
TBPO-Q
0.7
TBPO-A
25.0
Self-BLEU ( )
Predictive Entropy ( )
0.8
Two patterns emerge. (1) DPO/TDPO/BPO have moderate Distinct-1 but lower predictive entropy and higher self-BLEU, implying samples become more similar and decoding more deterministic (mild mode-seeking). TISDPO amplifies this, achieving the lowest entropy and highest self-BLEU, i.e., the strongest collapse even when Distinct-1 alone does not fully reveal it. (2) TBPO shifts the Pareto frontier toward the desirable upper-right region while keeping self-BLEU favorable, giving the best three-metric tradeoff: versus DPO-style methods it raises entropy and Distinct1 without the usual across-sample diversity loss; versus SFT (high entropy, low self-BLEU, lower Distinct-1) it preserves non-collapsed sampling while increasing lexical variety. Overall, TBPO better balances preference satisfaction with diversity preservation, producing outputs that are less redundant across samples and less over-confident at the token level.
0.6
22.5 DPO
0.5
BPO
TDPO
20.0 17.5
0.4
TISDPO 0.62
0.63
0.64
Distinct-1 ( )
0.65
15.0 0.66
Figure 3. Generation diversity trade-offs: Distinct-1 vs. predictive entropy (higher is better), colored by self-BLEU (lower is better). TBPO achieves the best three-way trade-off across all three metrics.
Additional experiments on generalization and out-ofdomain tasks for TBPO are reported in Appendix B.
6. Conclusion response quality rather than judge bias or verbosity effects.
We presented a principled token-level preference optimization framework aligned with the autoregressive nature of language models. We show theoretically that minimizing the proposed objective recovers a token-wise optimal policy, establishing a direct link between preference modeling and optimization behavior. Experiments across benchmarks and model families demonstrate consistent improvements over strong baselines, particularly in challenging reasoning and alignment settings.
Generation Diversity. Preference optimization can reduce generation diversity (Kirk et al., 2024). We evaluate diversity on 100 held-out prompts from the test split of UltraFeedback Binarized using predictive entropy (Zeng et al., 2024), Self-BLEU (Zhu et al., 2018), and Distinct-1 (Li et al., 2016). Figure 3 summarizes the trade-off, where higher Distinct-1 and predictive entropy indicate greater diversity, while lower Self-BLEU indicates less redundancy. 8
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Impact Statement
Gutmann, M. U. and Hyvärinen, A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(11):307–361, 2012. URL http: //jmlr.org/papers/v13/gutmann12a.html.
This work focuses on the development of a principled learning algorithm with theoretical guarantees. The primary impact of this research is methodological, contributing to the design of more reliable and well-understood machine learning systems. While the proposed method is evaluated on standard benchmarks, it is not tied to a specific highrisk application domain. We do not foresee any immediate negative societal consequences arising directly from this work; however, as with most advances in machine learning, downstream applications should be deployed responsibly and with appropriate consideration of ethical and societal implications.
Han, S., Junior, G. T., Balough, T., and Zhou, W. Judge’s verdict: A comprehensive analysis of llm judge capability through human agreement, 2025. URL https://ar xiv.org/abs/2510.09738. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https: //arxiv.org/abs/2009.03300.
References
Hong, J., Lee, N., and Thorne, J. ORPO: monolithic preference optimization without reference model. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 11170– 11189. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.EMNLP-MAIN.626. URL https://doi.org/10.18653/v1/2024.emn lp-main.626.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pp. 4447–4455. PMLR, 2024. URL https://proceedi ngs.mlr.press/v238/gheshlaghi-azar24a .html.
Huang, J., Gretton, A., Borgwardt, K., Schölkopf, B., and Smola, A. Correcting sample selection bias by unlabeled data. In Schölkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper _files/paper/2006/file/a2186aa7c086b 46ad4e8bf81e2a3a19b-Paper.pdf.
Bregman, L. M. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. Ussr Computational Mathematics and Mathematical Physics, 7: 200–217, 1967. URL https://api.semanticsc holar.org/CorpusID:121309410. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hugging Face. Open LLM leaderboard v1. https://hu ggingface.co/docs/leaderboards/open_ llm_leaderboard/archive, 2024. Leaderboards documentation; archived June 2024; accessed 2026-0125.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.0 5457.
Kim, Y., Bae, H., Na, B., and Moon, I.-C. Preference optimization by estimating the ratio of the data distribution, 2025. URL https://arxiv.org/abs/2505.1 9601.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=PXD3FAVHJT.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference 9
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., 1994. ISBN 9780471619772. doi: 10.1002/9780470316887.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models, 2016. URL https://arxiv.org/ abs/1510.03055.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_ eval, 2023.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/19 07.10641.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https: //arxiv.org/abs/2109.07958.
Schulman, J. Approximating kl divergence. https://jo schu.net/blog/kl-approx.html, March 2020. Blog post. Accessed: 2026-01-28.
Liu, A., Bai, H., Lu, Z., Sun, Y., Kong, X., Wang, X. S., Shan, J., Jose, A. M., Liu, X., Wen, L., Yu, P. S., and Cao, M. TIS-DPO: token-level importance sampling for direct preference optimization with estimated weights. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025a. URL https: //openreview.net/forum?id=oF6e2WwxX0.
Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P., and Kawanabe, M. Direct importance estimation with model selection and its application to covariate shift adaptation. In Platt, J., Koller, D., Singer, Y., and Roweis, S. (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper _files/paper/2007/file/be83ab3ecd0db 773eb2dc1b0a17836a1-Paper.pdf.
Liu, S., Fang, W., Hu, Z., Zhang, J., Zhou, Y., Zhang, K., Tu, R., Lin, T., Huang, F., Song, M., Li, Y., and Tao, D. A survey of direct preference optimization. CoRR, abs/2503.11701, 2025b. doi: 10.48550/ARXIV.2503.11 701. URL https://doi.org/10.48550/arXiv .2503.11701.
Sugiyama, M., Suzuki, T., and Kanamori, T. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64:1009–1044, October 2012. doi: 10.1007/s10463-011-0343-8. URL https: //doi.org/10.1007/s10463-011-0343-8.
Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data, 2023. URL https://arxiv.org/abs/2211.0 0802. Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_f iles/paper/2024/hash/e099c1c9699814a f0be873a175361713-Abstract-Conference. html. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
Sutawika, L., Schoelkopf, H., Gao, L., Abbasi, B., Biderman, S., Tow, J., et al. EleutherAI/lm-evaluation-harness. URL https://doi.org/10.5281/zenodo.5 371628. Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. β-dpo: Direct preference optimization with dynamic β. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http: //papers.nips.cc/paper_files/paper/2 024/hash/ea888178abdb6fc233226d12321 d754f-Abstract-Conference.html. Xiao, W., Wang, Z., Gan, L., Zhao, S., He, W., Tuan, L. A., Chen, L., Jiang, H., Zhao, Z., and Wu, F. A comprehensive survey of datasets, theories, variants, and applications in direct preference optimization. CoRR, abs/2410.15595,
10
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
2024. doi: 10.48550/ARXIV.2410.15595. URL https: //doi.org/10.48550/arXiv.2410.15595. Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=51iw kioZpn. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.0 7830. Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization, 2024. URL https://arxiv.org/abs/2404.11999. Zhang, T., Cao, M., Lam, A., Zhang, S., and Chen, K. Compassjudger-2: Towards generalist judge model via verifiable rewards, 2025. URL https://arxiv.or g/abs/2507.09104. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Brooks, D., Xing, E., Gonzalez, J. E., Jordan, M. I., and Stoica, I. Judging LLM-as-ajudge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models, 2018. URL https://arxiv.or g/abs/1802.01886.
11
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
A. Theory Development A.1. Proof for Prop 4.1 Proof. We prove the proposition by deriving the policy ratio induced by a token-level Bradley–Terry model parameterized by the reference-policy Q-function. By the token-level Bradley–Terry model in Eq. (7), we have w l w l p(ytw ≻ ytl | x, y<t , y<t ) = σ Qπref ([x, y<t ], ytw ) − Qπref ([x, y<t ], ytl ) . Using σ(u) = (1 + exp(−u))−1 and conditioning on the same pair of prefixes, taking the odds ratio yields w l p(ytw ≺ ytl | x, y<t , y<t ) l l w w = exp Q ([x, y ], y ) − Q ([x, y ], y ) . π π <t t <t t ref ref w , yl ) p(ytw ≻ ytl | x, y<t <t Substituting the closed-form expression of the reference-policy Q-function from Eq. (2) gives w w l l πθ∗ (ytw | [x, y<t ]) p(ytw ≺ ytl | x, y<t , y<t ) πθ∗ (ytl | [x, y<t ]) = exp − β log β log w w w l w l l l πref (yt | [x, y<t ]) p(yt ≻ yt | x, y<t , y<t ) πref (yt | [x, y<t ]) ! l w + β log Z([x, y<t ]; β) − β log Z([x, y<t ]; β) .
Rearranging terms, we obtain l w ) , y<t p(ytw ≺ ytl | x, y<t = w l w l p(yt ≻ yt | x, y<t , y<t )
w l ]) ]) πref (ytw | [x, y<t πθ∗ (ytl | [x, y<t l l ∗ w w πref (yt | [x, y<t ]) πθ (yt | [x, y<t ])
β
Z([x,y l ];β) exp β log Z([x,y<t w ];β) . <t
Defining the prefix-dependent weight Z([x,y l ];β) wt = exp log Z([x,y<t w ],β) , <t
the above expression can be written as l w ) , y<t p(ytw ≺ ytl | x, y<t = w l w l p(yt ≻ yt | x, y<t , y<t )
w l ]) ]) πref (ytw | [x, y<t πθ∗ (ytl | [x, y<t l l ∗ w w πref (yt | [x, y<t ]) πθ (yt | [x, y<t ])
β
wtβ .
Raising both sides to the power 1/β and rearranging terms yields w w ]) πθ∗ (ytw | [x, y<t πref (ytw | [x, y<t ]) = ∗ l l l l πθ (yt | [x, y<t ]) πref (yt | [x, y<t ])
l w , y<t ) p(ytw ≻ ytl | x, y<t w l w l ) p(yt ≺ yt | x, y<t , y<t
1/β wt ,
which completes the proof. A.2. Proof for Prop 4.2 Proof. We prove Proposition 4.2 by deriving the policy ratio induced by a token-level Bradley–Terry model parameterized by reference-policy advantages. By definition, the token-level Bradley–Terry likelihood is given by w l w l p ytw ≻ ytl x, y<t , y<t = σ Aπref ([x, y<t ], ytw ) − Aπref ([x, y<t ], ytl ) . Using σ(u) = (1 + e−u )−1 and taking the odds ratio while conditioning on the same pair of prefixes yields w l p ytw ≺ ytl x, y<t , y<t l l w w = exp A ([x, y ], y ) − A ([x, y ], y ) . π π <t t <t t ref ref w , yl p ytw ≻ ytl x, y<t <t 12
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Substituting the closed-form expression of the reference-policy advantage from Eq. (3) into the above expression gives w l l w , y<t p ytw ≺ ytl x, y<t πθ∗ (ytl | [x, y<t ]) πref (ytw | [x, y<t ]) = exp β log ∗ w w l l w , yl π (y | [x, y ]) π (y | [x, y ]) p ytw ≻ ytl x, y<t ref t <t <t <t θ t h l l + β DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) ! i w ∗ w − DKL πref (· | [x, y<t ]) πθ (· | [x, y<t ]) . Raising both sides to the power 1/β and rearranging terms yields w w πref (ytw | [x, y<t ]) πθ∗ (ytw | [x, y<t ]) = ∗ l l l l πθ (yt | [x, y<t ]) πref (yt | [x, y<t ])
where
!1/β w l , y<t p ytw ≻ ytl x, y<t · wt , w , yl p ytw ≺ ytl x, y<t <t
l l w w wt = exp DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) − DKL πref (· | [x, y<t ]) πθ∗ (· | [x, y<t ]) .
This establishes Proposition 4.2. A.3. Proof for Theorem 4.3 Theorem. Under sufficient model capacity, we have πθ∗ = πθ̂ where πθ̂ = argminπθ Dh (Rdata , Rθ ). Proof. Let arg minπθ Dh (Rdata ∥Rθ ) = πθ̂ . We have the following decomposition:: Dh Rdata , Rθ̂
T T h i 1X 1X t t t t Dh Rdata , Rθ̂ = Epdata (yw ≻yl |x) Bh Rdata , Rθ̂t . = ≤t ≤t T t=1 T t=1
(16)
where T denotes the minimum sequence length. t t By the non-negativity of the Bregman divergence, Bh Rdata , Rθ̂t attains its minimum value 0 if and only if Rdata = Rθ̂t for w l t a given (y<t , y<t , x). Assuming that the data distribution has full support, the minimizer πθ̂ therefore satisfies Rdata = Rθ̂t w l t t for all (y<t , y<t , x). Substituting the definitions of Rdata and Rθ̂ yields the following condition:
#β " w l w l ]) ] πref (ytw | [x, y<t πθ̂ ytl | [x, y<t pdata ytw ≺ ytl | x, y<t , y<t = w . w , yl l ] π (y w | [x, y w ]) t pdata ytw ≻ ytl | x, y<t πref ytl | [x, y<t <t <t θ̂ t Looping over all time steps from t = 1 to T and combining with the recursive formula in Eq. 12, we obtain β w l l w pdata y≤T ≺ y≤T |x | x πref y≤T |x πθ̂ y≤T = wx , w ≻ yl l w |x pdata y≤T | x π y | x π y ref ≤T ≤T ≤T θ̂ where wx :=
QT
t=1 wt .
β1 w l pdata y≤T ≻ y≤T | x = wx . l l w ≺ yl πθ̂ y≤T | x πref y≤T | x pdata y≤T | x ≤T
w πθ̂ y≤T |x
w πref y≤T |x
Similarly, we derive the sequence-level ratio for the optimal policy πθ∗ by aggregating the token-level conditions from Propositions 4.1 and 4.2 over t = 1 . . . T and applying the recursive preference formula (Eq. 12): β1 w w w l πref y≤T |x πθ∗ y≤T |x pdata y≤T ≻ y≤T |x = wx . l l w ≺ yl πθ∗ y≤T |x πref y≤T |x pdata y≤T | x ≤T 13
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Comparing this with the expression for πθ̂ derived above reveals that both policies satisfy the exact same likelihood ratio for w l all points (x, y≤T , y≤T ): w w πθ̂ y≤T |x πθ∗ y≤T |x = l l πθ̂ y≤T πθ∗ y≤T |x |x Using the completeness property of concrete score (Meng et al., 2023), we conclude that πθ̂ = πθ∗ . A.4. Proof for Theorem 4.4 Theorem. We have L (θ, pdata ) = Dh (Rdata , Rθ ) + C with C is a constant with respect to θ where we define " L(θ, pdata ) = Epdata (yw ≻yl |x)
# T 1 X ′ t t t ′ t −1 h Rθ Rθ − h Rθ − h Rθ . T t=1
Proof. By definition: h t i t t Dht Rdata , Rθt = Epdata(yw ≻yl |x) h Rdata − h Rθt − h′ Rθt Rdata − Rθt ≤t ≤t h i t t = Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − h′ (Rθt )Rdata ) + Epdata(yw ≻yl |x) h(Rdata ≤t ≤t ≤t ≤t w l pdata (ytw ≺ ytl | x, y<t , y<t ) −C = Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − h′ (Rθt ) w , yl ) ≤t ≤t pdata (ytw ≻ ytl | x, y<t <t w l = Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − Epdata(yw ≻yl |x) h′ (Rθt )pdata (ytw ≺ ytl | x, y<t , y<t ) − C. ≤t ≤t ≤t−1 ≤t−1 (17) Next, observe that: w l w l h′ (Rθ ) pdata (ytw ≺ ytl | x, y<t , y<t ) = pdata (ytw ≺ ytl | x, y<t , y<t ) h′ Rθ (x, ytw , ytl ) w l = pdata (ytw ≻ ytl | x, y<t , y<t ) h′ Rθ (x, ytl , ytw ) !β l l w w ] | [x, y ]) π y | [x, y π (y ref θ <t t <t t w l wt−1 = pdata (ytw ≻ ytl | x, y<t , y<t ) h′ w ]) π y l | [x, y l ] πref (ytw | [x, y<t θ t <t 1 w l = pdata (ytw ≻ ytl | x, y<t , y<t ) h′ Rθt (x, ytw , ytl ) w l = pdata (ytw ≻ ytl | x, y<t , y<t ) h′ ((Rθt )−1 )
Therefore: Epdata(yw
l ≻y≤t−1 |x ≤t−1
h i ′ t w l w l h (R ) p (y ≺ y | x, y , y ) = Epdata(yw data θ t t <t <t )
h
w l w l ′ t −1 ) pdata (yt ≻ yt | x, y<t , y<t ) h (Rθ ) h i = Epdata(yw ≻yl |x) h′ (Rθt )−1 . l ≻y≤t−1 |x ≤t−1
≤t
≤t
Substituting the above result into Eq. (17) yields: t Dht Rdata , Rθt = Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − h′ (Rθt )−1 − C ≤t ≤t = Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − h′ (Rθt )−1 − C 14
i
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Averaging over timesteps, we obtain: T
Dh (Rdata , Rθ ) =
1X t D Rt , Rt T t=1 h data θ
T 1X Epdata(yw ≻yl |x) h′ (Rθt )Rθt − h(Rθt ) − h′ (Rθt )−1 − C T t=1 # " T 1X ′ t t t ′ t −1 h (Rθ )Rθ − h(Rθ ) − h (Rθ ) −C = Epdata(yw ≻yl |x) T t=1
=
= L(θ, pdata ) − C
B. Additional Experimental Result B.1. Out-of-domain Result on Summarization Task To stress-test out-of-domain (OOD) transfer, we evaluate on 200 held-out samples from the test split of TLDR. This is a stringent OOD setting: TLDR is a summarization benchmark with long input documents and short, reference-style outputs, and we do not use any TLDR data during training (TBPO is trained only on UltraFeedback). As with Anthropic HH, we quantify OOD generalization by measuring length-controlled (LC) win rate to reduce verbosity bias, against the dataset-preferred completion and plotting it against average response length (Figure 4). Across both judges (Llama 3 70B and DeepSeek-V3), TBPO (A/Q variants) lies on the Pareto frontier, achieving the strongest win rates while producing substantially shorter outputs than prior preference-optimization baselines. Because TLDR differs in domain, task format, and implicit preference structure, these gains are difficult to attribute to UltraFeedback-specific stylistic overfitting; instead, they suggest TBPO learns preferences that transfer under distribution shift. The plot also highlights a clear mismatch between length and preference in this OOD task: SFT produces the longest responses yet attains low win rates under both judges, while DPO/BPO/TISDPO/TDPO generally trade longer generations for incremental gains and still fall short of TBPO. Since TLDR implicitly rewards concise, faithful summarization, TBPO’s ability to win while staying short strengthens the interpretation that it adapts to task-appropriate brevity rather than relying on generic verbosity. The consistent ordering across two independent judges further supports that the observed advantage reflects genuine OOD generalization.
C. Detailed Experimental Setup C.1. Benchmark and Metrics Details Open LLM Leaderboard (Hugging Face, 2024). The leaderboard evaluates open-weight LLMs on six core benchmarks using the EleutherAI Language Model Evaluation Harness under a fixed few-shot setup: ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), TruthfulQA (0-shot; implemented as truthfulqa-mc), Winogrande (5-shot), and GSM8K (5-shot). Higher scores indicate better performance across these tasks. MT-Bench (Zheng et al., 2023). MT-Bench-101 is a fine-grained multi-turn dialogue benchmark organized by a three-tier ability taxonomy. It contains 1,388 multi-turn dialogues comprising 4,208 turns across 13 task types. For evaluation, the benchmark uses the golden context (the full dialogue history) as input context for each turn, and employs an LLM as an automatic judge to score each turn according to task-specific scoring guidelines. To reflect robustness across the whole interaction, the final dialogue score is taken as the lowest (minimum) round score among the turns. Anthropic HH-RLHF. HH-RLHF provides human preference data targeting helpfulness and harmlessness. The preference-modeling portion is distributed as JSONL records containing paired model responses: a chosen response preferred by annotators and a rejected response. The dataset is intended for training preference/reward models for RLHF, and the dataset card explicitly warns against using these data for supervised fine-tuning of dialogue agents. The repository also includes red-teaming dialogues and notes that the data may contain potentially offensive or upsetting content. 15
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
TLDR Llama 3 8B Judge: Llama 3 70B
Judge: Deepseek V3
100
LC Win Rate
80
60
40
20
0
200
400
600
800
1000
1200
Average Response Length
1400
1600
1800
200
400
600
800
1000
1200
Average Response Length
1400
1600
1800
Figure 4. LC win rate vs. average response length on TLDR for Llama 3 8B, judged by Llama 3 70B and DeepSeek-V3. TBPO achieves highest win rate with shorter outputs, suggesting strong OOD generalization despite training only on UltraFeedback dataset.
TLDR. The TL;DR dataset is a processed Reddit summarization corpus built from posts where authors append a “TL;DR” summary. It is provided in a prompt–completion format for summarization training, with a long-form post as input and its short TL;DR as the target summary. The dataset card specifies two columns: "pompt" (the full Reddit post text) and "completion" (the TL;DR summary). Length-controlled Win Rate. In a pairwise preference evaluation, each example i consists of an input (e.g., prompt) xi (A) (B) and two candidate outputs zi and zi , with a judge label yi ∈ {0, 1} indicating whether A is preferred (yi = 1) over B. The (raw) win rate is WR(A, B) = 100 · Ei [yi ] . (18) LLM-based judges (and sometimes humans) can exhibit length bias, i.e., a systematic preference for longer outputs, which can inflate win rates for more verbose systems. To control for this confounder, we model pairwise preferences with a logistic regression that includes the output-length difference as a covariate: (A) (B) Pr(yi = 1 | xi ) = σ ∆θ + λ g(∆ℓi ) + uxi , ∆ℓi := ℓ zi − ℓ zi , (19) where σ(t) = 1+e1 −t , ℓ(·) is a length measure (e.g., tokens), ∆θ captures the system advantage of A over B, and uxi optionally absorbs item-specific difficulty via fixed effects (or random effects). We use a bounded transform such as g(∆ℓi ) = tanh ∆ℓi /s (with scale s, e.g., the empirical standard deviation) to reduce sensitivity to extreme length gaps. The length-controlled win rate is the counterfactual preference probability when both outputs have equal length, i.e., conditioning on ∆ℓi = 0: LCWR(A, B) = 100 · Ei [Pr(yi = 1 | xi , ∆ℓi = 0)] = 100 · Ei σ ∆θ + uxi . (20) Intuitively, LCWR reports the implied win probability after removing any advantage attributable purely to verbosity. Diversity metrics.
We evaluate the diversity of model responses using three metrics:
• Predictive Entropy: Evaluates the diversity of model responses by directly reflecting the probabilities assigned to each generated token. For each prompt, we sample five responses and compute the predictive entropy. Higher entropy indicates more diverse outputs. • Self-BLEU: Measures the diversity of generated sentences. For each prompt, we generate five samples and compute the BLEU score (Zhu et al., 2018) for each pair of responses, then take the average. Lower Self-BLEU indicates higher diversity. • Distinct-1: Assesses the lexical diversity by computing the ratio of unique unigrams to the total number of unigrams (Li et al., 2016). Higher scores indicate greater lexical diversity. 16
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
C.2. Baseline details SFT. Supervised fine-tuning (SFT) adapts a pretrained LM using demonstration data (prompt/instruction → reference answer pairs) with standard maximum-likelihood training (token-level cross-entropy / next-token prediction). Its purpose is to directly teach instruction following from supervised targets, without preference comparisons or RL. In RLHF-style pipelines, SFT is commonly used as an initialization step before preference optimization. In our experiments, we perform SFT on the chosen responses from the preference dataset. DPO (Rafailov et al., 2023). Direct Preference Optimization (DPO) trains a policy from pairwise preferences (winner vs. loser responses for the same prompt) using a simple logistic objective. It can be derived as a closed-form solution to a KL-regularized RLHF objective, avoiding both an explicit reward model and on-policy RL. A fixed reference policy provides regularization while the learned policy increases the relative likelihood of preferred responses. TDPO (Zeng et al., 2024). Token-level DPO (TDPO) extends DPO from sequence-level preferences to token-level optimization. By introducing token-wise constraints (e.g., per-token forward-KL terms), TDPO better matches autoregressive generation and aims to improve alignment while offering finer control over diversity. TIS-DPO (Liu et al., 2025a). TIS-DPO addresses the limitation of treating an entire response as a single unit: different tokens can contribute unequally to preference. It proposes a token-level importance-sampling objective with per-token weights reflecting reward contribution, yielding an unbiased objective even when the dataset is not ideally balanced. Since token rewards/weights are not observed, TIS-DPO estimates importance weights from probability differences produced by contrastive LLMs (e.g., via contrastive prompting, separate win/lose models, or forward/reverse DPO). BPO (BPO-SBA) (Kim et al., 2025). Bregman Preference Optimization (BPO) generalizes DPO from a likelihood-ratio viewpoint, avoiding an explicit reward model and partition function while retaining theoretical guarantees. It defines a family of tractable objectives induced by Bregman divergences, with DPO as a special case, and introduces gradient scaling (Scaled Basu’s power divergence, SBA) to improve optimization behavior for certain instances. Empirically, BPO variants can improve win rate and output entropy relative to DPO, mitigating some fidelity–diversity trade-offs. Baseline Implementation. SFT. We initialize from the same SFT checkpoints as TBPO and train for 1 epoch on 2 H100 GPUs (batch size 8, gradient accumulation 4, learning rate 2 × 10−4 ) using AdamW. DPO/TDPO/TIS-DPO. We initialize from the same SFT checkpoints and train on 2 H100 GPUs (batch size 4, gradient accumulation 8, learning rate 5 × 10−7 ) using RMSProp with β = 0.1. Unless otherwise noted, we use the hyperparameters from the official implementations/papers for TDPO and TIS-DPO. BPO (BPO-SBA). Since the authors do not release code, we reimplement BPO and match the hyperparameters reported in the paper. C.3. Implementation Details C.3.1. P ER - STATE WEIGHT APPROXIMATION Estimating the state-only correction weight wt adds negligible overhead relative to standard preference optimization. TBPO-Q. We implement the baseline head bϕ (s) as a 1-hidden-layer MLP on top of the LLM’s final hidden state with MLP hidden size is 1024, trained with a separate AdamW optimizer (lr 10−3 ). We detach the LLM backbone (stop-gradient) when updating bϕ to avoid coupling the baseline fitting signal with the main preference-optimization updates. TBPO-A. We approximate π ∗ by the current policy πθ and estimate DKL (πref (· | s) ∥ πθ (· |)) using the K3 Monte-Carlo estimator from Schulman (2020). For an action a ∼ πref (· | s) and ratio r(a, s) = πθ (a | s)/πref (a | s), we use b (k3) (s; a) = (r(a, s) − 1) − log r(a, s) and plug the result into the KL-baseline difference defining wt(A) . D KL C.3.2. H YPERPARAMETERS . We train TBPO on 4 H100 GPUs with batch size 32 and max response length 2048, using RMSProp with a cosine learning-rate schedule (warmup ratio 0.05), lr 5 × 10−7 , for 1 epoch. Unless stated otherwise, we use β = 0.1 and Bregman parameters λ = 0 and s = 4. 17