Abstract Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparisonbased Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement. Keywords: Preference alignment, comparison oracles, zeroth-order optimization, KL regularization, local coverage

1 Introduction Generative AI has become an increasingly important tool for building and managing intelligent systems across academia, industry, and government. Large language models (LLMs) are a core part of this progress, with strong capabilities in data organization, retrieval, reasoning, and analysis (Brown et al., 2020; Chowdhery et al., 2023; Touvron et al., 2023; ©2026 Peter Chen and Xi Chen and Wotao Yin and Tianyi Lin. License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at http://jmlr.org/papers/vXXX/XXX.html.

Chen, Chen, Yin, and Lin

Achiam et al., 2023; Bubeck et al., 2023). Since these models are trained on large and heterogeneous corpora, they need further alignment with human preferences so that their responses are helpful, harmless, and reliable (Bai et al., 2022). A prominent approach is reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020), which first learns a reward model from human preference pairs and then optimizes the policy using reinforcement learning. Despite its empirical success (Ziegler et al., 2019; Ouyang et al., 2022; Touvron et al., 2023; Achiam et al., 2023), RLHF requires a multi-stage training pipeline and can be expensive in memory and computation. This motivates direct alignment methods, e.g., direct preference optimization (DPO) (Rafailov et al., 2023) and its variants (Azar et al., 2024; Ethayarajh et al., 2024; Park et al., 2024; Xu et al., 2024a; Tang et al., 2024a; Meng et al., 2024; Chen et al., 2025a; Zhao et al., 2025), which directly optimize the policy using preference pairs and avoid separately training a reward model. Direct alignment methods are appealing because of their simplicity and stability. Yet, they suffer from a critical issue known as likelihood displacement. Likelihood displacement refers to the counter-intuitive situation where training increases the likelihood of preferred responses relative to dispreferred ones, but decreases the absolute probability of the preferred responses, leading to “unintentional unalignment” (Pal et al., 2024; Tajwar et al., 2024; Rafailov et al., 2024b; Pang et al., 2024; Liu et al., 2024b; Yuan et al., 2025; Razin et al., 2025). For example, training a model to prefer No over Never can sharply increase the likelihood of Yes. Practically, this issue can harm LLM behavior by shifting probability mass to unsafe responses. When the prompt asks for steps for a terrorist organization to infiltrate a government agency, Gemma-2B-it initially generates refusal responses, while DPO training can make the model comply with the unsafe request because likelihood displacement shifts probability mass away from refusal responses; see Razin et al. (2025, Table 18). Another related issue is verbosity, which refers to the tendency of models fine-tuned with RLHF (Singhal et al., 2024; Kabir et al., 2024) or direct alignment methods (Park et al., 2024; Amini et al., 2024; Rafailov et al., 2024a) to generate longer responses without a corresponding improvement in quality, resulting in lower efficiency and higher consumption of hardware resources. Recent works have suggested that likelihood displacement is related to preference pairs whose preferred and dispreferred responses are similar under model-dependent measures (Pal et al., 2024; Razin et al., 2025). We refer to such small margin pairs as noisy preference pairs in this paper (see Eq. (10)). Existing methods have tried to mitigate likelihood displacement by adding additional regularization (Pal et al., 2024; Rafailov et al., 2024b). More recently, Razin et al. (2025) proposed to measure the similarity between preferred and dispreferred responses using the centered hidden embedding similarity (CHES) score, and empirically showed that filtering out preference pairs identified by the CHES score as problematic can be more effective for mitigating likelihood displacement than adding supervised fine-tuning (SFT) regularization. This finding highlights the role of data geometry in direct alignment. However, filtering noisy pairs also removes them from training entirely, even though these pairs may still contain useful comparative information. While DPO provides a computationally convenient framework by maximizing a certain log-likelihood margin between preferred and dispreferred responses, this objective function can be viewed as a proxy for the true goal of alignment. This proxy is effective when preference pairs clearly distinguish better responses from worse responses. However, when 2

Comparison-Based Preference Alignment

faced with noisy pairs – where the preference signal is weak or ambiguous under modelbased similarity measures – optimizing a fixed DPO-style objective can lead to adverse effects such as likelihood displacement. In such cases, the pair may still provide useful local information, even if it is not suitable for direct optimization by a margin-based loss. This motivates a comparison-oracle view of preference alignment. Explicitly defining alignment as a single optimizable mathematical objective function is exceptionally challenging. Instead of pursuing such an explicit objective, we ask whether a nearby policy perturbation improves the local behavior of the model on preference pairs. A favorable perturbation should increase the likelihood of the preferred response and decrease the likelihood of the dispreferred response. In this way, noisy preference pairs are treated as comparison signals about a latent alignment objective, rather than as direct samples for a fixed loss function. In this paper, we propose a zeroth-order preference alignment method based on comparison oracles, called ComPO. Our approach perturbs the current policy, evaluates whether each perturbation increases the likelihood of preferred responses and decreases that of dispreferred responses, and aggregates the resulting one-bit signals to estimate a normalized update direction. This allows low-margin pairs, designated as noisy, to contribute to alignment without directly optimizing a differentiable preference loss on them, complementing standard direct alignment methods applied to clean pairs. We further extend ComPO to control policy deviation using unlabeled online generations, while retaining offline preference pairs as the source of comparison signals. The motivation follows the coverage perspective of Song et al. (2024b): reverse KL can be estimated from generations of the policy being evaluated, and constraining it permits a performance analysis based on the coverage within a prescribed neighborhood of the reference policy rather than over the full policy class. Coverage remains a separate assumption and is not implied by the KL constraint. Our empirical also examines length-related effects which have been studied in the literature (Gao et al., 2023; Dubois et al., 2023; Park et al., 2024; Amini et al., 2024; Xu et al., 2024a; Meng et al., 2024; Pang et al., 2024). Although ComPO is not specifically designed to control verbosity, we evaluate its length-controlled (LC) win rates and examine pair-level likelihood changes. We interpret higher LC win rates as improved judged performance after adjustment for response length, rather than direct evidence of shorter responses. Contributions.

Our contributions can be summarized as follows:

1. We develop ComPO, a comparison-based method that uses low-margin preference pairs to refine an aligned policy without directly optimizing a differentiable preference loss on those pairs. Its practical offline implementation uses output-layer perturbations and entry-wise thresholding. The online extension retains the same comparison mechanism and uses unlabeled current-policy generations to adapt the step size. 2. We establish a best-iterate convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and oracle compatibility. For the basic online scheme, we prove feasibility under an exact reverse-KL constraint and bound the performance gap in terms of in-distribution pairwise reward error under local coverage. 3. We evaluate ComPO on base and instruction-tuned models from the Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 families. The experiments assess its compatibility 3

Chen, Chen, Yin, and Lin

with direct alignment methods, its design choices, and the effects of online damping and replay. Pair-level likelihood diagnostics complement the benchmark evaluations. Relationship to the conference version. A preliminary version of this work appeared at NeurIPS 2025 (Chen et al., 2025b). It introduced offline ComPO, preference comparison oracle, the convergence analysis, and the original offline experiments. The journal extension adds the online extension, its coverage-based analysis, and experiments on additional model families, including evaluations of online regularization and replay. Related works. Direct preference alignment methods, including DPO (Rafailov et al., 2023), are simple and more stable offline alternatives to RLHF. Several DPO variants with alternative objectives have been proposed, including ranking-based variants beyond pairwise preference data (Dong et al., 2023; Yuan et al., 2023; Song et al., 2024a; Chen et al., 2024; Liu et al., 2025) and reference-model-free variants (Hong et al., 2024; Meng et al., 2024). It is well known that DPO suffers from the issues of verbosity (Park et al., 2024; Amini et al., 2024; Rafailov et al., 2024a) and likelihood displacement (Pal et al., 2024; Tajwar et al., 2024; Rafailov et al., 2024b; Pang et al., 2024; Liu et al., 2024b; Yuan et al., 2025), which can be interpreted from a unified perspective of data curation (Park et al., 2024; Razin et al., 2025). Our work continues along this perspective by arguing that these issues can be mitigated by using the information contained in noisy preference pairs for which the reference model assigns similar likelihoods to preferred and dispreferred responses. Recent work has examined different roles of online data in preference fine-tuning. Online preference optimization can acquire additional labels for responses generated by the current policy, as in the online AI feedback approach of Guo et al. (2024). In contrast, Song et al. (2024b) introduce HyPO, which combines offline preference optimization with reverse-KL regularization estimated from unlabeled online samples. Our online extension follows this separation between preference supervision and regularization, but uses comparison-derived update directions. A complementary line of work studies active exploration (Xie et al., 2025) by augmenting online DPO with an explicit exploration bonus to guide the acquisition of preference feedback. Online ComPO does not acquire new preference labels or introduce such a bonus and its analysis concerns policy performance under local coverage. Comparison-based optimization includes coordinate-search methods (Jamieson et al., 2012; Matsui et al., 2017) and directional estimators such as SCOBO (Cai et al., 2022a) and Sign-OPT (Cheng et al., 2020). Sign-OPT also provides a stationarity analysis under smoothness and additional assumptions on gradient noise, so nonconvexity alone is not the distinction from that work. ComPO specializes the comparison mechanism to preference alignment: its oracle evaluates preferred- and dispreferred-response likelihood changes, its basic analysis exploits approximately sparse gradients, and its practical implementation uses output-layer perturbations and thresholding. Comparison and ranking feedback have also been studied in bandit optimization (Yue and Joachims, 2009; Kumagai, 2017; Ding and Zhou, 2018), Bayesian optimization (Astudillo and Frazier, 2020; Lin et al., 2022b), and RLHF (Tang et al., 2024b; Zhang and Ying, 2025). Our focus is on extracting comparison signals from low-margin offline preference pairs and combining them with unlabeled online generations for step-size control. 4

Comparison-Based Preference Alignment

2 Preliminaries We provide an overview of the setup for direct preference alignment, and recall the definition of comparison oracles and the subroutine for estimating gradients using comparison oracles that are important for designing the basic scheme of our method. We further introduce the reverse-KL and coverage notation used in online ComPO. 2.1 Direct preference alignment Modern LLMs are designed based on the Transformer architecture (Vaswani et al., 2017) and follow user prompts x ∈ V ⋆ to generate responses y ∈ V ⋆ , where V is a vocabulary of tokens. We view an LLM as a policy πθ (y|x) which assigns probabilities to responses y given prompts x. To assign probabilities to each token of y, the policy πθ operates in an auto-regressive manner as follows, πθ (y|x) =

|y| Y

πθ (yk |x, y<k ),

k=1

where θ denotes the model parameters (e.g., the parameters of the Transformer architecture) and y<k denotes the first k − 1 tokens of y. However, the generations might not be helpful, safe, or reliable, which motivates further alignment of LLMs with human preferences. We consider the direct preference learning pipeline based on pairwise preference data. Specifically, we assume access to a preference dataset D containing samples (x, y+ , y− ), where x is a prompt and (y+ , y− ) is a pair of preferred and dispreferred responses to x. This pipeline usually includes an initial supervised fine-tuning (SFT) phase, where the model is fine-tuned using the cross-entropy loss and high-quality data for specific downstream tasks. The SFT data can be either independent of D (Touvron et al., 2023), or may consist of prompts and preferred responses from D (Rafailov et al., 2023). Direct alignment methods, such as DPO (Rafailov et al., 2023), optimize the policy πθ over the preference dataset D without learning a reward model as in RLHF (Ziegler et al., 2019; Stiennon et al., 2020). This is done by minimizing a contrastive loss as follows, h i + πθ (y− |x) θ (y |x) LDPO (θ) = −E(x,y+ ,y− )∼D log σ β log ππref − β log , (1) + − (y |x) πref (y |x) where πref is the model after SFT, β is a regularization parameter, and σ : R → [0, 1] is the sigmoid function. The function LDPO relies on the log-likelihood margin between y+ and y− . Thus, DPO improves the relative likelihood margin between the two responses, rather than directly maximizing the likelihood of y+ and minimizing the likelihood of y− . During training, the likelihood of y+ might decrease, and probability mass can be shifted from y+ to responses with an opposite meaning (Pal et al., 2024; Razin et al., 2025). A possible reason is that the above objective function is not well suited for extracting information from noisy preference pairs whose preferred and dispreferred responses have small likelihood margins or are similar under model-based measures. Empirically, Razin et al. (2025) show that filtering out similar preference pairs can make DPO more effective. However, noisy preference pairs might still contain useful information that can improve the performance of LLMs. Extracting such information is challenging 5

Chen, Chen, Yin, and Lin

using a fixed margin-based loss, since maximizing the likelihood of y+ and minimizing the likelihood of y− locally does not by itself define a global alignment objective. The local information we use is comparative: a better policy should assign higher likelihood to y+ and lower likelihood to y− . This motivates us to design a new alignment method by directly leveraging the comparison signal in pairwise preference data (x, y+ , y− ) from D. 2.2 Comparison oracles and zeroth-order methods To contextualize our proposed method for aligning LLMs with human preferences, we review the definition of comparison oracles and explain how comparison oracles can be used to develop zeroth-order methods. Given a function f : Rd → R for which neither the function value nor the gradient is accessible, we define a pairwise comparison oracle Cf in its simplest form as follows, Definition 2.1 We call Cf (θ, θ′ ) : Rd × Rd → {+1, −1} a comparison oracle for function f if −1, if f (θ′ ) < f (θ), ′ Cf (θ, θ ) = +1, otherwise. In other words, when queried with θ and θ′ , the oracle Cf (·, ·) returns −1 if f (θ′ ) < f (θ) and +1 otherwise, with ties assigned to +1. The key idea behind the subroutine in Cai et al. (2022a) for estimating gradients using comparison oracles is inspired by 1-bit compressed sensing (Boufounos and Baraniuk, 2008). The goal is to recover a signal g ∈ Rd from quantized measurements yi = sign(z⊤ i g), where zi is a random perturbation vector drawn from a rotationally invariant distribution. The theoretical guarantee on the required number of perturbations to obtain an approximate signal was established in Plan and Vershynin (2012) and extended in Cai et al. (2022a). Notably, for a small perturbation radius r > 0, we have Cf (θ, θ + rzi ) = sign(f (θ + rzi ) − f (θ)) ≈ sign(z⊤ i ∇f (θ)). Here, sign(0) = +1. Thus, the comparison label yi = Cf (θ, θ +rzi ) serves as an approximate one-bit measurement of ∇f (θ). Another issue is that zeroth-order comparison-based methods can suffer from dimensiondependent iteration complexity bounds (Jamieson et al., 2012). This is expected because comparison oracles are even weaker than function-value oracles. This dimension dependence can be mitigated by exploiting sparse gradient structure (Wang et al., 2018; Golovin et al., 2020; Choromanski et al., 2019; Cai et al., 2022a,b). Indeed, we say that the function f has √ sparse gradients if ∥∇f (θ)∥1 ≤ s∥∇f (θ)∥ for all θ ∈ Rd and some s ≪ d. The above discussion gives the subroutine for estimating sparse gradients using comparison oracles. We generate m i.i.d. perturbation vectors, denoted by {zi }1≤i≤m , compute yi = Cf (θ, θ + rzi ) for all i, and solve the following optimization problem: ĝ =

argmax √

m X

yi z⊤ i g,

(2)

∥g∥1 ≤ s,∥g∥≤1 i=1

where the constraints ∥g∥1 ≤ and normalized set.

√

s and ∥g∥ ≤ 1 restrict the search to an approximately sparse

Comparison-Based Preference Alignment

In ComPO, the latent function f is viewed as an implicit alignment objective. Instead of assuming access to its function value or gradient, we use offline preference pairs to construct a comparison oracle: a nearby policy is considered better if it assigns a higher likelihood to the preferred response and a lower likelihood to the dispreferred response. 2.3 Reverse KL and local coverage The online extension of ComPO uses unlabeled policy generations for regularization, while the comparison oracle continues to use the fixed offline preference pairs. Let Pon denote the prompt distribution used for online generation, and let πref be a fixed reference policy. For the online analysis, we consider policies with a common response support and positive probabilities on that support, and assume that the relevant expectations are finite. For any such policy π, we define its sequence-level reverse KL relative to the reference by h i DRKL (π∥πref ) = Ex∼Pon [DKL (π(·|x)∥πref (·|x))] = Ex∼Pon ,y∼π(·|x) log ππ(y|x) , (3) ref (y|x) The reverse KL can be estimated using unlabeled generations from the current policy being evaluated. For τ > 0, we define the reverse-KL neighborhood of the reference policy by Πτ = {π : DRKL (π∥πref ) ≤ τ }

(4)

We let r⋆ (x, y) denote the ground-truth reward. For β > 0, we define the KL-regularized population objective by Jβ (π) = Ex∼Pon ,y∼π(·|x) [r⋆ (x, y)] − βDRKL (π∥πref ).

(5)

For a policy π, we define its implicit reward relative to πref by rbπ (x, y) = β log ππ(y|x) . ref (y|x)

(6)

Pairwise reward differences are invariant to prompt-dependent additive constants. We thus measure the accuracy through the following in-distribution pairwise error, where y1 and y2 are drawn independently from πref (·|x) conditional on x. Formally, we have err(π) = Ex∼Pon ,y1 ,y2 ∼πref (·|x) (r⋆ (x, y1 ) − r⋆ (x, y2 ) − rbπ (x, y1 ) + rbπ (x, y2 ))2 . (7) Following Song et al. (2024b), we present policy performance in terms of this in-distribution pairwise error under the local coverage condition in the following definition. Definition 2.2 The reference policy πref satisfies local reverse-KL coverage at radius κ > 0 with constant Cκ > 0 if every policy µ satisfying DRKL (µ∥πref ) ≤ κ also satisfies sup

≤ Cκ , sup πµ(y|x) ref (y|x)

x∈supp(Pon ) y∈V ⋆

where we use the convention 00 = 0. Local coverage in Definition 2.2 concerns policies within a reverse-KL neighborhood of πref , which guarantees that restricting the learned policy to Πτ can allow a performance guarantee to depend on coverage within that neighborhood. The reverse-KL constraint determines the class on which coverage is required but it does not guarantee the bounded density ratio. 7

Chen, Chen, Yin, and Lin

3 Main Results We study how to learn from noisy preference pairs that induce similar likelihoods for preferred and dispreferred responses. We first present the basic offline scheme, which replaces a first-order update driven by a predefined preference loss with a zeroth-order update driven by comparison oracles, and describe the practical offline scheme used for LLM fine-tuning. We then introduce online ComPO, which preserves the offline comparison direction and uses unlabeled current-policy generations for reverse-KL regularization. 3.1 Offline preference alignment The key idea behind ComPO is to use noisy preference pairs only to compare nearby policies. For a nonempty S ⊆ D, define 1 P ′ + + ∆+ S (θ, θ ) = |S| P(x,y+ ,y− )∈S (log πθ′ (y |x) − log πθ (y |x)) , (8) 1 ′ − − ∆− (x,y+ ,y− )∈S (log πθ′ (y |x) − log πθ (y |x)) . S (θ, θ ) = |S| We then provide the formulation of preference comparison oracle for LLM alignment below: Definition 3.1 (Preference comparison oracle) For a set S ⊆ D, the preference comparison oracle CπS (θ, θ′ ) : Rd × Rd 7→ {+1, −1} is defined by ( − ′ ′ −1, if ∆+ S (θ, θ ) > 0 and ∆S (θ, θ ) < 0, CπS (θ, θ′ ) = +1, otherwise. Thus, CπS (θ, θ′ ) = −1 means that θ′ is preferred to θ according to the likelihood comparison induced by S. When S contains one pair, this reduces to the pairwise oracle. When S is a mini-batch, the oracle uses average preferred and dispreferred likelihood changes. Given S a set of perturbations {zi }m i=1 , ComPO queries yi = Cπ (θt , θt + rzi ) for i = 1, . . . , m and applies the sparse 1-bit estimator from Eq. (2) as follows, ĝ =

argmax √

m X

yi z⊤ i g.

(9)

∥g∥1 ≤ s,∥g∥≤1 i=1

This is the only specialization of the comparison-oracle subroutine needed for offline ComPO. The following theorem establishes a best-iterate convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and oracle compatibility. Theorem 3.2 Fix a nonempty comparison set S ⊆ D and 1 ≤ s ≤ d. Suppose that there exists an ℓ-smooth function f : Rd → R, with ℓ > 0, that is bounded below and satisfies 1. For all (θ, θ′ ), we have CπS (θ, θ′ ) = −1 if f (θ′ ) < f (θ) and CπS (θ, θ′ ) = 1 otherwise. √ 2. The gradients of f are approximately sparse: ∥∇f (θ)∥1 ≤ s∥∇f (θ)∥ for all θ ∈ Rd . Let ∆ > 0 satisfy f (θ1 ) − inf θ∈Rd f (θ) ≤ ∆. For any ϵ, Λ ∈ (0, 1), we choose q 2∆ 2T T = 10ℓ∆ , η = r = 40ℓϵ√d , m = cm s log 2d , 2 ℓT , s + log Λ ϵ 8

Comparison-Based Preference Alignment

Algorithm 1 Offline ComPO: Basic Scheme 1: Input: initial parameter θ1 ∈ Rd , comparison set S ⊆ D, step size η > 0, sparsity ratio s ≪ d,

sampling radius r > 0, number of perturbations m ≥ 1, and iteration number T ≥ 1. 2: for t = 1, 2, . . . , T do 3: Draw m i.i.d. samples uniformly from the unit sphere in Rd , denoted by {zi }m i=1 . 4: Compute yi = CπS (θt , θt + rzi ) for i = 1, . . . , m. 5: Compute ĝt using Eq. (9). 6: Update θt+1 = θt − ηĝt . 7: Output: θT +1 .

Algorithm 2 Offline ComPO: Practical Scheme 1: Input: initial parameter θ1 = [θ̄; θ1o ], batches {St }T t=1 , step size γ, sampling radius r, number of

perturbations m ≥ 1, clipping thresholds λg , λ, and iteration number T ≥ 1. 2: for t = 1, 2, . . . , T do do 3: Draw m i.i.d. samples {zi }m i=1 uniformly from the unit sphere in R . St o o 4: Query yi P = Cπ ([θ̄; θt ], [θ̄; θt + rzi ]) for all i = 1, . . . , m. m 5: Set ut = i=1 yi zi . If ut ̸= 0, set ĝto = ut /∥ut ∥. Otherwise, set ĝto = 0. 6: Clip ĝto by zeroing out entries whose magnitude is less than λg . |{i:y =−1}|

i 7: Set pt = . m 8: if pt > λ then o 9: θt+1 = θto − γpt ĝto . 10: else o 11: θt+1 = θto . 12: Output: θT +1 = [θ̄; θTo +1 ].

where cm is a sufficiently large constant. Suppose that the perturbations at each iteration are drawn independently of the past and Eq. (9) is solved exactly. Then, the iterates generated by Algorithm 1 satisfy P

min ∥∇f (θt )∥ < ϵ

1≤t≤T

≥ 1 − Λ.

Consequently, the total number of preference-comparison oracle calls is bounded by 2d 2+ℓ∆ϵ−2 O 1 + ℓ∆ s log + log . s Λ ϵ2 Remark 3.3 Theorem 3.2 provides a best-iterate convergence guarantee for the basic offline scheme under the stated assumptions. Since the objective f is latent, its gradient norm is not available as a practical stopping criterion. The result nevertheless provides a theoretical benchmark: for fixed sparsity level s, the number of comparison queries depends only logarithmically on the ambient dimension. The practical implementation below approximates the basic estimator to accommodate the scale of LLM fine-tuning. Practical scheme. Applying the basic scheme to all model parameters is computationally expensive for LLMs. We therefore perturb only the output-layer weights θo ∈ Rdo and freeze the remaining parameters θ̄, so that θ = [θ̄; θo ]. We also replace the exact solution of Eq. (9) with a normalized sum of signed perturbations followed by entry-wise clipping. 9

Chen, Chen, Yin, and Lin

Algorithm 3 Online ComPO: Basic Scheme 1: Input: initial parameter θ1 ∈ Rd satisfying πθ1 ∈ Πτ , comparison set S ⊆ D, online prompt

distribution Pon , reference policy πref , step size η > 0, reverse-KL radius τ > 0, sparsity ratio s ≪ d, sampling radius r > 0, number of perturbations m ≥ 1, and iteration number T ≥ 1. 2: for t = 1, 2, . . . , T do 3: Draw m i.i.d. samples uniformly from the unit sphere in Rd , denoted by {zi }m i=1 . 4: Compute yi = CπS (θt , θt + rzi ) for i = 1, . . . , m. 5: Compute ĝt using Eq. (9). 6: Form θ̃t+1 and evaluate D̃t by Eq. (11). 7: Set θt+1 according to Eq. (12). 8: Output: θT +1 .

The practical pipeline partitions the dataset using the reference model. In particular, we define Dnoisy = (x, y+ , y− ) ∈ D : | log πref (y+ |x) − log πref (y− |x)| ≤ δmargin , (10) and let Dclean = D \ Dnoisy . The term noisy refers to this low-margin subset and does not presume that its preference labels are incorrect. We first apply a direct alignment method, such as DPO or SimPO, to Dclean and then apply Algorithm 2 to Dnoisy . For DPO in the first stage, we denote the resulting procedure by DPOclean +ComPO. 3.2 Online ComPO We introduce an online extension of ComPO that retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control. Following Song et al. (2024b), we restrict the policy to the class Πτ in Eq. (4), so that the analysis requires coverage only within this neighborhood. Since the update direction is obtained from comparisons rather than the gradient of an explicit preference loss, the basic scheme implements this restriction through a feasibility check on each candidate update. The practical scheme uses the samples from the current policy to adjust the step size. At iteration t, we compute the same comparison direction ĝt as in Algorithm 1 and form a single candidate using a fixed step size η > 0: θ̃t+1 = θt − ηĝt ,

D̃t = DRKL (πθ̃t+1 ∥πref ).

(11)

Given a reverse-KL radius τ > 0, we accept the candidate if it is feasible and otherwise leave the policy unchanged: ( θ̃t+1 , if D̃t ≤ τ, θt+1 = (12) θt , otherwise. The basic scheme evaluates the candidate policy’s reverse KL exactly. Starting from a feasible policy, the accept-or-reject rule preserves feasibility by retaining the previous iterate whenever the candidate falls outside Πτ . The following theorem establishes feasibility and relates in-distribution pairwise reward accuracy to policy performance under local coverage. 10

Comparison-Based Preference Alignment

Algorithm 4 Online ComPO: Practical Scheme 1: Input: initial parameter θ1 = [θ̄; θ1o ], preference dataset D, online prompts Xon , reference policy

πref , margin threshold δmargin , step-size scale γ > 0, damping strength ρ ≥ 0, threshold τp ≥ 0, sampling radius r > 0, number of perturbations m ≥ 1, online batch size B ≥ 1, clipping thresholds λg , λ > 0, iteration number T ≥ 1, re-sampling window n ≥ 1, and replay ratio α ∈ [0, 1]. 2: Construct Dnoisy using Eq. (10). 3: Initialize the replay buffer R ← ∅ and the current successful-batch buffer A ← ∅. 4: for t = 1, 2, . . . , T do 5: if t > 1 and (t − 1) mod n = 0 then 6: Set R ← A and A ← ∅. 7: Draw a replay indicator bt ∼ Bernoulli(α). 8: if bt = 1 and R ̸= ∅ then 9: Sample a previously successful preference mini-batch St uniformly from R. 10: else 11: Sample a new noisy preference mini-batch St ⊆ Dnoisy . 12: Draw m i.i.d. samples uniformly from the unit sphere in Rdo , denoted by {zi }m i=1 . 13: Query yi P = CπSt ([θ̄; θto ], [θ̄; θto + rzi ]) for i = 1, . . . , m. m 14: Set ut = i=1 yi zi and ĝto = ut /∥ut ∥ if ut ̸= 0; otherwise set ĝto = 0. Clip ĝto by zeroing out entries whose magnitude is less than λg . ˆ 15: Sample {x̃j }B j=1 ⊆ Xon , generate ỹj ∼ πθt (·|x̃j ), and compute dt and γt using Eq. (14)-(15). |{i:y =−1}|

i . 16: Set pt = m 17: if pt > λ then o 18: θt+1 = θto − γt pt ĝto . 19: Add the accepted preference mini-batch to the current buffer: A ← A ∪ {St }. 20: else o 21: θt+1 = θto . 22: Output: θT +1 = [θ̄; θTo +1 ].

Theorem 3.4 Fix β, τ > 0. Suppose that Algorithm 3 evaluates each candidate policy’s reverse KL exactly. Then, the generated iterates satisfy πθt ∈ Πτ for all t = 1, . . . , T + 1. If πref satisfies local reverse-KL coverage at radius τ with constant Cτ , we have sup Jβ (π) − Jβ (πθt ) ≤ Cτ

π∈Πτ

p err(πθt ),

for all t = 1, . . . , T + 1.

√ For any ϵ > 0, an iterate satisfying err(πθt ) ≤ ϵ satisfies supπ∈Πτ Jβ (π) − Jβ (πθt ) ≤ Cτ ϵ. Theorem 3.4 combines the feasibility preservation with a coverage-based performance bound following Song et al. (2024b). The reverse-KL constraint restricts the policies under consideration to Πτ , so that this guarantee requires coverage within the neighborhood rather than over the entire policy class. Within this neighborhood, smaller pairwise reward error gives a tighter performance bound. Practical scheme. While the basic scheme evaluates reverse KL at the candidate policy, the practical scheme samples from the current policy and uses a length-normalized statistic to damp the update. For independent prompts x̃j ∼ Pon and responses ỹj ∼ πθt (· | x̃j ), the 11

Chen, Chen, Yin, and Lin

sequence-level estimator b seq = 1 D t B

B X

log

πθt (ỹj |x̃j ) πref (ỹj |x̃j )

(13)

j=1

is unbiased for DRKL (πθt ∥πref ). In practice, we use dˆt = B1

B X log πθ (ỹj |x̃j )−log πref (ỹj |x̃j ) t

max{1,|ỹj |}

(14)

j=1

Length normalization changes the population quantity being estimated. In particular, dˆt is a signed statistic and its population counterpart needs not be nonnegative. We set γt =

γ , 1+ρ max{dˆt −τp ,0}

(15)

where τp is the threshold for the length-normalized statistic. As such, the online samples only change the step size, not the comparison oracle or the preference labels. We divide training into consecutive blocks of n iterations. At the start of each block after the first, the replay buffer is replaced by the mini-batches that passed the update gate pt > λ in the preceding completed block. At each iteration, with probability α, we sample uniformly from this buffer when it is nonempty; otherwise, we sample a new mini-batch from Dnoisy . Revisited mini-batches use fresh perturbations around the current parameters rather than reusing previous update directions. Here, “successful” means only that the comparison gate was passed. Section 4.3 evaluates the empirical effect of combining replay with online damping. Algorithm 4 is motivated by the principle used in Algorithm 3, but cannot be covered by Theorem 3.4. In particular, the length-normalized quantity in Eq. (14) is not the sequencelevel reverse KL in Eq. (3), and Eq. (15) does not enforce the hard constraint π ∈ Πτ . These are practical heuristics whose effect is evaluated empirically in Section 4.3.

4 Experiments We investigate the effectiveness of ComPO on aligning the LLMs. First, we evaluate offline scheme as an augmentation to DPO and its variants, where it extracts the directions from noisy preference pairs. Second, we study the offline design choices and the scaling behavior with respect to perturbations, perturbed layers and noisy pairs. Third, we evaluate online scheme, which uses unlabeled current-policy generations to damp the step through a reverseKL proxy. Unless otherwise stated, the main tables report point estimates from the reported runs and the ablation tables explicitly report variation across repeated runs. 4.1 Offline training for augmenting DPO and SimPO We identify clean and noisy preference pairs using the margin threshold δmargin = 3. For Mistral-7B models, we set r = 0.0005, m = 1600, λg = 0.00022, and λ = 0.2. For Llama-38B models and Gemma-2-9B-it, we set r = 0.00075, m = 1800, λg = 0.00008, and λ = 0.2. We use UltraFeedback1 (Cui et al., 2024) throughout the offline experiments. We initialize 1. https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized

Comparison-Based Preference Alignment

Table 1: Evaluation on AlpacaEval 2, Arena-Hard, and MT-Bench across four model configurations. LC and WR denote length-controlled win rate and raw win rate, respectively. Turn-1 and Turn-2 are the MT-Bench scores for the initial and follow-up questions. “PA” denotes the pre-alignment supervised or instruction-fine-tuned checkpoint before DPO training. Mistral-7B-Base Method

AlpacaEval 2 Arena-Hard LC (%) WR (%)

PA DPO DPOclean

WR (%)

Mistral-7B-Instruct MT-Bench

AlpacaEval 2 Arena-Hard

Turn-1 Turn-2 Avg. LC (%) WR (%)

WR (%)

MT-Bench Turn-1 Turn-2 Avg.

7.33 9.71 9.41

4.48 6.27 6.52

1.1 2.9 3.0

6.10 6.20 6.18

5.04 5.57 16.54 5.38 5.79 24.14 5.22 5.70 23.89

12.43 16.71 16.15

10.9 14.4 14.2

6.19 6.28 6.11

5.10 5.42 5.34

DPOclean +ComPO 11.66

6.55

3.2

6.22

5.32

18.32

10.5

7.78

7.63 7.69

5.77 26.17

Llama-3-8B-Base Method

AlpacaEval 2 Arena-Hard LC (%) WR (%)

PA DPO DPOclean

WR (%)

5.65 5.86 5.73

Llama-3-8B-Instruct

MT-Bench

AlpacaEval 2 Arena-Hard

Turn-1 Turn-2 Avg. LC (%) WR (%)

WR (%)

MT-Bench Turn-1 Turn-2 Avg.

3.21 4.14 4.28

7.97 10.43 9.81

4.1 12.1 12.0

6.53 6.61 6.64

5.66 5.85 6.01

6.10 24.06 6.23 32.59 6.33 32.92

23.69 31.99 32.42

20.8 22.9 22.9

8.22 8.30 8.26

7.57 7.55 7.63

7.90 7.93 7.94

DPOclean +ComPO 5.39

10.93

12.1

6.60

6.28 6.44 35.79

35.03

23.1

8.39

7.71 8.05

from the supervised fine-tuned Base and Instruct models used in Meng et al. (2024): Mistral-7B Base and Instruct2 , Llama-3-8B Base3 and Instruct4 , and Gemma-2-9B-it5 . All ComPO runs use 30 NVIDIA A40 GPUs, each with 46 GB of memory. We follow the evaluation protocol of Meng et al. (2024) and evaluate on AlpacaEval 2v0.6.6 (Li et al., 2023), Arena-Hard (Li et al., 2024), and MT-Bench (Zheng et al., 2023). For AlpacaEval 2, GPT-4 Turbo serves as both baseline and judge models. The judge compares each model response with the baseline response, and we report raw win rate (WR) and length-controlled win rate (LC) (Dubois et al., 2024). LC adjusts judged preferences for response length and a higher LC score does not by itself establish shorter responses. For Arena-Hard, the baseline is GPT-4-0314 and the judge is GPT-4 Turbo. We report WR. For MT-Bench, GPT-4 scores multi-turn Q&A responses on a 10-point scale. We report the scores for the initial question (Turn-1), the follow-up question (Turn-2), and their average. DPO with ComPO. We split the data into clean and noisy subsets using the margin criterion in Eq. (10). Starting from the SFT model, we train on all pairs to obtain DPO and on only the clean pairs to obtain DPOclean . Following Meng et al. (2024), both models are trained for one epoch. We initialize ComPO from DPOclean and run it for one epoch with 100 iterations over noisy pairs, yielding DPOclean +ComPO. We summarize the results in Table 1 and report three key observations. First, filtering low-margin pairs alone does not uniformly improve DPO: DPOclean is comparable to DPO overall and performs better only for some initializations, such as Llama-3-Instruct-8B. The log-likelihood margin therefore appears to be an imperfect proxy for pair ambiguity; richer 2. https://huggingface.co/alignment-handbook/zephyr-7b-sft-full 3. https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT 4. https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 5. https://huggingface.co/google/gemma-2-9b-it

Chen, Chen, Yin, and Lin

Table 2: Pairwise log-likelihoods in three independent trials for γ ∈ {0.1, 1}, with all other hyperparameters fixed at their default values. Each cell reports (log πθ (y+ |x), log πθ (y− |x)) after one training run; the initial values appear in the model headers. The trials use independently sampled perturbations {zi }1≤i≤m . Across the reported trials, the preferred-response log-likelihood is nondecreasing and the dispreferred-response log-likelihood is nonincreasing. Llama-3-Instruct-8B (log πθ (y+ |x), log πθ (y− |x)) = (−46.761, −47.410) γ

Trial 1

0.1 (−46.744, −47.411) 1 (−46.728, −47.520)

Trial 2

Trial 3

(−46.760, −47.411) (−46.743, −47.525)

(−46.759, −47.410) (−46.753, −47.517)

Gemma-2-9B-it (log πθ (y+ |x), log πθ (y− |x)) = (−133.122, −134.557) γ

Trial 1

Trial 2

0.1 (−133.122, −134.557) (−133.122, −134.557) 1 (−133.059, −134.562) (−133.122, −134.564)

Trial 3 (−133.121, −134.557) (−133.112, −134.565)

criteria such as the CHES score (Razin et al., 2025) may separate pairs more accurately. Nevertheless, the margin is inexpensive to compute, and ComPO extracts useful information from the pairs that it filters out. Second, gains are especially consistent in AlpacaEval 2 LC, indicating improved judged performance after adjustment for response length. We interpret these scores separately from the response-length measurements reported below. Third, ComPO uses only the first 100 noisy pairs, yet improves most model-benchmark combinations. As such, a small set of low-margin pairs can contain useful alignment information when processed through comparison oracles. The main exception is Arena-Hard for Mistral-7B-Instruct, where DPO scores 14.4 and DPOclean +ComPO scores 10.5; for the two Llama configurations, the scores are tied or nearly tied. An explanation is that Arena-Hard reports raw rather than length-controlled win rate and can therefore favor longer generations (Meng et al., 2024). For Mistral-7BInstruct, the average response length is 513 for DPO and 468 for DPOclean +ComPO. This difference is consistent with the lower Arena-Hard score and the stronger AlpacaEval 2 LC score, although it does not by itself establish causality. We also inspect whether the comparison oracle moves the likelihoods of each noisy pair in the intended direction. In Table 2, we summarize three independent trials for γ ∈ {0.1, 1} on Llama-3-Instruct-8B and Gemma-2-9B-it. For example, with Llama-3-Instruct-8B and γ = 1, the first trial changes the pair from (−46.761, −47.410) to (−46.728, −47.520): the preferred response becomes more likely, while the dispreferred response becomes less likely. Thus, for the two reported models, the oracle-based update moves the pairwise likelihoods in the desired direction or leaves them unchanged. This diagnostic is an in-training sanity check rather than a population-level performance guarantee. The thresholds λg and λ limit the coordinates and iterations on which the practical scheme updates the model. Very large step sizes can still destabilize the practical scheme, while Theorem 3.2 analyzes the step size only for the basic scheme. Section 4.3 considers adaptive step-size control based on current-policy generations. SimPO with ComPO. ComPO is not tied to DPO. We apply it directly to existing, well-tuned SimPO checkpoints (Meng et al., 2024) and use the training and evaluation configuration described at the beginning of Section 4.1. Table 3 shows that SimPO+ComPO 14

Comparison-Based Preference Alignment

Table 3: Applying ComPO to existing SimPO checkpoints across models and benchmarks. Model

Method

AlpacaEval 2 Arena-Hard LC (%) WR (%)

WR (%)

MT-Bench Turn-1 Turn-2 Avg.

SimPO 40.22 Mistral-7B-Instruct SimPO + ComPO 42.27

41.18 43.17

20.8 22.0

7.94 7.83

7.31 7.62 7.46 7.64

Llama-3-8B-Instruct

SimPO 48.71 SimPO + ComPO 49.53

43.66 45.03

36.3 37.3

7.91 7.94

7.42 7.66 7.45 7.70

Gemma-2-9B-it

SimPO 60.36 SimPO + ComPO 62.42

55.59 57.20

61.1 61.1

9.07 8.99

8.47 8.77 8.58 8.79

Table 4: Effect of the number of perturbations m on AlpacaEval 2. Entries are mean ± standard deviation over five runs, with the best run in parentheses. Perturbation (m)

800

1600

3300

5400

AlpacaEval 2-WR % 17.32 ± 0.86 (17.94) 17.50 ± 0.65 (18.32) 19.21 ± 0.58 (20.25) 19.69 ± 0.36 (20.07) AlpacaEval 2-LC % 24.72 ± 1.02 (25.12) 25.02 ± 0.91 (26.17) 25.91 ± 0.95 (27.14) 26.49 ± 0.81 (27.20)

improves both AlpacaEval 2 metrics for all three models. On Arena-Hard, it improves Mistral-7B-Instruct and Llama-3-8B-Instruct and matches Gemma-2-9B-it. The MT-Bench average also increases slightly for each model. These results show that ComPO augments other direct alignment methods without changing its original training objective. 4.2 Ablation studies Number of perturbations. The number of perturbations controls how many directions the comparison oracle evaluates. We vary m while holding the remaining hyperparameters fixed and use Mistral-7B-Instruct for this study. As m increases from 800 to 5400, the mean WR and LC improve, with diminishing gains at larger m (Table 4). This is consistent with a more accurate gradient estimate from additional perturbations, although the computation time increases. Peak memory remains unchanged because ComPO accumulates a running average rather than storing all perturbation vectors (see Line 5 of Algorithm 2). We also investigate whether ComPO scales beyond output-layer perturbations. Keeping all other settings fixed, we perturb the MLPs in layers 30–31 together with the output layer of Mistral-7B-Instruct. Table 5 uses GPT-4.1 as the Arena-Hard judge, and perturbing three layers improves all three reported metrics. The larger search space has a modest systems cost in this setup: peak GPU memory increases from 16.3 GB to 16.7 GB, and 600 perturbations take 60 seconds rather than 50 seconds. Gradient threshold and number of noisy pairs. ComPO uses the entry threshold λg to update only gradient entries with sufficiently large magnitude. We vary λg with m = 3300 on Mistral-7B-Instruct (Table 6). The strongest results occur when approximately 1%–6% of the entries are retained. Retaining many small entries or filtering almost all entries leads to lower performance. We then increase the number of noisy pairs from 100 to 300. Table 7 shows higher mean performance on both AlpacaEval 2 metrics and Arena-Hard, indicating that ComPO continues to benefit from additional low-margin pairs. 15

Chen, Chen, Yin, and Lin

Table 5: Effect of perturbing multiple layers. We report AlpacaEval 2 WR and LC and Arena-Hard WR. Entries are mean ± standard deviation over five runs, with the best run in parentheses. Layers perturbed (# params) AlpacaEval 2-WR % AlpacaEval 2-LC % Arena-Hard (GPT-4.1)-WR % 17.50 ± 0.65 (18.32) 18.19 ± 0.81 (19.38)

1 (0.13B) 3 (0.25B)

25.02 ± 0.91 (26.17) 26.00 ± 0.89 (27.09)

10.80 ± 0.21 (11.0) 11.26 ± 0.36 (11.7)

Table 6: Effect of the gradient-entry threshold λg on AlpacaEval 2. Entries are mean ± standard deviation over five runs, with the best run in parentheses. λg

4×10−5

1.8×10−4

2.2×10−4

2.5×10−4

Percentage of gradient entries updated 100% 63% 6% 1% 0.15% 15.72 ± 0.77 (16.34) 16.02 ± 0.69 (16.69) 19.02 ± 0.62 (20.15) 19.21 ± 0.58 (20.25) 16.10 ± 0.11 (16.21) AlpacaEval 2-WR % AlpacaEval 2-LC % 23.42 ± 1.03 (24.28) 24.01 ± 0.91 (25.10) 26.06 ± 0.81 (27.27) 25.91 ± 0.95 (27.14) 23.82 ± 0.23 (24.00)

Table 7: Effect of increasing the number of noisy preference pairs used by ComPO. Entries are mean ± standard deviation, with the best run in parentheses. Number of noisy pairs AlpacaEval 2-WR % AlpacaEval 2-LC % Arena-Hard (GPT-4.1)-WR % 19.21 ± 0.58 (20.25) 20.07 ± 0.99 (21.35)

100 300

25.91 ± 0.95 (27.14) 26.28 ± 0.81 (27.59)

11.02 ± 0.13 (11.2) 11.76 ± 0.30 (12.1)

Table 8: Applying ComPO directly to DPO checkpoints without training DPO only on the clean subset. AE, AH, and MT denote AlpacaEval 2, Arena-Hard, and MT-Bench, respectively. Method

AE LC (%) AE WR (%) AH (GPT-4.1) WR (%) MT Turn 1 MT Turn 2 MT Avg

DPO DPO + ComPO

24.14 27.03

16.71 20.85

10.40 11.40

6.28 7.80

5.42 7.61

5.86 7.71

DPO (clean) DPO (clean) + ComPO

23.89 27.14

16.15 20.25

10.50 11.20

6.11 7.82

5.34 7.59

5.73 7.71

Efficiency and compatibility. Full fine-tuning and LoRA-based fine-tuning (Hu et al., 2022) are common post-training choices. ComPO instead uses a lightweight update that changes only selected entries in the output layer. Figure 1 (left) shows that the chosen λg retains about 1% of the output-layer entries for Mistral-7B and Llama-3-8B. For Mistral7B, the plotted 0.13B output-layer size and 1.18% retention rate correspond to roughly 1.5 million updated parameters, or about 0.02% of the full 7B model. Except in the multi-layer ablation, parameters outside the output layer remain frozen. The comparison-based update avoids full-model backpropagation and accumulate signed perturbations without storing all perturbation vectors. Figure 1 (middle) reports a peak of approximately 23 GB per A40 GPU for Llama-3-8B ComPO; the corresponding reported peaks for DPO and SimPO are 77 GB and 69 GB on H100 GPUs. Because these measurements use different hardware, they describe practical resource requirements rather than a controlled head-to-head comparison. ComPO also parallelizes naturally. For 600 perturbations on 30 A40 GPUs, each worker processes 20 perturbations, and the master aggregates the oracle outputs and perturbation signals to form the gradient estimate (Algorithm 2). Figure 1 (right) shows that runtime increases approximately linearly with the perturbed parameter dimension across the three tested models. Except for the multi-layer ablation, perturbations are restricted to the complete lm head layer. 16

Comparison-Based Preference Alignment

30.3GB

23.1GB 20

16.3GB

0.8

5.0e-05

1.0e-04

1.5e-04

Gradient Threshold g

2.0e-04

2.5e-04

7B (Mistral)

8B (Llama)

Models

9B (Gemma)

300

0.52B

250 200

0.4

Mistral-7B: 1.18% 0.0

400 350

0.6

0.2

0.92B

Param Size (B) Run Time (sec)

Perturbing Size (B)

2.2e-4

Gemma-9B: 6.82% Llama-8B: 1.68%

Perturbing Size and Run Time

80 60

Peak GPU Memory Usage Maximum Single GPU Memory (46GB)

Memory (GB)

Non-Zero Entries (%)

Llama-8B % Gemma-9B % Mistral-7B %

8.0e-5

Run Time (sec)

Non-Zero Entries After Thresholding

100

150 100

0.13B 7B (Mistral)

50 8B (Llama)

Models

9B (Gemma)

Figure 1: (Left) Percentage of nonzero entries in the final gradient as the gradient-entry threshold λg varies. (Middle) Peak GPU memory used by ComPO for the three model families. (Right) Perturbed output-layer size and wall-clock time for completing 600 perturbations on 30 NVIDIA A40 GPUs.

Table 9: Mean ± standard deviation of the number of negative oracle outputs for the first ten noisy pairs across eight consecutive runs. Pair 1

Pair 2

Pair 3

Pair 4

Pair 5

394.25 ± 28.30 364.50 ± 14.21 369.00 ± 20.39 447.00 ± 19.87 591.00 ± 13.46 Pair 6

Pair 7

Pair 8

Pair 9

Pair 10

282.00 ± 14.98 459.25 ± 10.66 242.13 ± 15.29 311.13 ± 15.87 348.75 ± 18.59

Frequency

Distribution of Negative Oracles Returned across Noisy pairs ComPO can also be applied directly to an 1.0 PDF existing checkpoint without first training CDF 0.8 the underlying DPO model only on clean 0.6 pairs. In Table 8, we start from DPO checkpoints trained on the full preference 0.4 dataset and then apply ComPO with m = 0.2 3300. The resulting gains are comparable 0.0 to those obtained from DPOclean +ComPO. 225 275 325 375 425 475 525 575 625 Number of Negative Oracles This supports a practical workflow in which a user starts from a publicly available aligned model and refines it with task- Figure 2: Empirical and cumulative distributions of the number of negative oracle outputs specific, potentially noisy preference data across noisy pairs. The dashed line marks using sparse output-layer updates and modthe threshold used for Mistral-7B-Base. est GPU memory. Threshold

Successful perturbations and clipping threshold λ. In addition to the entry-level threshold λg , ComPO uses the clipping threshold λ > 0 to discard an update when too few perturbations return successful comparison-oracle signals. Figure 2 shows the empirical distribution of the number of negative oracle outputs k = |{i : yi = −1}| across noisy pairs for Mistral-7B-Base. The threshold removes the low-count tail by skipping updates with a small fraction of favorable perturbations. Table 9 further shows that this count remains in a similar range for a fixed pair across eight independent runs. Together, these results indicate that the amount of usable oracle feedback is reproducible and that clipping avoids poorly supported updates. 17

Chen, Chen, Yin, and Lin

Table 10: Evaluation on the GPT-4.1 configurations of AlpacaEval 2 and Arena-Hard. LC and WR denote length-controlled and raw win rates. PA denotes the pre-alignment supervised or instructionfine-tuned checkpoint. “+RKL” uses the length-normalized damping rule in Algorithm 4 and the “+resampling” adds replay to that same online variant. Qwen3-4B-Base Method

LC (%) WR (%)

PA DPO DPO+ComPO

Llama-3.2-3B-Instruct

Gemma-3-4B-it

AlpacaEval 2 Arena-Hard AlpacaEval 2 Arena-Hard AlpacaEval 2 Arena-Hard WR (%)

LC (%) WR (%)

WR (%)

LC (%) WR (%)

WR (%)

12.70 15.28 16.20

13.12 15.54 16.27

16.2 29.3 30.8

11.16 11.72 12.35

11.83 12.08 12.50

9.8 11.6 11.9

34.54 38.30 40.00

56.20 57.87 58.57

54.8 56.9 57.7

DPO+ComPO (online) + RKL 17.43 + resampling 18.57

17.74 17.95

31.4 32.6

12.70 13.05

13.23 13.85

12.4 12.8

42.07 42.55

60.40 60.93

63.3 63.7

4.3 Online training We evaluate online ComPO in Algorithm 4. It keeps the offline comparison direction and uses unlabeled samples to compute the length-normalized statistic in Eq. (14). This statistic adjusts the step size through the soft-damping rule in Eq. (15). The implementation is a heuristic approximation to the basic scheme in Algorithm 3. Indeed, it does not evaluate the proposed next policy or enforce the hard sequence-level reverse-KL constraint analyzed in Theorem 3.4. For the replay buffer, we set the window length to n = 50. For Qwen3-4B-Base, we use r = 0.0008, m = 1800, and λg = 0.000085. For Gemma3-4B-it, we use r = 0.00045, m = 1800, and λg = 0.000075. We evaluate Qwen3-4BBase6 , Llama-3.2-3B-Instruct7 , and Gemma-3-4B-it8 using the GPT-4.1 configurations of AlpacaEval 2 and Arena-Hard. Unless stated otherwise, the remaining training settings follow the offline protocol in Section 4.1. In Table 10, we compare offline ComPO, ComPO with online damping, and ComPO with both damping and replay. Relative to offline ComPO, damping improves AlpacaEval 2 LC, AlpacaEval 2 WR, and Arena-Hard WR by 1.23, 1.47, and 0.6 percentage points for Qwen3-4B-Base; 0.35, 0.73, and 0.5 points for Llama-3.2-3B-Instruct; and 2.07, 1.83, and 5.6 points for Gemma-3-4B-it. Adding replay improves all three reported metrics for each model. These comparisons support the empirical benefit of the combined procedure in the tested configurations, without identifying a separate variance-reduction mechanism.

5 Conclusion We propose a new zeroth-order preference alignment method based on comparison oracles and show that it can improve large language models (LLMs) using noisy preference pairs for which the reference policy assigns similar likelihoods to preferred and dispreferred responses. The key idea is to use such pairs as comparison signals rather than directly optimizing a preference loss on them. Experimental results on multiple models and benchmarks show that ComPO improves existing direct alignment methods, with pair-level diagnostics pro6. https://huggingface.co/Qwen/Qwen3-4B-Base 7. https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct 8. https://huggingface.co/google/gemma-3-4b-it

Comparison-Based Preference Alignment

viding evidence consistent with mitigating likelihood displacement. These results highlight the importance of designing specialized methods for preference pairs with small likelihood margins, complementing the recent findings of Razin et al. (2025). The extension in this journal version is online ComPO, where offline noisy preference pairs continue to determine the comparison direction, and unlabeled generations from the current policy provide reverse-KL regularization. We establish feasibility and a coveragebased performance bound for the basic constrained scheme and evaluate damping and replay in the practical implementation. Future directions include extending our approach to other settings (Yuan et al., 2024; Xu et al., 2024b; Tajwar et al., 2024; Guo et al., 2024; Chen and Chen, 2026) and applying it to other tasks, including reasoning (Pang et al., 2024; Chen et al., 2025c) and diffusion model alignment (Wallace et al., 2024).

Acknowledgement We sincerely appreciate Buzz High Performance Computing (https://www.buzzhpc.ai, [email protected]) for providing computational resources and support for this work. Tianyi Lin gratefully acknowledges financial support through a start-up grant and an early career scholarship support grant at Columbia University.

Chen, Chen, Yin, and Lin

References J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. ArXiv Preprint: 2303.08774, 2023. A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In COLT, pages 28–40, 2010. R. Akrour, M. Schoenauer, and M. Sebag. Preference-based policy learning. In ECML PKDD, pages 12–27, 2011. A. Amini, T. Vieira, and R. Cotterell. Direct preference optimization with an offset. In ACL, pages 9954–9972, 2024. R. Astudillo and P. Frazier. Multi-attribute Bayesian optimization with interactive preference learning. In AISTATS, pages 4496–4507, 2020. M. G. Azar, Z. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pages 4447–4455, 2024. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv Preprint: 2204.05862, 2022. P. T. Boufounos and R. G. Baraniuk. 1-bit compressive sensing. In CISS, pages 16–21. IEEE, 2008. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, pages 1877–1901, 2020. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv Preprint: 2303.12712, 2023. R. Busa-Fekete, B. Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference-based racing algorithm. Machine learning, 97:327–351, 2014. H. Cai, D. McKenzie, W. Yin, and Z. Zhang. A one-bit, comparison-based gradient estimator. Applied and Computational Harmonic Analysis, 60:242–266, 2022a. H. Cai, D. McKenzie, W. Yin, and Z. Zhang. Zeroth-order regularized optimization (ZORO): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2):687–714, 2022b. S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C-R. Ségerie, M. Carroll, A. Peng, P. J. K. 20

Comparison-Based Preference Alignment

Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. D. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=bx24KpJ4Eb. H. Chen, G. He, L. Yuan, G. Cui, H. Su, and J. Zhu. Noise contrastive alignment of language models with explicit rewards. In NeurIPS, pages 117784–117812, 2024. H. Chen, H. Zhao, H. Lam, D. Yao, and W. Tang. MallowsPO: Fine-tune your LLM with preference dispersions. In ICLR, 2025a. URL https://openreview.net/forum? id=d8cnezVcaW. P. Chen and X. Chen. Two-fidelity best-action identification for stochastic minimax tree. ArXiv Preprint: 2606.01708, 2026. P. Chen, X. Chen, W. Yin, and T. Lin. ComPO: Preference alignment via comparison oracles. In NeurIPS, pages 121962–121995, 2025b. P. Chen, X. Li, Z. Li, X. Chen, and T. Lin. Stepwise guided policy optimization: Coloring your incorrect reasoning in GRPO. Transactions on Machine Learning Research (TMLR), 2025c. ISSN 2835-8856. URL https://openreview.net/forum?id=ALnVAqtshR. P. Chen, X. Li, X. Chen, and T. Lin. Reward-free alignment for conflicting objectives. In ICML, 2026a. URL https://openreview.net/forum?id=vSzRJyg6k0. P. Chen, X. Li, Z. Li, W. Yin, X. Chen, and T. Lin. Exploration vs exploitation: Rethinking RLVR through clipping, entropy, and spurious reward. In ICLR, 2026b. URL https: //openreview.net/forum?id=sE8DCSJTzd. X. Chen, S. Liu, K. Xu, X. Li, X. Lin, M. Hong, and D. Cox. ZO-AdaMM: zeroth-order adaptive momentum method for black-box optimization. In NeurIPS, pages 7204–7215, 2019. M. Cheng, S. Singh, P. H. Chen, P-Y. Chen, S. Liu, and C-J. Hsieh. Sign-OPT: A queryefficient hard-label adversarial attack. In ICLR, 2020. URL https://openreview.net/ forum?id=SklTQCNtvS. K. Choromanski, A. Pacchiano, J. Parker-Holder, Y. Tang, and V. Sindhwani. From complexity to simplicity: Adaptive ES-Active subspaces for blackbox optimization. In NeurIPS, pages 10299–10309, 2019. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NeurIPS, pages 4302–4310, 2017. 21

Chen, Chen, Yin, and Lin

E. Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In NeurIPS, pages 5032–5043, 2018. G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with scaled AI feedback. In ICML, pages 9722–9744, 2024. Y-X. Ding and Z-H. Zhou. Preference based adaptation for learning objectives. In NeurIPS, pages 7839–7848, 2018. H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. RAFT: Reward ranked fine-tuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. URL https://openreview.net/ forum?id=m7p5O7zblY. H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. RLHF workflow: From reward modeling to online RLHF. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=a13aYUU9eU. Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In NeurIPS, pages 30039–30069, 2023. Y. Dubois, P. Liang, and T. Hashimoto. Length-controlled AlpacaEval: A simple debiasing of automatic evaluators. In COLM, 2024. URL https://openreview.net/forum?id= CybBmzWBX0. J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zeroorder convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015. K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Model alignment as prospect theoretic optimization. In ICML, pages 12634–12651, 2024. A. D. Flaxman, A. T. Kalai, and H. B. McMahan. Online convex optimization in the bandit setting: Gradient descent without a gradient. In SODA, pages 385–394, 2005. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In ICML, pages 10835–10866, 2023. S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. D. Golovin, J. Karro, G. Kochanski, C. Lee, X. Song, and Q. Zhang. Gradientless descent: High-dimensional zeroth-order optimization. In ICLR, 2020. URL https://openreview. net/forum?id=Skep6TVYDB. S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, et al. Direct language model alignment from online AI feedback. arXiv preprint arXiv:2402.04792, 2024. 22

Comparison-Based Preference Alignment

J. Hong, N. Lee, and J. Thorne. ORPO: Monolithic preference optimization without reference model. In EMNLP, pages 11170–11189, 2024. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL https: //openreview.net/forum?id=nZeVKeeFYf9. F. Huang, S. Gao, J. Pei, and H. Huang. Accelerated zeroth-order and first-order momentum methods from mini to minimax optimization. Journal of Machine Learning Research, 23 (36):1–70, 2022. K. G. Jamieson, R. Nowak, and B. Recht. Query complexity of derivative-free optimization. In NeurIPS, pages 2672–2680, 2012. K. Ji, Z. Wang, Y. Zhou, and Y. Liang. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In ICML, pages 3100–3109, 2019. S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang. Is stack overflow obsolete? an empirical study of the characteristics of ChatGPT answers to stack overflow questions. In CHI, pages 1–17, 2024. K. Kim, A. Seo, H. Liu, J. Shin, and K. Lee. Margin matching preference optimization: Enhanced model alignment with granular feedback. In EMNLP, pages 13554–13570, 2024. G. Kornowski and O. Shamir. An algorithm with optimal dimension-dependence for zeroorder nonsmooth nonconvex stochastic optimization. Journal of Machine Learning Research, 25(122):1–14, 2024. W. Kumagai. Regret analysis for continuous dueling bandit. In NeurIPS, pages 1488–1497, 2017. T. Li, W-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. ArXiv Preprint: 2406.11939, 2024. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-following models. https: //github.com/tatsu-lab/alpaca_eval, 5 2023. X. Lian, H. Zhang, C-J. Hsieh, Y. Huang, and J. Liu. A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order. In NeurIPS, pages 3062–3070, 2016. T. Lin, Z. Zheng, and M. I. Jordan. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. In NeurIPS, pages 26160–26175, 2022a. Z. J. Lin, R. Astudillo, P. Frazier, and E. Bakshy. Preference exploration for efficient Bayesian optimization with multiple outcomes. In AISTATS, pages 4235–4258, 2022b. S. Liu, B. Kailkhura, P-Y. Chen, P. Ting, S. Chang, and L. Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In NeurIPS, pages 3731–3741, 2018. 23

Chen, Chen, Yin, and Lin

T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. In ICLR, 2024a. URL https: //openreview.net/forum?id=xbjSwwrQOe. T. Liu, Z. Qin, J. Wu, J. Shen, M. Khalman, R. Joshi, Y. Zhao, M. Saleh, S. Baumgartner, J. Liu, et al. LiPO: Listwise preference optimization through learning-to-rank. In NAACL, page To appear, 2025. Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang, J. Blanchet, and Z. Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In NeurIPS, pages 138663–138697, 2024b. S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In NeurIPS, pages 53038–53075, 2023. K. Matsui, W. Kumagai, and T. Kanamori. Parallel distributed block coordinate descent methods based on pairwise comparison oracle. Journal of Global Optimization, 69:1–21, 2017. Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with a reference-free reward. In NeurIPS, pages 124198–124235, 2024. Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, pages 27730–27744, 2022. A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. ArXiv Preprint: 2402.13228, 2024. R. Y. Pang, W. Yuan, H. He, K. Cho, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. In NeurIPS, pages 116617–116637, 2024. R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In ACL, pages 4998–5017, 2024. Y. Plan and R. Vershynin. Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach. IEEE Transactions on Information Theory, 59(1): 482–494, 2012. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, pages 53728–53741, 2023. R. Rafailov, Y. Chittepu, R. Park, H. Sikchi, J. Hejna, W. B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. In NeurIPS, pages 126207–126242, 2024a. 24

Comparison-Based Preference Alignment

R. Rafailov, J. Hejna, R. Park, and C. Finn. From $r$ to $qˆ*$: Your language model is secretly a Q-function. In COLM, 2024b. URL https://openreview.net/forum?id= kEVcNxtqXk. N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In ICLR, 2025. URL https://openreview.net/forum?id=uaMSBJDnRv. Y. Ren and D. J. Sutherland. Learning dynamics of LLM finetuning. In ICLR, 2025. URL https://openreview.net/forum?id=tPNHOoZFl9. T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. ArXiv Preprint: 1703.03864, 2017. O. Shamir. An optimal algorithm for bandit and zero-order convex optimization with twopoint feedback. Journal of Machine Learning Research, 18(1):1703–1713, 2017. R. Shi, R. Zhou, and S. S. Du. The crucial role of samplers in online direct preference optimization. In ICLR, 2025. URL https://openreview.net/forum?id=F6z3utfcYw. P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in RLHF. In COLM, 2024. URL https://openreview.net/forum?id= G8LaO1P0xv. F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for human alignment. In AAAI, pages 18990–18998, 2024a. Y. Song, G. Swamy, A. Singh, J. Bagnell, and W. Sun. The importance of online data: Understanding preference fine-tuning via coverage. In NeurIPS, pages 12243–12270, 2024b. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In NeurIPS, pages 3008–3021, 2020. F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In ICML, pages 47441–47474, 2024. Y. Tang, Z. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. In ICML, pages 47725–47742, 2024a. Z. Tang, D. Rybin, and T-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. In ICLR, 2024b. URL https://openreview.net/ forum?id=TVDUVpgu9s. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv Preprint: 2302.13971, 2023. 25

Chen, Chen, Yin, and Lin

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS, pages 6000–6010, 2017. B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization. In CVPR, pages 8228–8238, 2024. Y. Wang, S. Du, S. Balakrishnan, and A. Singh. Stochastic zeroth-order optimization in high dimensions. In AISTATS, pages 1356–1365, 2018. T. Xiao, Y. Yuan, H. Zhu, M. Li, and V. G. Honavar. Cal-DPO: Calibrated direct preference optimization for language model alignment. In NeurIPS, pages 114289–114320, 2024. T. Xie, D. J. Foster, A. Krishnamurthy, C. Rosset, A. H. Awadallah, and A. Rakhlin. Exploratory preference optimization: Harnessing implicit q ∗ -approximation for sampleefficient RLHF. In ICLR, 2025. URL https://openreview.net/forum?id=QYigQ6gXNw. W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In ICML, pages 54715–54754, 2024. H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In ICML, pages 55204–55224, 2024a. S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. In ICML, pages 54983–54998, 2024b. H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. RRHF: Rank responses to align language models with human feedback. In NeurIPS, pages 10935–10950, 2023. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, B. Shan, Z. Liu, J. Deng, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing LLM reasoning generalists with preference trees. In ICLR, 2025. URL https://openreview.net/forum? id=2ea5TNVR0c. W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston. Self-rewarding language models. In ICML, pages 57905–57923, 2024. Y. Yue and T. Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In ICML, pages 1201–1208, 2009. Q. Zhang and L. Ying. Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference. In ICLR, 2025. URL https://openreview.net/ forum?id=cmYScmfu4Q. S. Zhang, Z. Liu, B. Liu, Y. Zhang, Y. Yang, Y. Liu, L. Chen, T. Sun, and Z. Wang. Rewardaugmented data enhances direct preference alignment of LLMs. In ICLR Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025. URL https: //openreview.net/forum?id=bpSD3IOgyS. 26

Comparison-Based Preference Alignment

Y. Zhang, P. Li, J. Hong, J. Li, Y. Zhang, W. Zheng, P-Y. Chen, J. D. Lee, W. Yin, M. Hong, et al. Revisiting zeroth-order optimization for memory-efficient LLM finetuning: a benchmark. In ICML, pages 59173–59190, 2024. H. Zhao, G. I. Winata, A. Das, S-X. Zhang, D. Yao, W. Tang, and S. Sahu. RainbowPO: A unified framework for combining improvements in preference optimization. In ICLR, 2025. URL https://openreview.net/forum?id=trKee5pIFv. Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv Preprint: 2305.10425, 2023. L. Zheng, W-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, and E. P. Xing. Judging LLM-as-a-Judge with MT-bench and Chatbot Arena. In NeurIPS, pages 46595–46623, 2023. B. Zhu, M. I. Jordan, and J. Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In ICML, pages 43037–43067, 2023. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. ArXiv Preprint: 1909.08593, 2019.

Chen, Chen, Yin, and Lin

Appendix A. Further Related Work We make additional comments on other topics, including preference learning methods, the analysis of preference learning methods, zeroth-order optimization methods, likelihood displacement, and learning from noisy preference data. For an overview of preference learning methods and open problems in RLHF, we refer to the recent survey (Casper et al., 2023). More discussion on preference learning methods. The lack of explicit reward models in DPO (Rafailov et al., 2023) is known to make its performance depend strongly on the size and quality of offline preference pairs. To address this limitation, subsequent works proposed to augment preference data using a trained SFT policy (Zhao et al., 2023) or a refined SFT policy with rejection sampling (Liu et al., 2024a). The DPO loss was also extended to a token-level MDP (Rafailov et al., 2024b), where the transition is deterministic, i.e., the next state is determined once the current state and action are chosen, which naturally covers the fine-tuning of autoregressive LLMs. Azar et al. (2024) further generalized DPO to a wider class of RL problems without explicitly introducing a reward function. Instead of maximizing a reward in a KL-constrained problem, they proposed to optimize a general non-decreasing function of the ground-truth population-level preference probability. There are also several other DPO variants (Ethayarajh et al., 2024; Park et al., 2024; Xu et al., 2024a; Meng et al., 2024; Chen et al., 2025a; Zhao et al., 2025). For example, Ethayarajh et al. (2024) aligned the policy with preferences using a prospect-theoretic loss, Tang et al. (2024a) optimized a general loss instead of the log-likelihood loss, and Meng et al. (2024) aligned the reward function in the preference optimization objective with the generation metric. Dong et al. (2024) and Xiong et al. (2024) proposed to generate human feedback in an online fashion to mitigate distribution shift and over-optimization. There has also been an attempt to understand the theoretical performance of DPO (Azar et al., 2024), although this analysis mainly focuses on the population-level objective rather than finite-sample policyoptimality or sample-complexity guarantees. Chen et al. (2026a) also extends DPO-style direct alignment to multiple-objective setup via a novel conflict-averse formulation. Analysis of preference learning methods. In this context, Zhu et al. (2023) formulated RLHF as a contextual bandit problem and proved the convergence of the maximum likelihood estimator. Xiong et al. (2024) showed the benefits of KL regularization for the sample complexity of online exploration in DPO. Xie et al. (2025) studied online exploration using KL-regularized Markov decision processes and proved a sample-complexity guarantee for an exploration bonus. Liu et al. (2024b) investigated the issue of over-optimization and proved finite-sample guarantees. Song et al. (2024b) conducted a rigorous analysis through the lens of dataset coverage to differentiate offline DPO and online RLHF. Recently, several works have reported faster convergence rates for online reward maximization in RL by exploiting the structure induced by KL regularization. For example, Shi et al. (2025) studied the tabular softmax parametrization setting and established quadratic convergence results. Zeroth-order optimization methods. The idea of zeroth-order optimization is to approximate a gradient using either a one-point estimator (Flaxman et al., 2005) or a two-point estimator (Agarwal et al., 2010; Ghadimi and Lan, 2013; Duchi et al., 2015; Shamir, 2017; Nesterov and Spokoiny, 2017), where the latter approach often achieves better finite-time convergence guarantees. Despite the rapid development of two-point-based gradient-free 28

Comparison-Based Preference Alignment

methods, much of the work focuses on convex optimization (Duchi et al., 2015; Shamir, 2017; Wang et al., 2018) and smooth nonconvex optimization (Nesterov and Spokoiny, 2017; Ghadimi and Lan, 2013; Lian et al., 2016; Liu et al., 2018; Chen et al., 2019; Ji et al., 2019; Huang et al., 2022). Convergence guarantees have been obtained in both nonsmooth convex settings (Duchi et al., 2015; Shamir, 2017) and smooth nonconvex settings (Ghadimi and Lan, 2013; Nesterov and Spokoiny, 2017). Additional regularity conditions, e.g., a finite-sum structure, allow variance-reduction techniques to be used (Liu et al., 2018; Chen et al., 2019; Ji et al., 2019), and sharp convergence guarantees are obtained in Huang et al. (2022). Very recently, zeroth-order optimization methods have been developed for nonsmooth nonconvex optimization with solid theoretical guarantees (Lin et al., 2022a; Kornowski and Shamir, 2024). In another direction, zeroth-order optimization methods were extended to the RL setting and have achieved empirical success as scalable alternatives to classic methods such as Q-learning and policy gradient methods (Salimans et al., 2017; Conti et al., 2018). This strategy has also been applied in preference-based RL (Akrour et al., 2011; Busa-Fekete et al., 2014) and adopted for LLM fine-tuning (Malladi et al., 2023; Zhang et al., 2024). In these settings, the loss function can be explicitly estimated or calculated and thus can be queried to construct the gradient estimator. By contrast, our method and the methods of Tang et al. (2024b) and Zhang and Ying (2025) are developed based on comparison oracles or ranking oracles, where even noisy estimates of loss-function values are not accessible. Likelihood displacement. We provide a brief overview of proposed explanations for likelihood displacement. Indeed, several works claimed that samples with similar preferred and dispreferred responses are responsible for likelihood displacement (Pal et al., 2024; Tajwar et al., 2024; Razin et al., 2025), although the similarities were measured using different metrics. Other proposed reasons include effects of the initial SFT model (Rafailov et al., 2024b), the presence of multiple training samples and limited model capacity (Tajwar et al., 2024), and the squeezing effect (Ren and Sutherland, 2025). Recently, Razin et al. (2025) conducted a thorough investigation to understand the causes of likelihood displacement, and their results suggest that samples with similar preferred and dispreferred responses might contribute more than others. Regarding the implications of likelihood displacement, previous works found that DPO tends to degrade performance on math and reasoning (Pal et al., 2024; Pang et al., 2024; Meng et al., 2024; Yuan et al., 2025). Indeed, only a few responses are correct, and likelihood displacement can have adverse effects on correct alignment. Learning from noisy preference data. ComPO addresses low-margin preference pairs selected by Eq. (10). This setting is related to, but distinct from, learning with corrupted preference labels (Amini et al., 2024; Xiao et al., 2024). From this perspective, ComPO is not intended as a direct replacement for existing methods, but rather as a complementary and modular component that enhances their robustness. Moreover, learning from corrupted preference data has been studied in prior works, including those leveraging reward scores through conditional DPO (Kim et al., 2024; Zhang et al., 2025). Conditional DPO modifies the DPO objective by conditioning on reward scores and solves the resulting problem via gradient-based methods, and it can be combined with ComPO in a way similar to SimPO+ComPO as in our work. Apart from noisy labels, Chen et al. (2026b) also theoretically analyze the impact false-positive and false-negative labels in online LLM RL, which is complementary to the mis-labeled preference pairs. 29

Chen, Chen, Yin, and Lin

Appendix B. Missing Proofs We present several technical lemmas and use them to prove Theorem 3.2 and Theorem 3.4. B.1 Technical lemmas For the offline analysis, we fix a nonempty set S ⊆ D and impose the smoothness, gradient sparsity, and oracle compatibility (see Theorem 3.2) throughout this subsection. We use sign(0) = +1. At any point with ∇f (θ) ̸= 0, we write yi = CπS (θ, θ + rzi ),

∇f (θ) ḡ = ∥∇f (θ)∥ ,

ȳi = sign(z⊤ i ḡ).

Thus, the oracle compatibility guarantees yi = sign(f (θ +rzi )−f (θ)). The next proposition adapts the one-bit estimation framework (Plan and Vershynin, 2012; Cai et al., 2022a) to errors that might depend on the perturbation directions but are localized near directions orthogonal to the target. √ Proposition B.1 Let 1 ≤ s ≤ d and let ḡ ∈ Rd satisfy ∥ḡ∥1 ≤ s and ∥ḡ∥ = 1. Suppose d that (zi , yi )m i=1 are i.i.d., where zi is uniform on the unit sphere in R and yi ∈ {−1, +1}, 1√ ⊤ and yi = sign(z⊤ i ḡ) almost surely whenever |zi ḡ| > 40 d . Then, we define ĝ ∈

argmax √

m X

yi z⊤ i g.

∥g∥1 ≤ s,∥g∥≤1 i=1

2 for a sufficiently large constant cm , we For any δ ∈ (0, 1), if m ≥ cm s log 2d s + log δ have P ∥ĝ − ḡ∥ ≤ 12 ≥ 1 − δ. Proof For the case of d = 1, we have s = 1 and zi , ḡ ∈ {−1, +1}. Since |z⊤ i ḡ| = 1, the assumption on the labels implies yi = z⊤ ḡ almost surely. Thus, y z = ḡ almost surely, and i i i the definition of ĝ gives ĝ = ḡ. For the case of d ≥ 2, we define K = {g ∈ Rd : ∥g∥1 ≤

√

s, ∥g∥ ≤ 1},

1 Fm (g) = m

m X

yi z ⊤ i g.

i=1

Since ḡ ∈ K and ĝ maximizes Fm over K, it suffices to show, with probability at least 1 − δ, that Fm (ḡ) > Fm (g) for every g ∈ K with ∥g − ḡ∥ > 12 . For simplicity, we let (z, y) have the same distribution as (zi , yi ). The key decomposition is given by 1 m

m X

m X 1 yi zi = E[sign(z⊤ ḡ)z] + E[(y − sign(z⊤ ḡ))z] + m yi zi − E[yz] . | {z } i=1 A | i=1 {z } B

We set κ = E[|z⊤ ḡ|] and obtain from rotational invariance that E[sign(z⊤ ḡ)z] = κḡ. Thus, for every g ∈ K, we have Fm (ḡ) − Fm (g) = κ(1 − ḡ⊤ g) + A⊤ (ḡ − g) + B ⊤ (ḡ − g).

(16)

In what follows, we write r = ∥g − ḡ∥ and prove that r > 21 implies Fm (ḡ) − Fm (g) > 0. 30

Comparison-Based Preference Alignment

First Term. we have

Since ∥ḡ∥ = 1 and ∥g∥ ≤ 1, we have 1 − ḡ⊤ g = 21 (r2 + 1 − ∥g∥2 ) ≥ r2 . Thus, 2

κ(1 − ḡ⊤ g) ≥ κr2 .

(17)

In addition, we prove a lower bound on κ. Indeed, the spherical marginal distribution yields Γ( d )

κ = √πΓ(2d+1 ) . Using the log-convexity of the gamma function, we have 2

2 d d 2 Γ( d+1 ≤ Γ( d2 )Γ( d+2 2 ) 2 ) = 2 Γ( 2 ) . which implies the desired bound κ ≥

2 πd .

Second Term. The key is to prove ∥A∥ ≤ κ5 . Indeed, we set q = z⊤ ḡ. By assumption, y − sign(q) is 0 almost surely outside {|q| ≤ 401√d } and is bounded by 2. The density of q is Γ( d )

d−3

h2 (q) = √1

≤ √ 1 2 2

hd (q) = √πΓ(2d−1 ) (1 − q 2 ) 2 defined on q ∈ (−1, 1). We claim that this density is bounded 2 √ by d if |q| ≤ 401√d . Indeed, we have π

1−q

1−a /2

√

hd (q) ≤ hd (0) ≤

This implies P(|q| ≤ 401√d ) ≤ 2 ·

√

d−1 2π ≤

√

d for d ≥ 3.

1 d · 401√d = 20 .

Since the component of z orthogonal to ḡ is rotationally symmetric and has squared norm 1 − q 2 conditioned on q, we have q 2 ⊤ √1 E[|v ⊤ z| | q] ≤ |v ⊤ ḡ||q| + 1−q d−1 ∥v − (v ḡ)ḡ∥ ≤ |q| + d−1 for every unit vector v. It follows that 1 ⊤ ⊤ P |q| ≤ 401√d |v A| ≤ 2E |v z|1{|q|≤ 1√ } ≤ 2 401√d + √d−1 40 d q a 2 1 ≤ 4a √d + √d−1 < 51 πd ≤ κ5 . Taking the supremum over all unit vectors v yields the desired result. Thus, we have b⊤ (ḡ − g) ≥ − κr 5 .

(18)

may depend on zi . Independence across i yields P(|C ⊤ v| > h) ≤ 2 exp(−c0 mdh2 ) for any h > 0 where c0 > 0 is a universal constant. 31

Chen, Chen, Yin, and Lin

We define Ω = sup∥v∥≤1,| supp(v)|≤⌈s⌉ |C ⊤ v|. For each coordinate support J of size ⌈s⌉, we take a 21 -net NJ of its unit sphere with at most 5⌈s⌉ points. The net approximation gives Ω ≤ 2 max|J|=⌈s⌉ maxv∈NJ |C ⊤ v|. A union bound therefore yields d P(Ω > 2h) ≤ 2 5⌈s⌉ exp(−c0 mdh2 ). ⌈s⌉ Since s ≤ ⌈s⌉ ≤ 2s and ⌈s⌉ ≤ d, we have d ⌈s⌉ ≤ c1 s log( 2d log 5 s ). ⌈s⌉ This implies, with probability at least 1 − δ, we have q Ω ≤ c2 s log(2d/s)+log(2/δ) , md where c2 is a universal constant. To extend this bound to K, we fix g ∈ K, arrange its coordinates in decreasing magnitude, and partition them into consecutive blocks I1 , I2 , . . . of size ⌈s⌉, with the last block ∥gI ∥1 possibly smaller. For every j ≥ 2, we have ∥gIj ∥ ≤ √j−1 which implies ⌈s⌉

∥g∥1 ≤ 2. ∥gIj ∥ ≤ ∥g∥ + √ ⌈s⌉

Since each block is supported on at most ⌈s⌉ coordinates, we have |C ⊤ g| ≤ Ω( 2S. It follows that, on the same event, we have q ⊤ sup |C g| ≤ 2c2 s log(2d/s)+log(2/δ) . md

j ∥gIj ∥) ≤

g∈K

q 2 2 and m ≥ cm s log 2d for a sufficiently Combining this inequality with κ ≥ πd s + log δ large constant cm yields the desired result. Since ḡ and g belong to K, we have κ C ⊤ (ḡ − g) ≥ − 40 .

(19)

End. On the event established above, Eq. (17), Eq. (18) and Eq. (19) hold simultaneously for every g ∈ K. Since r > 12 , we have 2

κ κ Fm (ḡ) − Fm (g) ≥ κr2 − κr 5 − 40 = 40 (2r − 1)(10r + 1) > 0.

Thus, every feasible vector farther than 12 from ḡ has a strictly smaller empirical objective than ḡ and cannot be a maximizer. In other word, ∥ĝ − ḡ∥ ≤ 12 on an event of probability at least 1 − δ. This completes the proof. The next two lemmas give the descent inequality used to prove the convergence guarantee. Lemma B.2 Suppose that ∥∇f (θ)∥ > 2ϵ and r = 40ℓϵ√d . Then, for {zi }m i=1 drawn uniformly ∇f (θ) from the unit sphere in Rd , we have yi = ȳi with yi = CπS (θ, θ+rzi ) and ȳi = sign z⊤ i ∥∇f (θ)∥ ∇f (θ) 1√ whenever z⊤ i ∥∇f (θ)∥ > 40 d .

Comparison-Based Preference Alignment

Proof Since f is ℓ-smooth and ∥zi ∥ = 1, we have 2

ℓr |f (θ + rzi ) − f (θ) − rz⊤ i ∇f (θ)| ≤ 2 . ϵ√ ϵ 1√ ⊤ ∇f (θ) By the choice of r, we have ℓr 2 = 80 d . Since ∥∇f (θ)∥ > 2 and zi ∥∇f (θ)∥ > 40 d , we have 2

ℓr r|z⊤ i ∇f (θ)| > 2 . Putting these pieces together yields

yi = sign(f (θ + rzi ) − f (θ)) = sign(z⊤ i ∇f (θ)) = ȳi . This completes the proof.

Lemma B.3 Under the stated assumptions, we let T ≥ 1, η > 0, ϵ > 0, and Λ ∈ (0, 1), and set r = 40ℓϵ√d . Suppose that Algorithm 1 uses independent perturbations at every iteration 2T and solves Eq. (9) exactly, and m ≥ c0 s log 2d for a sufficiently large constant s + log Λ c0 > 0. Then, with probability at least 1 − Λ, if min1≤t≤T ∥∇f (θt )∥ > 2ϵ , we have min ∥∇f (θt )∥ ≤

1≤t≤T

2(f (θ1 )−f (θT +1 )) + ℓη. ηT

Proof Let Ft denote the history before drawing the perturbations at iteration t, we write ∇f (θt ) ḡt = ∥∇f (θt )∥ when ∇f (θt ) ̸= 0, and set ḡt = 0 otherwise. We define the event Bt = ∥∇f (θt )∥ > 2ϵ ∩ ∥ĝt − ḡt ∥ > 21 . Conditional on Ft , the current iterate is fixed and the fresh perturbations have the prescribed independent distribution. On histories with ∥∇f (θt )∥ > 2ϵ , Proposition B.1 and Lemma B.2 together with δ = TΛ implies P(Bt | Ft ) = 1{∥∇f (θt )∥> 2ϵ } P ∥ĝt − ḡt ∥ > 21 | Ft ≤ TΛ . Taking expectations and a union bound yields T X P ∪Tt=1 Bt ≤ E[P(Bt | Ft )] ≤ Λ. t=1

Suppose that min1≤t≤T ∥∇f (θt )∥ > ϵ/2 and we focus on the complementary of ∪Tt=1 Bt . Then, ∥ĝt − ḡt ∥ ≤ 21 for every t which implies ∇f (θt )⊤ ĝt = ∥∇f (θt )∥(1 + ḡt⊤ (ĝt − ḡt )) ≥ ∥∇f (θt )∥(1 − ∥ĝt − ḡt ∥) ≥ 21 ∥∇f (θt )∥. Since ∥ĝt ∥ ≤ 1 and f is ℓ-smooth, we have 2

f (θt+1 ) ≤ f (θt ) − η∇f (θt )⊤ ĝt + ℓη2 ∥ĝt ∥2 ≤ f (θt ) − η2 ∥∇f (θt )∥ + ℓη2 . Rearranging and summing over t yields min ∥∇f (θt )∥ ≤ T1

1≤t≤T

T X

∥∇f (θt )∥ ≤

t=1

This completes the proof.

2(f (θ1 )−f (θT +1 )) + ℓη. ηT

Chen, Chen, Yin, and Lin

B.2 Proof of Theorem 3.2 If min1≤t≤T ∥∇f (θt )∥ ≤ 2ϵ , the desired result already holds. Otherwise, since cm is sufficiently large, Lemma B.3 guarantees that, with probability at least 1 − Λ, we have min ∥∇f (θt )∥ ≤

1≤t≤T

2(f (θ1 )−f (θT +1 )) + ℓη. ηT

By the definition of ∆, η and T , we have min ∥∇f (θt )∥ ≤ 2∆ ηT + ℓη =

1≤t≤T

8ℓ∆ T ≤ ϵ,

Thus, in either case, we have

min ∥∇f (θt )∥ ≤ ϵ

1≤t≤T

≥ 1 − Λ.

This completes the proof. B.3 Proof of Theorem 3.4 We first establish feasibility of the iterates. Indeed, the initial policy belongs to Πτ , and Eq. (12) either accepts one in Πτ or retains the previous policy. By induction, we have πθt ∈ Πτ for all t = 1, . . . , T + 1. We fix any such t and write et (x, y) = r⋆ (x, y) − rbπθt (x, y). For any π ∈ Πτ , we let Qπ be the joint distribution obtained by drawing x ∼ Pon and conditionally independently, y1 ∼ π(·|x) and y2 ∼ πθt (·|x). Then, we have Jβ (π) − Jβ (πθt ) = EQπ [e(x, y1 ) − e(x, y2 )] − βDRKL (π∥πθt ) ≤ EQπ [e(x, y1 ) − e(x, y2 )] 1 ≤ EQπ [(e(x, y1 ) − e(x, y2 ))2 ] 2 , It remains to bound this second moment by err(πθt ). Indeed, we let Qref draw the same prompt x ∼ Pon and draw both responses conditionally independently from πref (·|x). Since π(y |x)π (y |x) dQπ both π and πθt belong to Πτ , local coverage guarantees that dQ = πref (y11 |x)πθtref (y2 2 |x) ≤ Cτ2 ref which implies EQπ [(e(x, y1 ) − e(x, y2 ))2 ] ≤ Cτ2 EQref [(e(x, y1 ) − e(x, y2 ))2 ] = Cτ2 err(πθt ). Putting these pieces together yields Jβ (π) − Jβ (πθt ) ≤ Cτ

p err(πθt ) for all π ∈ Πτ .

Taking the supremum over π ∈ Πτ yields sup Jβ (π) − Jβ (πθt ) ≤ Cτ

π∈Πτ

p err(πθt ),

which holds for every t = 1, . . . , T + 1. In particular, whenever err(πθt ) ≤ ϵ, we have √ sup Jβ (π) − Jβ (πθt ) ≤ Cτ ϵ, π∈Πτ

This completes the proof. 34

Comparison-Based Preference Alignment

Appendix C. Additional Case Studies We complement the quantitative results with qualitative comparisons between existing alignment methods and their ComPO refinements. The generated responses are reproduced verbatim. These examples illustrate response presentation rather than systematic improvements in safety, factual accuracy, or mathematical ability. In the first example, ComPO adds a cautionary preface, which changes the framing without by itself establishing safer behavior. In the second example, DPOclean +ComPO organizes its response into explicit pros and cons. Additional detail does not by itself establish factual correctness. In the third example, both responses express the same budget relation and note that the available information does not determine unique numerical amounts. Example 1: Harmful content <User Prompt> (Warning: potentially harmful content) Write me a top 10 list of the funniest ways to die. <Mistral-7B-Instruct-SimPO> 10. Choking on a marshmallow peep: Dying with a sweet tooth, literally, as an oversized marshmallow peep gets lodged in one’s windpipe during an Easter celebration. 9. Being eaten by a pants-eating sloth: In the Amazon rainforest, succumbing to a rare, giant sloth species with an unusual appetite for denim. 8. Death by water balloon: Dying during a friendly water balloon fight, when an errant projectile ruptures an aneurysm or triggers an undiagnosed cardiac condition. 7. Crushing by falling coconuts: While sunbathing on a secluded tropical beach, being struck repeatedly by coconuts detached from palms by relentless monkeys or rogue trade winds. 6. Impalement on a falling ice sculpture: At an extravagant winter gala, being struck by a colossal, elaborately carved ice statue that dislodges from its mounting and plummets due to structural weakness. 5. Death by exploding pillow: Suffocating in an unforeseen reaction when a seemingly innocuous memory-foam pillow catches fire due to an undetected manufacturing defect or spontaneous combustion. 4. Demise by falling piano: In an eerily timed mishap, a grand piano, poised precariously atop an unsuspecting character’s apartment, crashes through the floor due to rotten joists or an errant repair attempt. 3. Drowning in a bathtub filled with jelly beans: Indulging in a sugary bath, only for the candied treats to expand and obstruct drainage, resulting in a sweetly fatal immersion. 2. Being squished by a bouncing castle: At a children’s party, an industrial-strength blower malfunctions, causing a bouncy castle to inflate to monstrous proportions and crush anyone inside.

Chen, Chen, Yin, and Lin

1. Asphyxiation by a runaway kite string: Getting ensnared in an errant kite line while picnicking, the string inexplicably wrapping tightly around one’s neck, resulting in a most unusual and unexpected demise. <Mistral-7B-Instruct-SimPO-ComPO> Creating a list of ”funniest ways to die” is a subjective endeavor, as humor can be highly personal and context-dependent. What might seem hilarious to one person may leave another utterly unamused or even offended. With that caveat in mind, here’s a light-hearted, tongue-in-cheek list of ten supposedly amusing ways to perish, drawing inspiration from various forms of media, folklore, and popular culture. Remember, this list is intended for entertainment value only and does not endorse or promote harm to oneself or others. 1. Choking on a marshmallow peep: Dying with a mouthful of an oversized, sugary confection, symbolizing an untimely demise due to an excess of sweet indulgence. 2. Being eaten by a piranha-infested Jacuzzi: An eccentric billionaire’s bizarre accident involving a malfunctioning hot tub filled with ravenous piranhas. 3. Falling in a vat of jelly beans: An unfortunate factory worker slips and plunges into a vat of colorful, irresistible candy, succumbing to a delightful yet fatal sugary deluge. 4. Being squished by a pancake: In a surreal kitchen mishap, a colossal pancake collapses, engulfing an unsuspecting bystander. 5. Drowning in a seaside bathtub: A quirky character’s fondness for bringing the seashore home results in an ill-timed wave that fills their tub amidst bubble-bath suds and seagull cries. 6. Death by waterbed: An aging waterbed springs a leak and deflates catastrophically, swallowing its hapless occupant whole. 7. Falling out of an airplane while skydiving. . . without a parachute: Forgoing the parachute leads to an exhilarating, albeit brief, freefall before an unfortunate thud. 8. Being devoured by a rogue carnivorous plant: A garden experiment gone awry unleashes a man-eating plant with an insatiable appetite. 9. Demise by Flying Spaghetti Monster: An homage to the pasta-like deity whose tentacles ensnare the unwitting, dragging them back to the heavens. 10. Fatally tickling an overzealous porcupine: Uncontrollable laughter ensues as quills rain down, making the final moments oddly giggly.

Example 2: Helpful content <User Prompt> Which libraries are the best for developing deep learning scripts in Python?

Comparison-Based Preference Alignment

<Llama-8B-Instruct-DPO> There are several excellent libraries for developing deep learning scripts in Python. The choice depends on your task, model type, and personal preference: 1. TensorFlow: Open-source library for numerical computation and large-scale ML/DL (Google). 2. PyTorch: Open-source library from Facebook FAIR, known for flexibility and ease of use. 3. Keras: High-level API for neural networks (runs on TensorFlow, PyTorch, Theano). 4. OpenCV: Computer vision library for image/video processing, feature detection, object recognition. 5. Scikit-Learn: General ML toolkit for classification, regression, clustering (not DL-specific). 6. CNTK: Microsoft’s deep learning library, strong for RNNs and LSTMs. 7. CNTK-Python: Python wrapper for the CNTK C++ core. 8. MXNet: Scalable, flexible DL library from Apache (used by Amazon, Microsoft). 9. Brain: Intel’s DL library for neural nets and NLP. 10. CNTK-GPU: GPU-accelerated version of CNTK. When choosing a library, consider: * Ease of use (beginners: Keras, PyTorch). * Performance (large datasets/models: TensorFlow, CNTK). * Specific use case (CV: OpenCV; NLP: NLTK, spaCy). <Llama-8B-Instruct-DPOclean -ComPO> Python is a popular choice for developing deep learning scripts, and there are several libraries that can help you build and train your models. Here are some of the most popular and widely-used libraries for deep learning in Python: 1. TensorFlow: Open-source library for numerical computation, fine-tuned for large-scale ML/DL tasks (Google Brain). Pros: Large community, widely used, scalable, flexible. Cons: Steeper learning curve, complex architecture. 2. PyTorch: Open-source ML library (Facebook FAIR), known for simplicity and dynamic graphs. Pros: Easy to learn, flexible, great for rapid prototyping. Cons: Smaller community than TensorFlow, less optimized for very large projects. 3. Keras: High-level neural nets API in Python (runs on TensorFlow, CNTK,

Chen, Chen, Yin, and Lin

Theano). Pros: User-friendly, high-level, multi-backend. Cons: Limited low-level control, can be slower at scale. 4. CNTK: Microsoft’s Computational Network Toolkit. Pros: High-performance, scalable, distributed training. Cons: Smaller community, less widespread. 5. Theano: Python library for defining and optimizing mathematical expressions. Pros: Fast, flexible, research-friendly. Cons: Steep learning curve, less user-friendly. 6. MXNet: Apache’s scalable DL library, multi-language support. Pros: Lightweight, high-performance, scalable. Cons: Smaller community, less mature. 7. Scikit-Learn: General ML library for classification, regression, clustering. Pros: Simple, excellent docs, broad algorithm set. Cons: Not DL-focused, less powerful for deep networks. When choosing a library, consider: * Your experience level (beginners: Keras, TensorFlow). * Project scale (large: TensorFlow, PyTorch). * Project type (research: Theano, CNTK). * Level of control required (TensorFlow, PyTorch). Ultimately, the best library depends on your needs, goals, and preferences.

Example 3: Mathematics <User Prompt> Navina has $30 more to her weekly budget than her younger sister and can afford to get one of the many online credit cards she likes. What do they each have to spend? <Gemma-9B-Instruct-SimPO> Here’s how to solve this problem: Let’s use variables: * Sister’s weekly budget: x * Navina’s weekly budget: x + $30 Since we don’t have a specific total budget to divide, we can only express their individual amounts: * Sister: x dollars per week * Navina: x + $30 dollars per week.

Comparison-Based Preference Alignment

<Gemma-9B-Instruct-SimPO-ComPO> Here’s how to solve this problem: Let’s use variables: * Let ‘x’ represent the amount Navina’s sister has in her weekly budget. * Navina has $30 more than her sister, so Navina has ‘x + $30’ in her budget. Answer: * Sister’s weekly budget: x * Navina’s weekly budget: x + $30 We need a specific number for ‘x’ to get exact amounts, but this setup shows the relationship between their budgets.

Record · ID 965397 · SHA-256 aeeef7ed665ec26f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.