arXiv:2605.10784v1 [cs.LG] 11 May 2026
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
Rohan Surana1 , Xintong Li1 , Sheldon Yu1 , Yiran Jenny Shen1 , Chuhan Wang1 , Tong Yu2 , Prithviraj Ammanabrolu1 , Jingbo Shang1 , Julian McAuley1 , Junda Wu1 1 UC San Diego 2 Adobe Research {rsurana,xil240,ziy040,jes038,chw136,prithvi,jshang,jmcauley,juw069}@ucsd.edu [email protected]
Abstract Multi-negative preference optimization under the Plackett–Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisherinformation objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool’s information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.
1
Introduction
Direct Preference Optimization (DPO) [50] aligns models with human preferences by optimizing pairwise comparisons without constructing reward functions [12, 46, 56]. Recent work generalizes DPO with the Plackett–Luce (PL) model [48, 39, 23, 66, 68] to compare one preferred response against multiple rejected responses, providing richer supervision. However, current multi-negative approaches such as Softmax-DPO (S-DPO) [11] and Direct Multi-Preference Optimization (DMPO) [6] typically sample or weight negatives randomly or heuristically. In large candidate pools, this can devote much of the training signal to near-duplicate negatives whose gradients point in similar directions, increasing computation without proportionally improving policy updates. To address this bottleneck, we propose MASS-DPO (Multi-negative Active Sample Selection for Direct Preference Optimization), an active negative selection framework derived from the multinegative PL preference objective. MASS-DPO formulates negative selection as a D-optimal design problem [49, 30], using a PL-specific Fisher-information objective to measure how much each candidate contributes to policy estimation [16, 10, 17, 32]. We favor D-optimality over alternatives such as A- or E-optimality because maximizing log-determinant information minimizes the volume of the joint confidence ellipsoid, promoting coverage across parameter directions rather than emphasizing a single mode [30, 49]. Without careful selection, the model can repeatedly update along alreadycovered directions, leading to poor parameter coverage and inefficient optimization. MASS-DPO Preprint.
addresses this by selecting negatives that span complementary directions in parameter space, as determined by the D-optimal design formulation. While D-optimal design provides a principled Fisher-information criterion for prioritizing preference data, most prior work applies it atathe instanceFisher-information level: selecting which preference samples, prompts, While D-optimal design provides principled criterion for prioritizing preference query or annotators to the acquire or retain 38, 14,which 41]. MASS-DPO instead applies op39 data, distributions, most prior work applies it at instance level:[35, selecting preference samples, prompts, timal design within each prompt. Given a shared pool of candidate negatives for a preferred response, 40 query distributions, or annotators to acquire or retain [30, 32, 14, 35]. MASS-DPO instead applies opittimal selects a compact subset tailored to the multi-negative PL objective using curvature/Fisher 41 design within each prompt. Given a shared pool of candidate negatives for our a preferred response, (Theorem 42 characterization it selects a compact subset4.3). tailored to the multi-negative PL objective using our curvature/Fisher 43 The characterization (Theorem 4.3).problem is combinatorial when the candidate pool is large [33, 32, resulting subset selection 45, 25]. We make it practical with anisincremental rank-one procedure subset one 44 The resulting subset selection problem combinatorial when the candidatethat poolbuilds is largethe [28, 27]. We negative at a time using marginal log-determinant gains. Sherman–Morrison updates avoid repeated 45 make it practical with an incremental rank-one procedure that builds the subset one negative at a time recomputation, the log-determinant efficient to optimize [53, 46 determinant/inverse using marginal log-determinant gains.making Sherman–Morrison updates objective avoid repeated determinant/inverse 35, 42]. 47 32, recomputation, making the log-determinant objective efficient to optimize [45, 27, 30]. efficiency and and downstream downstreamperforperfor48 Empirically, Empirically,we weshow showthat thatMASS-DPO MASS-DPO improves improves optimization efficiency benchmarks, while whileusing usingsubstansubstan49 mance manceacross acrossthree threemodel modelfamilies familiesand and four four recommendation/QA recommendation/QA benchmarks, 50 tially tiallyfewer fewernegatives. negatives.We Wesummarize summarize our our contributions contributions as follows: 38
52 53 54 55 56
selection framework frameworkfor formultimulti• •We Weintroduce introduceMASS-DPO, MASS-DPO, aa within-prompt within-prompt active negative selection negative its Fisher/curvature Fisher/curvaturestructure. structure. negativeDPO, DPO,derived derivedfrom from the the Plackett–Luce Plackett–Luce objective and its • •We efficient log-determinant log-determinantoptioptiWeprovide providean anincremental incremental rank-one rank-one selection selection algorithm for efficient mization bounds. mizationand andestablish establishfinite-sample finite-sample relative-logit relative-logit error bounds. • • Empirically, and downstream downstreamperformance performance Empirically,MASS-DPO MASS-DPOimproves improves optimization optimization efficiency and across benchmarks. acrossthree threelanguage languagemodel model families families and and four recommendation/QA recommendation/QA benchmarks. Candidate Pool
Negative Candidate Selection via D-Optimal Design (Sec. 4.2)
Feature Embeddings Diff 𝜑𝑖 𝜑 𝑥, 𝑦 − 𝜑 𝑥, 𝑦 ∗ Policy Offsets 𝑏 log 𝜋𝑟𝑒𝑓 𝑦 ∗ 𝑥 − log 𝜋𝑟𝑒𝑓 𝑦 𝑥)
Softmax Score (Eq. 7) 8
51
Information Matrix (Eq. 10) 1 𝑖𝑛𝑖𝑡: 𝐻0 ← 𝛾𝐼𝑑×𝑑
Active Negative Selection (Eq.13) 𝐼𝑘 ← arg max log det 𝐻𝑘−1 + 𝑣𝑖 𝑣𝑖⊤ 𝑖
Selected negative sample
DPO
SDPO
Update design (Eq.13) (𝐻𝑘 ← 𝐻𝑘−1 + 𝑣𝐼𝑘 𝑣𝐼𝑘𝑇 )
MASS-DPO
Figure1:1:Overview OverviewofofMASS-DPO’s MASS-DPO’s D-optimal D-optimal selection. selection. Each candidate Figure candidate is is scored scored using usingthe thefeature feature ∗∗ ) and policy offset bi = log πref (y ∗∗ | x) − log πref (yi | x), difference ϕ = ϕ(x, y ) − ϕ(x, y i i difference ϕi = ϕ(x, yi ) − ϕ(x, y ) and policy offset bi = log πref (y | x) − log πref (yi | x), withsoftmax softmaxweights weightsdefined definedin inEquation Equation (8). (8). The The green green loop loop denotes with denotes the the subset-construction subset-constructionstep step Algorithm1:1: starting startingfrom from H H00,, we we incrementally incrementally pick pick the ininAlgorithm the negative negative that that maximally maximally increases increases logdet detH, H,then thenupdate updateH Haccordingly accordingly until until nn samples samples are are selected. selected. log
57
RelatedWork Work 22 Related
DirectPreference PreferenceOptimization. Optimization. DPO DPO [50] [42] aligns language models Direct models with with human human preferences preferences by optimizing likelihood ratios of preferred over dispreferred responses, avoiding by optimizing likelihood ratios of preferred over responses, avoiding explicit explicitreward reward 60 modeling modelingand andassociated associated complexities complexities such such as as reward rewardmisgeneralization misgeneralizationininRLHF RLHF[12, [12,38, 46,48]. 56,Recent 58, 26, 61 65]. extensions dynamic margins (ODPO; [3]) and prefix[3]) sharing computational [52]. Recent include extensions include dynamic margins (ODPO; and for prefix sharing forefficiency computational 62 efficiency However,[62]. standard DPO is restricted to binary preference pairs, limiting the diversity of supervision. However, standard DPO is restricted to binary preference pairs, limiting the diversity 63 ofOur approach[61]. extends beyond binary bycomparisons leveraging actively selected, informative supervision Our approach extendscomparisons beyond binary by leveraging actively selected, 64 multi-negative samples. samples. informative multi-negative 65 Multi-negativePreference Preference Optimization. Optimization. Recent Recent work work has Multi-negative has extended extended standard standard DPO’s DPO’s binary binary 66 preference pairs to leverage multiple negatives for richer comparative signals and enhanced preference pairs to leverage multiple negatives for richer comparative signals and enhancedalignment. alignment. 67 Softmax-DPO(S-DPO) (S-DPO)[11] [11]generalizes generalizes the the pairwise pairwise Bradley–Terry Bradley–Terry loss Softmax-DPO loss [9] [9] to to Plackett–Luce Plackett–Lucerankrank68 ing [40, 33], providing richer gradient signals. Direct Multi-Preference Optimization ing [48, 39, 22], providing richer gradient signals. Direct Multi-Preference Optimization(DMPO) (DMPO)[6] [6] 69 averages over multiple negatives to promote diverse negative learning. Multi Pair-wise Preference 70 Optimization (MPPO) [54] extends DPO by directly modeling multi-negative feedback with average71 likelihood loss, removing the need for a reference model and enabling flexible use of negative samples. 2 58 59
2
averages over multiple negatives to promote diverse negative learning. Multi Pair-wise Preference Optimization (MPPO) [67] extends DPO by directly modeling multi-negative feedback with averagelikelihood loss, removing the need for a reference model and enabling flexible use of negative samples. Tree Preference Optimization (TPO) [37] structures multi-negative alignment through hierarchical preference decomposition. Despite these advances in multi-negative preference optimization, current methods still largely depend on heuristic or random negative selection strategies. Our work addresses this limitation by proposing MASS-DPO, which leverages D-optimal design for theoretically grounded, strategic negative sample selection. Information-theoretic sample selection and optimal design. A broad literature in optimal experimental design and active learning selects informative data by maximizing information about model parameters, with D-optimality (maximizing log det of the Fisher information) as a standard criterion [30, 10, 49]. In modern batch active learning, related objectives are used to promote coverage/diversity in representation or Fisher/gradient space [53, 32, 5]. In preference optimization, recent work adopts such principles primarily at the instance level—selecting which prompts/comparisons (and, in some settings, teachers) to acquire or retain for training [35, 38, 14, 41, 24, 64]. MASS-DPO instead applies D-optimal design within each prompt: given a shared pool of negative candidates, we select a small subset tailored to the multi-negative Plackett–Luce objective via our PL-specific curvature/Fisher characterization. Unlike online hard-negative mining and dynamic sampling strategies [19, 70, 15, 40, 36], which recompute candidates at every training step and often rely on task-specific heuristics, MASS-DPO operates in a fixed-budget regime: negatives are selected once as a preprocessing step and remain fixed during training, incurring no per-step mining cost. The Fisher-information criterion selects negatives spanning complementary directions in parameter space, a geometric property that persists across training (Table 4).
3
Preliminaries
3.1
Direct Preference Optimization
Direct Preference Optimization (DPO) [50] aligns a learned policy with human pairwise judgments [12, 55, 46] without an explicit reward model. Under the Bradley-Terry-Luce framework [9], two responses y1 , y2 to prompt x with latent scores r(x, y1 ), r(x, y2 ) satisfy p∗ (y1 ≻ y2 | x) = σ(r(x, y1 ) − r(x, y2 )),
(1)
−z
where σ(z) = 1/(1 + e ). Rearranging the optimal-policy relation gives the implicit reward representation up to an x-dependent additive constant β log Z(x): π ∗ (y | x) + β log Z(x), πref (y | x) X Z(x) = πref (y ′ | x) · exp β1 r(x, y ′ ) .
r(x, y) = β log
(2)
y′
Substituting equation 2 into equation 1 and simplifying leads to the DPO training objective ii h h πθ (y2 |x) θ (y1 |x) − log . LDPO (θ) = −E(x,y1 ,y2 )∼D log σ β log ππref (y1 |x) πref (y2 |x) 3.2
(3)
Multi-negative Preference Optimization
Multi-negative preference optimization generalizes the Direct Preference Optimization framework [50] to better align language models with multiple negative preferences. While traditional DPO employs the Bradley-Terry (BT) model [9] to capture pairwise comparisons, multi-negative preference optimization leverages the Plackett-Luce (PL) model [48, 39] to accommodate the ranking of a preferred item against multiple disfavored items. Consider a user prompt xu that is formed from historical interactions, along with a preferred item ep and a set of dispreferred items Ed . The aim is to maximize the probability that the preferred item ep is ranked above every item in Ed , as described by exp(r(xu , ep )) , ed ∈{ep }∪Ed exp(r(xu , ed ))
p∗ (ep ≻ Ed | xu ) = P
3
(4)
where r(xu , e) is the latent reward function defined over the prompt-response pairs in the RLHF framework [46]. From Equation (4), we obtain the following multi-negative preference loss: h X i L(θ) = −E(xu ,ep ,Ed )∼D log σ − log exp β ∆(xu , ed , ep ) (5) ed ∈Ed π (e |x )
p u θ θ (ed |xu ) with σ(·) denoting the sigmoid function and ∆(xu , ed , ep ) = log ππref (ed |xu ) − log πref (ep |xu ) . When |Ed | = 1, this reduces to the standard pairwise DPO objective.
4
MASS-DPO: Multi-negative Active Sample Selection
In multi-negative preference optimization tasks (e.g., recommendation, multiple-choice QA, information retrieval), the selection of negative samples significantly influences alignment efficiency and effectiveness. Uninformative negatives, already well-separated from preferred responses, waste gradient computations and hinder convergence [69, 28, 52, 71]. Thus, the key challenge is strategically selecting a compact yet informative subset of negatives to highlight the policy’s weaknesses while maintaining numerical stability [40, 31, 15]. To address this, we propose MASS-DPO (Figure 1), an active negative selection method formulated as a D-optimal design problem [49, 13, 32], maximizing a Fisher-information objective [16, 27, 38, 43, 54, 10, 4]. By maximizing this objective, MASS-DPO minimizes the volume of the confidence ellipsoid of policy parameters, connecting computational efficiency with statistical guarantees (Section 5). Algorithm 1 D-Optimal Multi-negative Active Sample Selection 1: Input: context x, preferred response y ∗ , candidate set C = {yi }N i=1 , preprocessing parameter θ0 , scale β,
ridge γ, number of negatives n 2: Compute feature differences and offsets, for each i ∈ [N ],
ϕi ← ϕ(x, yi ) − ϕ(x, y ∗ ), bi ← log πref (y ∗ | x) − log πref (yi | x) 3: Compute scores and softmax weights, for each i ∈ [N ], si ← β ϕ⊤ i θ0 + bi , P qi0 ← exp(si )/ N k=1 exp(sk ) 4: Center and weight features, for all i ∈ [N ] p PN 0 ϕ̄0 ← j=1 qj ϕj , ϕ̃0i ← ϕi − ϕ̄0 , vi0 ← qi0 ϕ̃0i 5: Compute fixed Fisher scale: P α0 ← β 2 (1 − σ(ZC0 )) ZC0 ← − log N i=1 exp(si ), 6: Initialize matrices and selected index set: H0 ← γId×d , I0 ← ∅ 7: for k = 1, . . . , n do 8: Select index: ik ← arg maxi∈[N ]\Ik−1 log det Hk−1 + α0 vi0 (vi0 )⊤ 9: Update selected indices and design matrix: Ik ← Ik−1 ∪ {ik }, Hk ← Hk−1 + α0 vi0k (vi0k )⊤ 10: end for 11: Output: selected negatives set Sn = {yi : i ∈ In }
4.1
Setting
Following prior work in regret minimization and reward-model active learning [35, 51, 14, 41, 38, 60], we adopt a log-linear policy model for the selection criterion. We assume: Assumption 4.1. We assume the policy under consideration takes a log-linear form: π(y | x; θ) ∝ exp ϕ(x, y)⊤ θ ,
(6)
where ϕ(x, y) ∈ Rd denotes the feature embedding of the context-response pair (x, y), and θ ∈ Rd the model parameters. We now specialize the multi-negative loss L from Equation (5) to a single prompt x with preferred response y ∗ and candidate negatives C = {yi }N i=1 . Under Assumption 4.1, defining the feature 4
∗
(y |x) difference ϕi = ϕ(x, yi ) − ϕ(x, y ∗ ) and reference-policy offset bi = log ππref for each negative ref (yi |x) ∗ yi relative to the preferred response y , the multi-negative DPO loss takes the compact form: X L(θ; Sn ) = − log σ − log exp β (ϕ⊤ , (7) i θ + bi ) i∈Sn
where Sn ⊆ C is a subset of size n drawn from the candidate pool. Our goal is to choose Sn so as to maximize the information it provides about θ. The following lemmas quantify how each candidate negative alters the gradient and curvature, showing that negatives with diverse and orthogonal feature differences enlarge the information matrix the most, while redundant examples leave its volume almost unchanged. Lemma 4.2 (Gradient of Multi-negative Loss). Define the normalization factor Zn and subsetnormalized softmax weights qjSn (θ) as exp β(ϕ⊤ j θ + bj ) Sn , qj (θ) = P ⊤ k∈Sn exp β(ϕk θ + bk ) (8) h i X Zn = − log exp β(ϕ⊤ θ + b ) . i i i∈Sn
Then the gradient of equation 7 with respect to θ is given by X ∇θ L(θ; Sn ) = β (1 − σ(Zn )) qjSn (θ) ϕj .
(9)
j∈Sn
The detailed derivation is in Appendix A.1. The gradient is a weighted combination of feature differences scaled by the misranking probability (1 − σ(Zn )). At training time, the subset weights qjSn (θ) emphasize negatives with small score margins, indicating that the highest-leverage gradient directions correspond to borderline, hard-to-rank candidates. P Lemma 4.3 (Hessian and Curvature). Let ϕ̄Sn (θ) = j∈Sn qjSn (θ) ϕj denote the expected feature difference under the subset softmax distribution. The Hessian of Equation (7) is then h i X ∇2 L(θ; Sn ) = β 2 (1 − σ(Zn )) σ(Zn ) ϕ̄Sn ϕ̄⊤ qjSn (θ)(ϕj − ϕ̄Sn )(ϕj − ϕ̄Sn )⊤ Sn + j∈Sn 2
⪰ β (1 − σ(Zn ))
X
qjSn (θ)(ϕj − ϕ̄Sn )(ϕj − ϕ̄Sn )⊤ .
(10)
j∈Sn
The inequality follows because σ(Zn ) ϕ̄Sn ϕ̄⊤ Sn ⪰ 0; dropping it yields a Loewner lower bound that isolates the dispersion of feature differences around their mean. The full derivation is in Appendix A.2. The Hessian lower bound motivates maximizing the determinant of the weighted covariance: subsets whose feature differences spread along orthogonal directions yield the largest information volume, providing a natural selection criterion. 4.2
Negative Selection via D-Optimal Design
While a larger negative pool can in principle improve parameter estimates, many candidates contribute redundant information already conveyed by a smaller, well-chosen subset. MASS-DPO casts negative selection as a D-optimal design [30, 49, 32] problem that maximizes the information gain [10] about the policy parameters. Fisher-information objective. Following standard practice in D-optimal active learning [10, 35], Algorithm 1 evaluates the design at a fixed reference point: the full-pool weight qj0 and center P ϕ̄0 = j∈C qj0 ϕj are computed once before training, defining each candidate’s Fisher contribution q as vj0 = qj0 (ϕj − ϕ̄0 ) and giving a fixed information matrix H(S) amenable to efficient rank-one optimization. Given a subset S ⊆ C we define the regularized information matrix: X vj0 (vj0 )⊤ , α0 = β 2 (1 − σ(ZC0 )), γ > 0, (11) H(S) = γI + α0 j∈S
5
P where ZC0 = − log i∈C exp[β(ϕ⊤ i θ0 + bi )] is fixed during subset construction. The ridge γ > 0 ensures H(S) is well conditioned for all subsets. The D-optimal criterion seeks Sn∗ = arg
max
S⊆C, |S|=n
log det H(S),
(12)
which maximizes the information volume, equivalently minimizing the volume of the confidence ellipsoid for the policy parameters. However, Equation (12) is NP-hard [63, 2] as it requires searching over |C| n subsets. We therefore build the subset incrementally using marginal log-determinant gains [44, 34]. Incremental subset construction. Starting from H0 = γI and selected index set I0 = ∅, we build the selected negative set Sn = {yi : i ∈ In } one element at a time via rank-one updates. At iteration k, select the next negative by maximizing the marginal log-determinant gain: ik ← arg max log det Hk−1 + α0 vi0 (vi0 )⊤ , i∈[N ]\Ik−1
Ik ← Ik−1 ∪ {ik },
Hk ← Hk−1 + α0 vi0k (vi0k )⊤ .
(13)
Using the matrix determinant lemma, −1 0 vi , log det Hk−1 + α0 vi0 (vi0 )⊤ = log det Hk−1 + log 1 + α0 (vi0 )⊤ Hk−1
(14)
0 ⊤ −1 0 so the selection rule is equivalently ik = arg maxi∈I / k−1 (vi ) Hk−1 vi (Alg. 1). After scoring the remaining candidates, the Sherman–Morrison inverse update costs O(d2 ) per selected negative, and −1 0 scoring all remaining candidates costs O(|C|d2 ) per step. The score (vi0 )⊤ Hk−1 vi is the Hk−1 0 induced squared norm of vi ; the procedure thus prefers negatives that probe the least-covered directions of the parameter space. We empirically verify that fixed selections remain stable across training (Section C.4, Table 4).
5
Theoretical Analysis
Having established the D-optimal selection criterion, we now analyze how well a policy trained on the selected subset Sn approximates one trained on the full pool. Our goal is to bound the relative logit error—the worst-case distortion of pairwise candidate-margin rankings—as a function of subset size n and dimension d. The analysis relies on standard assumptions on feature boundedness, design-weight regularity, and candidate diversity; full statements are deferred to Section C. Theorem 5.1 (Relative Logit Error Bound). For a fixed prompt with candidate-negative pool C = {yi }N i=1 , let θ∗ minimize the regularized full-pool loss and θ̂n minimize the regularized loss on the subset Sn ⊆ C of size n returned by Algorithm 1. Define the relative logit error Erel (θ̂n , θ∗ ) = max (ϕi − ϕj )⊤ (θ̂n − θ∗ ) , i,j∈C
(15)
measuring the worst-case distortion of pairwise candidate-margin rankings induced by using a subset-trained estimator in place of the full-pool optimum. Under Assumptions C.1–C.4, ! r log(1/δ) e Erel (θ̂n , θ∗ ) ≤ O d , n e with probability at least 1 − δ, where √ O hides logarithmic factors and candidate-pool regularity constants. The bound decays as 1/ n with the selected-negative budget, showing that the subsettrained policy converges to the full-pool policy in relative-logit error as n grows. The formal statement with explicit constants, the Fisher-compatibility and estimator-stability conditions, and the proof are given in Section B. 6
Theorem 5.2 (Batch Design Estimation Error). With probability at least 1 − δ, given k prompts each with n selected negatives Sk,n , the deviation of the regularized batch estimator θ̂k,n from the full-pool optimum θ∗ is bounded in the Σk,n -norm: v ! u ud 1/δ + k cmin /γ t θ̂k,n − θ∗ Σ ≤ + 2γ 1/2 , (16) log 1/d k,n 4 1 − cmin k/γ δ where Σk,n = γI + ∇2 L(θ∗ ; Sk,n ), and cmin , γ, β are the constants from Assumptions C.1–C.2. The probability is over i.i.d. sampling of k prompts; this follows [1, 35] by treating the multi-negative loss as a generalized linear model and applying self-normalized concentration to the stochastic gradients. In practice, Theorem 5.1 shows that even with a small selected-negative budget MASSDPO can already achieve bounded logit error, which translates into faster convergence; Theorem 5.2 further implies that the selected negatives ensure stable generalization across prompts, which we verify empirically in Section 6. Algorithm 1 optimizes the log-determinant objective via incremental rankone updates; the connection to relative-logit error is carried by the leverage bound in Appendix C.2. Table 1: Accuracy (%) on four tasks across three base models. Each entry reports accuracySE , where the subscript denotes standard error. Bold = best, underlined = second best.
6
Model
Setting
Medmcqa
QASC
LastFM
MovieLens
Avg
Qwen3
DPO DMPO DPO-k S-DPO MASS-DPO
43.490.76 28.910.72 55.560.77 52.560.77 56.660.77
68.431.12 66.781.12 71.961.06 71.081.07 72.191.05
45.750.80 43.400.78 51.100.80 50.250.80 52.300.79
31.960.74 25.660.69 44.560.77 48.190.78 47.580.80
47.410.86 41.190.83 55.800.85 55.520.86 57.180.85
SmolLM3
DPO DMPO DPO-k S-DPO MASS-DPO
33.270.73 25.500.68 44.090.79 44.990.79 44.190.79
67.001.09 65.231.10 69.981.01 69.431.06 71.631.07
51.900.80 50.100.80 55.700.78 55.900.78 57.250.79
37.600.77 28.680.69 51.360.79 55.700.78 54.030.77
47.440.85 42.380.82 55.280.84 56.500.85 56.780.85
Llama3
DPO DMPO DPO-k S-DPO MASS-DPO
52.250.80 25.700.69 71.040.73 72.190.72 71.290.74
71.081.04 69.871.08 73.950.96 74.610.97 73.621.03
54.600.82 49.950.80 55.650.80 56.550.79 57.350.81
33.520.75 28.180.72 44.460.77 49.550.80 49.700.80
52.860.85 43.420.82 61.270.82 63.230.82 62.990.84
Experiments
Section 6.1 isolates MASS-DPO’s D-optimal selection criterion by comparing against the random softmax weighting in S-DPO. Section 6.2 reports downstream accuracy against existing preferenceoptimization baselines across three backbone families. Section 6.3 evaluates the quality of the negatives produced by the incremental selection procedure using standard ranking metrics. Datasets. Following recent DPO-based recommendation work [11, 57, 21], we utilize two widely adopted real-world recommendation benchmarks: LastFM [8] and MovieLens [20]. For QA tasks, we adopt two challenging multiple-choice QA datasets: MedMCQA [47], a medical-domain QA benchmark, and QASC [29], a scientific reasoning QA dataset. These tasks naturally feature large candidate pools with well-defined negatives, providing controlled benchmarks for evaluating active selection strategies. We report Accuracy, Margin, Chosen Rewards, and additional utility metrics; detailed methodology is in Appendix C.3. Methods. We benchmark MASS-DPO against established preference alignment approaches: pairwise DPO [50], the multi-negative extension DPO-k, Softmax-DPO (S-DPO) [11], and DMPO [6]. To maintain fairness and manage computational costs, the number of negative candidates during training is set to 3 for all multi-negative methods (DPO-k, DMPO, S-DPO, MASS-DPO) and 1 for DPO. At test time, we evaluate against all available candidates (up to 20) to measure the model’s ability to rank under a larger search space. Implementation details are provided in Section C.4. 7
Table 2: Recall (R) and NDCG (N) at k={1,3} on LastFM and MovieLens. Each entry reports metricSE , where the subscript denotes standard error. Model
Method
LastFM
MovieLens R@3 N@1
R@1
R@3
N@1
N@3
R@1
Qwen3
DPO DMPO DPO-k S-DPO MASS-DPO
46.151.11 44.501.11 49.501.12 48.551.12 51.101.12
72.601.00 72.051.00 76.450.95 75.100.97 77.200.94
46.151.11 44.501.11 49.501.12 48.551.12 51.101.12
61.600.93 60.570.93 65.360.90 64.140.91 66.480.90
29.641.03 24.500.97 41.631.11 45.921.12 45.971.12
59.481.10 56.301.11 68.951.04 71.471.01 71.521.01
29.641.03 24.500.97 41.631.11 45.921.12 45.971.12
46.890.95 42.880.92 57.710.95 60.860.94 61.100.94
SmolLM3
DPO DMPO DPO-k S-DPO MASS-DPO
51.701.12 50.301.12 56.301.11 55.601.11 57.051.11
78.150.92 77.900.93 80.550.89 81.350.87 80.700.88
51.701.12 50.301.12 56.301.11 55.601.11 57.051.11
67.290.89 66.540.89 70.710.86 70.840.85 71.080.86
37.251.09 28.231.01 51.011.12 55.091.12 54.181.12
65.681.07 60.431.10 75.710.96 78.180.93 77.570.94
37.251.09 28.231.01 51.011.12 55.091.12 54.181.12
53.770.95 47.020.93 65.410.92 68.640.90 68.030.90
Llama3
DPO DMPO DPO-k S-DPO MASS-DPO
55.151.11 49.951.12 56.051.11 56.501.11 56.601.11
80.350.89 78.350.92 80.300.89 80.850.88 81.150.87
55.151.11 49.951.12 56.051.11 56.501.11 56.601.11
70.060.87 66.760.88 70.410.87 70.950.86 71.170.86
34.481.07 27.821.01 43.951.11 48.941.12 50.661.12
63.561.08 58.721.11 70.771.02 73.390.99 76.010.96
34.481.07 27.821.01 43.951.11 48.941.12 50.661.12
51.310.95 45.690.94 59.690.94 63.250.94 65.570.91
LlaMA3_MASS-DPO LlaMA3_S-DPO
Qwen3_MASS-DPO Qwen3_S-DPO
Margin LastFM
4 2
MedMCQA
0
0
5
10
15
20
0.4
1.0
0.2
0.5
0.0
0
5
10
15
10
Step
15
0.0
0
5
0
5
10
15
10
15
0
0.2 5
0.0 2
0.4
0
Chosen Rewards
1.5
0.6
10 0
SmolLM3_MASS-DPO SmolLM3_S-DPO
Accuracy
6
N@3
2 0
5
10
Step
15
Step
Figure 2: Training dynamics on LastFM and MedMCQA. MASS-DPO (solid) achieves larger margins, higher accuracy, and more stable chosen rewards than S-DPO (dashed). 6.1
How effectively does D-optimal active negative selection optimize the multi-negative preference learning objective?
We compare MASS-DPO’s active negative selection to the softmax-based random selection in SDPO across all four datasets. Figure 2 and Figure 3 (Appendix C.3) track three alignment metrics during training: margin (logit gap between preferred vs. rejected), accuracy, and chosen rewards. Across datasets, MASS-DPO (solid) achieves larger margins and faster early gains than S-DPO (dashed), with the gap emerging early and persisting through training. Accuracy follows the same pattern: curves for MASS-DPO rise more quickly and attain higher plateaus. Finally, chosen-reward trajectories under MASS-DPO are smoother and more stable across steps, while S-DPO exhibits noticeably noisier dynamics. 6.2
How does MASS-DPO improve downstream policy performance compared to existing preference optimization methods?
We benchmark MASS-DPO against DPO, DMPO, DPO-k, and S-DPO on four datasets (MedMCQA, QASC, LastFM, MovieLens) using Accuracy, reporting results for three base models in Table 1. MASS-DPO achieves the highest average accuracy on Qwen3 and SmolLM3, leads both recommendation tasks on Llama3, and remains competitive on every dataset across all three model families. Baselines without active negative selection (DPO, DMPO, and DPO-k) generally underperform, confirming that which negatives enter the multi-negative objective matters. At matched wall-clock 8
Table 3: MASS-DPO ablations on two hyperparameters. Each entry reports accuracySE , where the subscript denotes standard error. Bold = best. (a) Varying the scale β ∈ {0.1, 0.5, 1.0} while holding n = 3 fixed. (b) Varying n ∈ {1, 3, 5} while holding β = 0.1 fixed. (a) β ablation
(b) Negatives n ablation
Model
β
Medmcqa
LastFM
MovieLens
Model
n Medmcqa
LastFM
MovieLens
Qwen3
0.1 0.5 1.0
56.660.77 72.191.05 52.300.79 46.290.79 71.411.06 48.150.81 43.490.77 69.651.06 44.150.80
47.580.80 39.820.79 34.120.74
Qwen3
1 3 5
50.950.78 68.211.13 47.800.80 56.660.77 72.191.05 52.300.79 57.310.75 73.731.03 54.500.79
32.860.73 47.580.80 58.110.78
0.1 SmolLM3 0.5 1.0
44.190.79 71.631.07 57.250.79 39.730.76 71.631.03 54.750.79 35.420.75 68.981.06 52.000.80
54.030.77 56.300.79 52.120.80
1 SmolLM3 3 5
29.260.72 65.671.09 50.700.81 44.190.79 71.631.07 57.250.79 46.590.79 71.631.04 59.500.77
34.580.75 54.030.77 65.070.74
0.1 0.5 1.0
71.290.74 73.621.03 57.350.81 69.690.74 73.511.01 55.750.80 66.280.76 72.191.02 52.250.81
49.700.80 51.060.78 45.920.78
Llama3
1 3 5
46.990.78 71.961.03 52.700.81 71.290.74 73.621.03 57.350.81 73.550.71 74.940.98 60.050.80
32.710.73 49.700.80 60.990.77
Llama3
QASC
QASC
budgets (Table 7, Appendix C.6), MASS-DPO with a small selected subset matches or exceeds training with all available negatives (Table 8), indicating that well-chosen negatives provide more useful signal per gradient step than the full pool. 6.3
How informative are the negatives produced by incremental selection?
We assess negative-selection quality using downstream utility metrics, MRR and Margin (Table 6, Appendix C.5), and ranking quality on recommendation and QA via Recall/NDCG at k ∈ {1, 3} (main Table 1; Appendix C.5, Tables 2 and 5). Across base models and datasets, MASS-DPO improves MRR over S-DPO and delivers higher or comparable Margins. On ranking metrics, MASS-DPO attains best or tied-best scores on most cells across all four metrics {R@1, R@3, N@1, N@3}, demonstrating stronger ranking quality on both recommendation and QA. These results confirm that active negative selection produces harder, more informative training pairs that translate into stronger ranking and alignment quality. 6.4
Ablation Studies
MASS-DPO’s behavior is governed by the Fisher-information (Equation (11)) and the D-optimal selection objective (Equation (12)). We therefore ablate two key knobs predicted by theory to matter most: the preference-logit scale β and the number of selected negatives n. Effect of β. The scale β is shared between the DPO training loss and the D-optimal selection objective, where it controls the sharpness of the softmax weights over candidates. Sweeping β ∈ {0.1, 0.5, 1.0} across three model families (Table 3a), we find β = 0.1 consistently yields the strongest results. Number of negatives (n). D-optimal design predicts that adding more negatives improves parameter estimation until coverage of the information space saturates. Varying the selected-negative budget n ∈ {1, 3, 5} shows monotonic gains from n=1 → 3 → 5 across models and datasets (Table 3b). These results indicate the incremental sample selection procedure reliably assembles complementary negatives that expand log det of the information matrix, aligning empirical improvements with our D-optimal design analysis.
7
Conclusion
We introduced MASS-DPO, a within-prompt active negative selection method for multi-negative preference optimization under the Plackett–Luce model. By deriving a PL-specific Fisher-information objective and formulating negative selection as a D-optimal design problem, MASS-DPO selects compact subsets that retain the full pool’s information while reducing redundancy via an efficient incremental rank-one algorithm. Experiments across four benchmarks and three model families confirm that MASS-DPO delivers stronger alignment with substantially fewer negatives. 9
References [1] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011. [2] Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal discrete optimization for experimental design: A regret minimization approach. Mathematical Programming, 186:439–478, 2021. [3] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024. [4] Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927–8939, 2021. [5] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671, 2019. [6] Zhuoxi Bai, Ning Wu, Fengyu Cai, Xinyi Zhu, and Yun Xiong. Finetuning large language model for personalized ranking. arXiv preprint arXiv:2405.16127, 2024. [7] Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3, 2025. [8] Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011. [9] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029. [10] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science, pages 273–304, 1995. [11] Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. arXiv preprint arXiv:2406.09215, 2024. [12] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4302–4310, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. [13] David Cohn. Neural network exploration using optimal experiment design. Advances in neural information processing systems, 6, 1993. [14] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402.10500, 2024. [15] Lu Fan, Jiashu Pu, Rongsheng Zhang, and Xiao-Ming Wu. Neighborhood-based hard negative mining for sequential recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2042–2046, 2023. [16] R. A. Fisher and Edward John Russell. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594-604):309–368, 1922. doi: 10.1098/rsta.1922.0009. URL https://royalsocietypublishing.org/doi/abs/ 10.1098/rsta.1922.0009. [17] Patrick Flaherty, Adam Arkin, and Michael Jordan. Robust design of biological experiments. Advances in neural information processing systems, 18, 2005. 10
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [19] Janghoon Han, Dongkyu Lee, Joongbo Shin, Hyunkyung Bae, Jeesoo Bang, Seonghwan Kim, Stanley Jungkyu Choi, and Honglak Lee. Efficient dynamic hard negative sampling for dialogue selection. In Elnaz Nouri, Abhinav Rastogi, Georgios Spithourakis, Bing Liu, Yun-Nung Chen, Yu Li, Alon Albalak, Hiromi Wakaki, and Alexandros Papangelis, editors, Proceedings of the 6th Workshop on NLP for Conversational AI (NLP4ConvAI 2024), pages 89–100, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.nlp4convai-1.6/. [20] F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015. [21] Xiaoxin He, Nurendra Choudhary, Jieyi Jiang, Edward W Huang, Bryan Hooi, Xavier Bresson, and Karthik Subbian. Reclaif: Reinforcement learning from ai feedback for recommendation systems. 2025. [22] Chengkai Huang, Junda Wu, Zhouhang Xie, Yu Xia, Rui Wang, Tong Yu, Subrata Mitra, Julian McAuley, and Lina Yao. Pluralistic off-policy evaluation and alignment. arXiv preprint arXiv:2509.19333, 2025. [23] Hongtao Huang, Chengkai Huang, Junda Wu, Tong Yu, Julian McAuley, and Lina Yao. Listwise preference diffusion optimization for user behavior trajectories prediction. Advances in Neural Information Processing Systems, 38:159383–159408, 2026. [24] Zihan Huang, Junda Wu, Rohan Surana, Raghav Jain, Tong Yu, Raghavendra Addanki, David Arbour, Sungchul Kim, and Julian McAuley. Traceable and explainable multimodal large language models: An information-theoretic view. In Second Conference on Language Modeling, 2025. [25] Zihan Huang, Junda Wu, Rohan Surana, Tong Yu, David Arbour, Ritwik Sinha, and Julian McAuley. Image difference captioning via adversarial preference optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33746– 33758, 2025. [26] Zihan Huang, Xintong Li, Rohan Surana, Tong Yu, Rui Wang, Julian McAuley, Jingbo Shang, and Junda Wu. Amps: Adaptive modality preference steering via functional entropy. arXiv preprint arXiv:2602.12533, 2026. [27] Yongsu Jung and Ikjin Lee. Optimal design of experiments for optimization-based model calibration using fisher information matrix. Reliability Engineering & System Safety, 216: 107968, 2021. ISSN 0951-8320. doi: https://doi.org/10.1016/j.ress.2021.107968. URL https: //www.sciencedirect.com/science/article/pii/S0951832021004798. [28] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 33:21798–21809, 2020. [29] Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8082–8090, Apr. 2020. doi: 10.1609/aaai.v34i05.6319. URL https://ojs.aaai.org/index.php/AAAI/article/view/6319. [30] Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series B (Methodological), 21(2):272–304, 1959. [31] Andreas Kirsch and Yarin Gal. Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities. arXiv preprint arXiv:2208.00549, 2022. [32] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019. [33] Andreas Krause and Carlos E Guestrin. Near-optimal nonmyopic value of information in graphical models. arXiv preprint arXiv:1207.1394, 2012. 11
[34] Andreas Krause, Ajit Singh, and Carlos Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research, 9(2), 2008. [35] Branislav Kveton, Xintong Li, Julian McAuley, Ryan Rossi, Jingbo Shang, Junda Wu, and Tong Yu. Active learning for direct preference optimization. arXiv preprint arXiv:2503.01076, 2025. [36] Xintong Li, Chuhan Wang, Junda Wu, Rohan Surana, Tong Yu, Julian McAuley, and Jingbo Shang. Importance sampling for multi-negative multimodal direct preference optimization. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=HEFPwoGtTj. [37] Weibin Liao, Xu Chu, and Yasha Wang. Tpo: Aligning large language models with multi-branch & multi-step preference trees. arXiv preprint arXiv:2410.12854, 2024. [38] Pangpang Liu, Chengchun Shi, and Will Wei Sun. Dual active learning for reinforcement learning from human feedback. arXiv preprint arXiv:2410.02504, 2024. [39] R Duncan Luce et al. Individual choice behavior, volume 4. Wiley New York, 1959. [40] Haokai Ma, Ruobing Xie, Lei Meng, Fuli Feng, Xiaoyu Du, Xingwu Sun, Zhanhui Kang, and Xiangxu Meng. Negative sampling in recommendation: A survey and future directions. arXiv preprint arXiv:2409.07237, 2024. [41] Subhojyoti Mukherjee, Anusha Lalitha, Kousha Kalantari, Aniket Anand Deshmukh, Ge Liu, Yifei Ma, and Branislav Kveton. Optimal design for human preference elicitation. Advances in Neural Information Processing Systems, 37:90132–90159, 2024. [42] Gagan Mundada, Zihan Huang, Rohan Surana, Sheldon Yu, Jennifer Yuntong Zhang, Xintong Li, Tong Yu, Lina Yao, Jingbo Shang, Julian McAuley, et al. Ws-grpo: Weakly-supervised grouprelative policy optimization for rollout-efficient reasoning. arXiv preprint arXiv:2602.17025, 2026. [43] Tracianne B. Neilsen, David F. Van Komen, Mark K. Transtrum, Makenzie B. Allen, and David P. Knobles. Optimal experimental design for machine learning using the fisher information. Proceedings of Meetings on Acoustics, 35(1):055004, 01 2019. ISSN 1939-800X. doi: 10.1121/ 2.0000953. URL https://doi.org/10.1121/2.0000953. [44] G. L. Nemhauser and L. A. Wolsey. Best algorithms for approximating the maximum of a submodular set function. Mathematics of Operations Research, 3(3):177–188, 1978. ISSN 0364765X, 15265471. URL http://www.jstor.org/stable/3689488. [45] Bo Ni, Yu Wang, Leyao Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Luera, Samyadeep Basu, Subhojyoti Mukherjee, et al. A survey on llm-based conversational user simulation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4266–4301, 2026. [46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [47] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors, Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR, 07–08 Apr 2022. URL https://proceedings. mlr.press/v174/pal22a.html. [48] Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975. [49] Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006. [50] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 12
[51] Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127, 2018. [52] Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020. [53] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. [54] Jamshid Sourati, Murat Akcakaya, Todd K Leen, Deniz Erdogmus, and Jennifer G Dy. Asymptotic analysis of objectives based on fisher information in active learning. Journal of Machine Learning Research, 18(34):1–41, 2017. [55] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. [56] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. [57] Chao Sun, Yaobo Liang, Yaming Yang, Shilin Xu, Tianmeng Yang, and Yunhai Tong. Direct preference optimization for llm-enhanced recommendation systems. arXiv preprint arXiv:2410.05939, 2024. [58] Rohan Surana, Gagan Mundada, Xunyi Jiang, Chuhan Wang, Zhenwei Tang, Difan Jiao, Zihan Huang, Yuxin Xiong, Junda Wu, Sheldon Yu, et al. Generate, filter, control, replay: A comprehensive survey of rollout strategies for llm reinforcement learning. arXiv preprint arXiv:2605.02913, 2026. [59] Qwen Team. Qwen3, April 2025. URL https://qwenlm.github.io/blog/qwen3/. [60] Kiran Koshy Thekumparampil, Gaurush Hiranandani, Kousha Kalantari, Shoham Sabach, and Branislav Kveton. Comparing few to rank many: Active human preference learning using randomized frank-wolfe. arXiv preprint arXiv:2412.19396, 2024. [61] Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, and Jingbo Shang. Scenealign: Aligning multimodal reasoning to scene graphs in complex visual scenes. arXiv preprint arXiv:2601.05600, 2026. [62] Franklin Wang and Sumanth Hegde. Accelerating direct preference optimization with prefix sharing. arXiv preprint arXiv:2410.20305, 2024. [63] William J. Welch. Branch-and-bound search for experimental designs based on d optimality and other criteria. Technometrics, 24(1):41–48, 1982. ISSN 00401706. URL http://www. jstor.org/stable/1267576. [64] Junda Wu, Tong Yu, Rui Wang, Zhao Song, Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, and Ricardo Henao. Infoprompt: Information-theoretic soft prompt tuning for natural language understanding. Advances in neural information processing systems, 36:61060–61084, 2023. [65] Junda Wu, Xintong Li, Ruoyu Wang, Yu Xia, Yuxin Xiong, Jianing Wang, Tong Yu, Xiang Chen, Branislav Kveton, Lina Yao, et al. Ocean: Offline chain-of-thought evaluation and alignment in large language models. In International Conference on Learning Representations, volume 2025, pages 100570–100589, 2025. [66] Junda Wu, Rohan Surana, Zhouhang Xie, Yiran Shen, Yu Xia, Tong Yu, Ryan A. Rossi, Prithviraj Ammanabrolu, and Julian McAuley. In-context ranking preference optimization. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum? id=L2NPhLAKEd. [67] Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, and Bo Zheng. Mppo: Multi pair-wise preference optimization for llms with arbitrary negative samples. arXiv preprint arXiv:2412.15244, 2024. [68] Zhouhang Xie, Junda Wu, Yiran Shen, Raghav Jain, Yu Xia, Xintong Li, Aaron Chang, Ryan A. Rossi, Tong Yu, Sachin Kumar, Bodhisattwa Prasad Majumder, Jingbo Shang, Prithviraj 13
Ammanabrolu, and Julian McAuley. A survey on personalized and pluralistic preference alignment in large language models. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=lSWOMjonL7. [69] Zhi Yang, Jiwei Qin, Chuan Lin, Yanping Chen, Ruizhang Huang, and Yongbin Qin. Ganrec: A negative sampling model with generative adversarial network for recommendation. Expert Systems with Applications, 214:119155, 2023. ISSN 0957-4174. doi: https://doi.org/10.1016/ j.eswa.2022.119155. URL https://www.sciencedirect.com/science/article/pii/ S095741742202173X. [70] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives, 2021. URL https://arxiv.org/abs/ 2104.08051. [71] Zhaoyang Zhang, Xuying Wang, Xiaoming Mei, Chao Tao, and Haifeng Li. False: False negative samples aware contrastive learning for semantic segmentation of high-resolution remote sensing image. IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
A
Appendix
Lemma A.1 (Gradient Derivation). Consider the loss for a single sample h i X , L(θ) = − log σ Z(θ) , with Z(θ) = − log exp β ϕ⊤ j θ + bj
(17)
j∈Sn
1 σ(Z(θ))(1 − σ(Z(θ))) d − log σ(Z(θ)) = − · σ ′ (Z(θ)) = − = −(1 − σ(Z(θ))). dz σ(Z(θ)) σ(Z(θ)) ∂L = −(1 − σ(Z(θ))). ∂Z(θ) h X i A(θ) = exp β ϕ⊤ j θ + bj ,
(18)
j∈Sn
so that Z(θ) = − log A(θ). Then, ∂Z(θ) 1 ∂A(θ) =− , ∂θ A(θ) ∂θ h X i ∂A(θ) = exp β ϕ⊤ θ + b βϕj , j j ∂θ j∈Sn h i X exp β ϕ⊤ X j θ + bj ∂Z(θ) = −β ϕj = −β qjSn (θ)ϕj , ∂θ A(θ) j∈Sn
j∈Sn
where the softmax weights are defined as qjSn (θ) =
h i exp β ϕ⊤ j θ + bj A(θ)
.
h i X X ∂L ∂L ∂Z(θ) = · = −(1 − σ(Z(θ))) · −β qjSn (θ)ϕj = β(1 − σ(Z(θ))) qjSn (θ)ϕj . ∂θ ∂Z(θ) ∂θ j∈Sn
j∈Sn
Thus, the gradient of the loss is ∇θ L = β(1 − σ(Z(θ)))
X
qjSn (θ)ϕj .
j∈Sn
Equivalently, defining ϕ̄Sn (θ) =
Sn j∈Sn qj (θ)ϕj , we have ∇θ L = β(1 − σ(Z(θ)))ϕ̄Sn (θ).
P
14
(19)
Lemma A.2 (Hessian Derivation). Recall the multi-negative DPO loss: ! X
L(θ; Sn ) = − log σ − log
exp(β(ϕ⊤ i θ + bi )) ,
i∈Sn
where σ(·) denotes the sigmoid function. Throughout this proof we abbreviate the subset-normalized weights and subset mean from Theorem 4.3 as qj := qjSn (θ) and ϕ̄ := ϕ̄Sn (θ), and write Zn = − log
X
exp(β(ϕ⊤ i θ + bi )),
qj = P
exp(β(ϕ⊤ j θ + bj ))
⊤ k∈Sn exp(β(ϕk θ + bk ))
i∈Sn
,
ϕ̄ =
X
q j ϕj .
j∈Sn
Starting from the gradient Equation (9), ∇θ L(θ; Sn ) = β(1 − σ(Zn ))ϕ̄, we derive the Hessian by differentiating again with respect to θ: ∇2θ L(θ; Sn ) = β∇θ (1 − σ(Zn ))ϕ̄
(20)
= β(1 − σ(Zn ))∇θ ϕ̄ − βσ(Zn )(1 − σ(Zn ))ϕ̄ ∇θ Zn⊤ . Expanding the first term using the definition of qj gives: ⊤ X X X ∇θ ϕ̄ = β qj ϕj ϕ⊤ q j ϕj qj ϕ j j −β j∈Sn
=β
X
j∈Sn
(21)
(22)
j∈Sn
qj (ϕj − ϕ̄)(ϕj − ϕ̄)⊤ .
(23)
j∈Sn
Note also that: ∇θ Zn = −β
X
qj ϕj = −β ϕ̄.
j∈Sn
Thus, substituting back, the Hessian becomes: X ∇2θ L(θ; Sn ) = β 2 (1 − σ(Zn )) qj (ϕj − ϕ̄)(ϕj − ϕ̄)⊤ + β 2 σ(Zn )(1 − σ(Zn ))ϕ̄ϕ̄⊤
(24)
j∈Sn
= β 2 (1 − σ(Zn )) σ(Zn ) ϕ̄ϕ̄⊤ +
X
qj (ϕj − ϕ̄)(ϕj − ϕ̄)⊤ .
(25)
j∈Sn
Remark A.3 (Loewner lower bound in Equation (10)). From the decomposition above, h i X ∇2θ L(θ; Sn ) = β 2 (1 − σ(Zn )) σ(Zn ) ϕ̄ϕ̄⊤ + qj (ϕj − ϕ̄)(ϕj − ϕ̄)⊤ . j∈Sn
The rank one term σ(Zn ) ϕ̄ϕ̄⊤ is positive semidefinite, and the weighted covariance term is also positive semidefinite. Therefore dropping the rank one term yields the Loewner lower bound X ∇2θ L(θ; Sn ) ⪰ β 2 (1 − σ(Zn )) qj (ϕj − ϕ̄)(ϕj − ϕ̄)⊤ , j∈Sn
which is Equation (10). Theorem A.1 (Selected-Subset Estimator Stability). Let γ γ FC (θ) = L(θ; C) + ∥θ∥22 , FSn (θ) = L(θ; Sn ) + ∥θ∥22 , 2 2 and let θ∗ = arg minθ∈Rd FC (θ) and θ̂n = arg minθ∈Rd FSn (θ). Define the integrated subset curvature Z 1 Σ̄n = ∇2 FSn θ∗ + t(θ̂n − θ∗ ) dt. 0
15
If Σ̄n ⪰ Σ0 ≻ 0, then ∥θ̂n − θ∗ ∥Σ0 ≤ ∥∇FSn (θ∗ )∥Σ−1 = ∥∇L(θ∗ ; Sn ) − ∇L(θ∗ ; C)∥Σ−1 . 0
0
Thus an estimator-stability event of the form used in Theorem 5.1 is guaranteed whenever the selected-subset gradient discrepancy on the right is at most ηn in the corresponding dual norm. Proof. The first-order condition for the selected regularized objective gives ∇FSn (θ̂n ) = 0. By the fundamental theorem of calculus, 0 = ∇FSn (θ∗ ) + Σ̄n (θ̂n − θ∗ ). Therefore θ̂n − θ∗ = −Σ̄−1 n ∇FSn (θ∗ ). Since Σ̄n ⪰ Σ0 , the stated norm bound follows. Finally, ∇FC (θ∗ ) = 0, so ∇FSn (θ∗ ) = ∇L(θ∗ ; Sn ) − ∇L(θ∗ ; C). Theorem A.2 (Batch Estimator Stability). For an analysis block of k prompts, define the averaged full-pool and selected-subset losses k
Lk (θ; C1:k ) =
k
1X Li (θ; Ci ), k i=1
Lk (θ; S1:k,n ) =
1X Li (θ; Si,n ). k i=1
Let FC,k (θ) = Lk (θ; C1:k ) + γ2 ∥θ∥22 and FS,k (θ) = Lk (θ; S1:k,n ) + γ2 ∥θ∥22 , with minimizers θ∗,k and θ̂k,n respectively. If the integrated Hessian of FS,k along the segment from θ∗,k to θ̂k,n lower-bounds Σ0,k ≻ 0, then ∥θ̂k,n − θ∗,k ∥Σ0,k ≤ ∥∇Lk (θ∗,k ; S1:k,n ) − ∇Lk (θ∗,k ; C1:k )∥Σ−1 . 0,k
Moreover, k
∇Lk (θ; S1:k,n ) =
X S βX (1 − σ(Zi (θ))) qi,ji,n (θ)ϕi,j . k i=1 j∈Si,n
Proof. The proof is identical to the single-prompt perturbation argument in Theorem A.1, with FC and FSn replaced by the averaged objectives FC,k and FS,k . The displayed gradient follows by differentiating the average loss term by term, which introduces the factor 1/k.
B
Formal Relative Logit Error Bound
Theorem B.1 (Relative Logit Error Bound — Formal Version of Theorem 5.1). For a fixed prompt, let C = {yi }N i=1 be its candidate-negative pool. Let h i h i γ γ θ∗ = arg min L(θ; C) + ∥θ∥22 , θ̂n = arg min L(θ; Sn ) + ∥θ∥22 , 2 2 θ∈Rd θ∈Rd where Sn ⊆ C is the selected subset. Define the fixed full-pool weights used by Algorithm 1, exp(β(ϕ⊤ i θ0 + bi )) , ⊤ ℓ∈C exp(β(ϕℓ θ0 + bℓ ))
qi0 = P with ZC0 = − log
ϕ̄0 =
X i∈C
⊤ 0 0 ℓ∈C exp[β(ϕℓ θ0 + bℓ )]. Set ϕ̃i = ϕi − ϕ̄0 , vi =
P
In = {i : yi ∈ Sn },
Hn0 = γI + α0
X
vi0 (vi0 )⊤ ,
qi0 ϕi ,
p
qi0 ϕ̃0i ,
α0 = β 2 (1 − σ(ZC0 )).
i∈In 0 Let qmin = mini∈C qi0 , L0v = maxi∈C ∥vi0 ∥22 , and define
Bn =
α0 nL0v 1 + α0 L0v /γ d κ · · log 1 + . 0 ρqmin α0 n γd 16
Under Assumptions C.1–C.4, with probability at least 1−δ over the stochastic preference observations drawn from the PL model, ! r log(1/δ) e Erel (θ̂n , θ∗ ) ≤ O d , n e hides logarithmic factors and the candidate-pool regularity constants ρ, q 0 , κ, α0 , γ, L0v . where O min Proof. Treating the multi-negative loss as a generalized linear model and applying self-normalized concentration to the regularized estimator [1, 35], with probability at least 1 − δ, p ∥θ̂n − θ∗ ∥Σn ≤ C d log(1/δ), for a constant C depending on β, cmin , cmax , γ from Assumptions C.1–C.2. For any i, j ∈ C, since ϕi − ϕj = ϕ̃0i − ϕ̃0j , Cauchy–Schwarz gives |(ϕi − ϕj )⊤ (θ̂n − θ∗ )| ≤ ∥ϕ̃0i ∥Σ−1 + ∥ϕ̃0j ∥Σ−1 ∥θ̂n − θ∗ ∥Σn . n n √ 0 By Theorem C.1 under Assumption C.4, (ϕ̃0i )⊤ Σ−1 Bn . n ϕ̃i ≤ Bn , so each leverage term is at most Combining, p p e d log(1/δ)/n , Erel ≤ 2 Bn ∥θ̂n − θ∗ ∥Σn ≤ O e where the candidate-pool constants are absorbed into O.
C
Technical Assumptions
C.1
Assumption Details
Assumption C.1 (Bounded Feature Differences and Bias). For each prompt and candidate set, the feature-difference vectors used in the loss, ϕi = ϕ(x, yi ) − ϕ(x, y ∗ ), and the reference-policy offsets are bounded: ∥ϕi ∥2 ≤ Lϕ , |bi | ≤ Lb . For the theoretical analysis, the ridge-regularized objectives are optimized over Rd ; we assume the relevant minimizers lie in a ball ∥θ∗ ∥2 , ∥θ̂n ∥2 ≤ Rθ . The bounds hide polynomial dependence on the finite constants Lϕ and Lb . Assumption C.2 (Bounded Curvature Scale on the Relevant Region). Let ZC (θ) = P − log j∈C exp[β(ϕ⊤ j θ + bj )]. On the compact parameter region containing θ0 , θ∗ , θ̂n , and the line segments used in the perturbation arguments of Appendix A, there exist constants 0 < cmin ≤ cmax ≤ β 2 such that, for all θ in this region, cmin ≤ β 2 (1 − σ(ZC (θ))) ≤ cmax . Assumption C.3 (Diverse Candidate Set). Let qj0 and ϕ̄0 be the full-pool softmax weights and centerqcomputed by Algorithm 1 at preprocessing parameter θ0 , and write ZC0 = ZC (θ0 ). Let vj0 =
qj0 (ϕj − ϕ̄0 ) be the fixed full-pool-centered Fisher contribution used by Algorithm 1, and
let α0 = β 2 (1 − σ(ZC0 )). There exists a constant κ ≥ 1 such that for any selected index set Ik produced by Algorithm 1 after k steps (k = 0, 1, . . . , n − 1, with I0 = ∅), with design matrix P Hk0 = γI + α0 j∈Ik vj0 (vj0 )⊤ , we have (vi0 )⊤ (Hk0 )−1 vi0 ≤ κ · max (vj0 )⊤ (Hk0 )−1 vj0 ,
∀ i ∈ [N ], ∀ k ∈ {0, 1, . . . , n − 1}.
j∈[N ]\Ik
Here κ measures how well the remaining pool continues to cover high-leverage directions after each greedy step; κ = 1 in the variant allowing reselection. Assumption C.4 (Fisher-Compatibility). There exists ρ ∈ (0, 1] such that the regularized subset Hessian dominates the full-pool selection objective up to a constant: Σn := γI + ∇2 L(θ∗ ; Sn ) ⪰ ρ Hn0 , where Hn0 = γI + α0 selected index set.
0 0 ⊤ i∈In vi (vi ) is the selection objective of Equation (11) evaluated on the
P
17
C.2
Centered Leverage Score Bound
0 The following result shows that the centered leverage scores decay at rate Õ(d/(ρqmin n)) under Assumptions C.1–C.4. This is the key ingredient linking the Σn -norm estimation error to the relative logit error in Theorem 5.1. p Theorem C.1 (Centered Leverage Score Decay). Let ϕ̃0i = ϕi − ϕ̄0 , vi0 = qi0 ϕ̃0i , and Hk0 = Pk 0 γI + α0 t=1 vi0t (vi0t )⊤ , where α0 = β 2 (1 − σ(ZC0 )) and ZC0 = ZC (θ0 ). Let qmin = mini∈C qi0 and 0 0 2 Lv = maxi∈C ∥vi ∥2 . Under Assumptions C.1–C.4, for any candidate i ∈ C and any subset Sn of size n produced by Algorithm 1, κ 1 + α0 L0v /γ d α0 nL0v (ϕi − ϕ̄0 )⊤ Σ−1 · . · log 1 + n (ϕi − ϕ̄0 ) ≤ 0 ρqmin α0 n γd
Proof. Let 0 xk = (vi0k )⊤ (Hk−1 )−1 vi0k . 0 ⊤ 0 −1 0 By the max-marginal construction in Algorithm 1, xk = maxj ∈I vj . By Assump/ k−1 (vj ) (Hk−1 ) tion C.3, for any i ∈ C, 0 (vi0 )⊤ (Hk−1 )−1 vi0 ≤ κxk . 0 0 Since Hn0 ⪰ Hk−1 , we also have (Hn0 )−1 ⪯ (Hk−1 )−1 , and therefore
(vi0 )⊤ (Hn0 )−1 vi0 ≤ κxk , ∀k ∈ [n]. P n Thus (vi0 )⊤ (Hn0 )−1 vi0 ≤ κ mink xk ≤ nκ k=1 xk . P It remains to bound k xk . The matrix determinant lemma gives 0 log det Hk0 − log det Hk−1 = log(1 + α0 xk ). 0 Because Hk−1 ⪰ γI and ∥vi0 ∥22 ≤ L0v , we have xk ≤ L0v /γ. Hence α0 xk log(1 + α0 xk ) ≥ , 1 + α0 L0v /γ
and
n X
xk ≤
k=1
1 + α0 L0v /γ log det Hn0 − log det H00 . α0
Using the determinant upper bound under a fixed trace, Pn α0 k=1 ∥vi0k ∥22 α0 nL0v 0 0 log det Hn − log det H0 ≤ d log 1 + ≤ d log 1 + . γd γd Combining these inequalities yields α0 nL0v 1 + α0 L0v /γ d (vi0 )⊤ (Hn0 )−1 vi0 ≤ κ log 1 + . α0 n γd p Finally, since ϕ̃0i = vi0 / qi0 and Σn ⪰ ρHn0 by Assumption C.4, 0 (ϕ̃0i )⊤ Σ−1 n ϕ̃i ≤
1 1 (v 0 )⊤ (Hn0 )−1 vi0 ≤ 0 (vi0 )⊤ (Hn0 )−1 vi0 , ρqi0 i ρqmin
0 which proves the claimed bound. When qmin is treated as a fixed candidate-pool regularity constant, the displayed expression simplifies to Õ(d/(ρn)).
C.3
Experimental Settings
To further manage computational costs, we cap the number of response candidates at 20 for the LastFM, MovieLens, and MedMCQA datasets, and at 8 for QASC. Although MedMCQA natively provides only four options per question, we expand this to 20 by pooling all candidates with the same subject_name field. We also subsample each dataset to 20k training samples, 200 samples for online evaluation, and 2,000 samples for testing. All prompts are formatted using each model’s provided chat template to ensure consistent input structure across tasks. 18
LlaMA3_MASS-DPO LlaMA3_S-DPO
Margin
QASC
8
SmolLM3_MASS-DPO SmolLM3_S-DPO
Accuracy
Chosen Rewards
0.8
6
0.6
2
4
0.4
1
2
0.2
0
MovieLens
Qwen3_MASS-DPO Qwen3_S-DPO
0
5
10
15
6
0.0 0.6
0 0
5
10
15
2
0.2
0
0.0
0
5
10
Step
15
0
5
0
5
10
15
10
15
0.5
0.4
4
0.0 1.0 1.5
0
5
10
Step
15
2.0
Step
Figure 3: Margin, accuracy, and chosen reward comparisons on MovieLens and QASC datasets. MASS-DPO achieves higher margins, superior accuracy, and more stable chosen rewards than S-DPO. The x-axis (Step) counts evaluations during training. Scope. Our evaluation is scoped to settings with a finite per-prompt candidate pool: recommendation (LastFM, MovieLens) and multiple-choice QA (MedMCQA, QASC), where negatives are welldefined and bounded in number. Extending MASS-DPO to open-ended generation or instruction tuning would require an upstream pool-construction step (e.g., sampling negatives from the reference policy), which we do not evaluate here. C.4
Implementation Details
We implement our experiments using PyTorch, leveraging three widely used pre-trained LLMs: Llama-3.2-3B-Instruct [18], SmolLM3 [7], and Qwen3-4B [59]. Each model undergoes full finetuning on 8 NVIDIA A100 GPUs with a per-device batch size of 2, gradient accumulation steps of 8, learning rate of 10−5 , a cosine learning-rate scheduler with warmup ratio 0.05, and the Paged AdamW optimizer for 3 epochs with a fixed DPO scale β = 0.1 across all main experiments. We enable gradient checkpointing, gradient clipping is applied with a maximum norm of 0.3, and evaluation uses a batch size of 2. We extract representation vectors by mean-pooling the final hidden states, using either (a) all tokens from the concatenated prompt–response sequence or (b) only the response tokens, where prompt positions are masked out. Both strategies use the same pretrained LLM and tokenization pipeline. We compute the negative subset Sn during dataset preprocessing, using the frozen preprocessing checkpoint θ0 to obtain embeddings and log-probabilities; the selected subsets remain fixed throughout training. The preprocessing score is si = β ϕ⊤ bi = log πref (y ∗ | x) − log πref (yi | x), i θ0 + b i , where β is the same value used in the DPO/PL training loss and θ0 is the pretrained initialization before preference fine-tuning. Stability of fixed selection across training. To assess whether the pre-selected negatives remain informative as training progresses, we recompute the selection at initialization, mid-training (epoch 1), and the final checkpoint on a 500-sample subset of MovieLens with Llama-3.2-3B-Instruct for selected-negative budgets n ∈ {5, 10}. Across both budgets, the subset obtained at initialization and the one recomputed at the final checkpoint share at least 98.8% exact overlap and 0.997 mean Jaccard similarity (Table 4); the self log-det values vary by at most 1% across checkpoints. These results indicate that the Fisher geometry is stable over the training horizon and dynamic re-selection would recover essentially the same subset. For the main comparison, we fix n=3 negatives for all multi-negative methods, DPO-k, DMPO, S-DPO, and MASS-DPO, so these methods use the same number of negative responses per prompt. DPO uses a single negative by construction. MASS-DPO incurs only the additional selection overhead of Algorithm 1, which is amortized as a one-time preprocessing cost. 19
Table 4: Stability of fixed MASS-DPO selection across training checkpoints on MovieLens with Llama-3.2-3B-Instruct. Metrics compare the initialization-time subset against the subset recomputed at the final checkpoint. n
Exact match
Mean Jaccard
Top-1 match
Top-3 exact match
Obj. retention
5 10
99.2% 98.8%
0.9969 0.9970
100% 100%
99.8% 99.6%
0.9994 0.9990
Hyperparameters. For the D-optimal selection objective (Equation (11)), we set the ridge γ = 0.1 for all runs to ensure H(S) is well conditioned, and we use the same β as in training. In the main experiments β = 0.1, and we only vary β in the ablation. C.5
Results
Table 5: Recall (R) and NDCG (N) at k={1,3} on MedMCQA and QASC. Each entry reports metricSE , where the subscript denotes standard error. MedMCQA R@3 N@1
Model
Method
Qwen3
DPO DMPO DPO-k S-DPO MASS-DPO
39.251.09 26.020.98 54.591.11 51.031.12 56.341.11
84.510.81 74.890.97 89.670.68 86.520.76 89.720.68
SmolLM3
DPO DMPO DPO-k S-DPO MASS-DPO
33.331.06 26.020.98 44.161.11 45.461.11 44.811.11
Llama3
DPO DMPO DPO-k S-DPO MASS-DPO
51.481.12 24.360.96 71.131.01 72.331.00 71.231.01
R@1
QASC N@3
R@1
R@3
N@1
N@3
39.251.09 26.020.98 54.591.11 51.031.12 56.341.11
65.370.77 53.600.81 74.860.72 71.540.77 75.620.72
67.771.55 68.211.55 71.081.51 70.421.52 71.851.49
90.620.97 90.510.97 92.720.86 91.500.93 91.610.92
67.771.55 68.211.55 71.081.51 70.421.52 71.851.49
81.291.04 81.401.04 83.950.96 83.041.00 83.730.99
81.750.86 75.390.96 85.910.78 87.220.75 87.470.74
33.331.06 26.020.98 44.161.11 45.461.11 44.811.11
61.010.78 53.930.80 68.090.77 69.540.75 69.400.74
67.111.56 66.111.57 70.311.52 70.531.51 72.521.48
90.070.99 88.741.05 90.180.99 91.390.93 91.830.91
67.111.56 66.111.57 70.311.52 70.531.51 72.521.48
80.701.06 79.541.10 82.061.05 82.801.01 83.820.99
87.820.73 74.990.97 93.880.54 94.340.52 94.490.51
51.481.12 24.360.96 71.131.01 72.331.00 71.231.01
72.300.75 52.840.80 84.510.62 85.200.61 84.840.61
71.301.50 69.321.53 73.951.46 74.171.45 73.841.46
91.720.92 91.940.90 92.380.88 92.380.88 92.600.87
71.301.50 69.321.53 73.951.46 74.171.45 73.841.46
83.411.00 82.670.99 84.930.96 85.130.96 85.130.95
Table 6: MRR and Margin across four datasets. Each cell shows MRR / Margin. Model
Method
MedMCQA MRR/Margin
QASC MRR/Margin
LastFM MRR/Margin
MovieLens MRR/Margin
Average↑ MRR/Margin
Qwen3
S-DPO MASS-DPO
69.74 / 9.33 73.30 / 11.76
81.87 / 5.29 82.71 / 6.22
64.12 / 4.67 66.28 / 5.51
61.29 / 4.98 61.61 / 4.94
69.26 / 6.07 70.97 / 7.11
SmolLM3
S-DPO MASS-DPO
66.64 / 19.10 66.30 / 14.42
81.64 / 8.07 82.73 / 7.91
70.13 / 7.62 70.79 / 7.29
68.73 / 7.32 68.13 / 7.41
71.78 / 10.53 71.99 / 9.26
Llama3
S-DPO MASS-DPO
83.44 / 23.93 82.86 / 21.36
84.13 / 7.26 84.06 / 7.24
70.58 / 6.42 70.71 / 6.53
63.92 / 5.20 65.58 / 5.75
75.52 / 10.70 75.80 / 10.22
C.6
Computational Cost and Full-Pool Comparison
To quantify the computational trade-off between active selection and full-pool training, we measure wall-clock times on MovieLens with Llama-3.2-3B using 4×H100 GPUs. Training with the full negative pool (S-DPO-19) is 1.85× slower per epoch than MASS-DPO with n=5 selected negatives. The one-time selection cost of 45.5 min is dominated by feature and log-probability extraction (42 min); the core selection step (Algorithm 1) itself takes only 3.5 min via rank-one updates. Over two epochs, S-DPO-19 incurs ∼9.4 extra GPU-hours relative to MASS-DPO-5, far exceeding the entire selection cost. Table 8 compares MASS-DPO (n=5) at epoch 2 against S-DPO-all (n=19) at epoch 1 on MovieLens with Llama-3.2-3B at matched wall-clock budgets (∼11h vs. ∼10.2h). 20
Table 7: Wall-clock cost breakdown. MASS-DPO selection is a one-time preprocessing step; perepoch training cost scales with the number of negatives. Component
Wall-clock
Feature & log-prob extraction D-optimal subset selection (Algorithm 1) MASS-DPO selection total (one-time)
42.0 min 3.5 min 45.5 min
MASS-DPO-5 training (per epoch) S-DPO-19 training (per epoch)
5.52 h 10.21 h
Table 8: MASS-DPO-5 vs. S-DPO-all at matched wall-clock budget on MovieLens + Llama-3.2-3B. Method
Epochs
MASS-DPO (n=5) S-DPO-all (19 neg)
2 1
Runtime (h)
Acc
R@1
R@3
R@5
NDCG@1
NDCG@3
NDCG@5
MRR
Margin
11.03 10.21
0.604 0.605
0.606 0.613
0.810 0.781
0.870 0.853
0.606 0.613
0.726 0.711
0.751 0.740
0.725 0.718
5.049 3.904
At matched compute, MASS-DPO-5 matches S-DPO-all on accuracy while outperforming it on R@3, R@5, NDCG@3, NDCG@5, and MRR, indicating that actively selected negatives provide more useful training signal per gradient step than the full pool.
21