arXiv CS open access

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging Aaryan Ajay Sharma1,2,∗,† Sai Nishanth Padala1,∗,† Seganrasan Subramanian1 1 ServiceNow 2 University of Twente aaryan.sharma@utwente.nl, p.snishanth996@gmail.com, seganrasan.subramanian@servicenow.com

Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, from code generation (Chen et al., 2021; Rozière et al., 2024) and mathematical reasoning (Cobbe et al., 2021; Luo et al., 2025; Wei et al.,

vs llama2-7b-code (code)

1.6 1.4

100

1.2

vs wizardmath-7b-v1.0 (math)

150

0.8 0.6

200

vs llama2-7b-instruct (instruct)

250 3000

1.0

L1 Bias

Model merging combines multiple task-specific fine-tuned LLMs into a single multi-task model without additional training. However, merged models are known to suffer from representation bias: systematic drift between the merged model’s hidden states and those of each individual source model. Prior work (Yang et al., 2024a) study and mitigate this bias for encoderbased vision models using a lightweight correction module trained with L1 loss. However, such bias is not studied for decoder models due to their autoregressive nature. Therefore, we first analyze the problem of representation bias in decoder models, and show two challenges absent in encoders: (1) the causal attention mask causes bias to accumulate across token positions, requiring position-dependent correction; and (2) not all token positions are equally important, i.e., high-entropy (decisioncritical) positions matter far more than lowentropy ones. To address these challenges, we propose Decoder-Aware Representation Tuning via Surgery (DARTS). DARTS employs a novel entropy-weighted L1 loss to upweight correction at high-entropy positions where errors most affect generation quality, and a perposition additive bias that captures positiondependent error without overparameterization. We perform extensive evaluation on three domains: code generation (HumanEval), mathematical reasoning (GSM8K), and instruction following (AlpacaEval) on Llama-2-7B models, and show DARTS achieves significant improvement over the standard surgery approach while adding negligible parameters (0.1% of total parameters).

Sample Index

arXiv:2608.28547v1 [cs.LG] 28 Aug 2026

Abstract

100

150

Token Index

200

0.4 0.2

250

Figure 1: Per-token L1 representation bias between a merged Llama-2-7B model and each individual finetune, measured at the final transformer layer. Each row is a calibration sample; each column is a token position. Across all three domains (code, math, instruct), we see an increasing trend of bias with token position due to error accumulation under the causal attention mask. This position-dependent structure motivates the decoder-aware corrections proposed in DARTS.

2022) to open-ended instruction following (Ouyang et al., 2022). However, these capabilities are typically obtained through task-specific fine-tuning (Hu et al., 2021; Touvron et al., 2023), producing a proliferation of specialized models, each of which excels on its target domain but requires independent deployment. As the number of specialized models grows, so does the cost of serving them, making multi-task consolidation increasingly attractive. Model merging (Wortsman et al., 2022; Ilharco et al., 2023; Yadav et al., 2023; Yang et al., 2024b; Yadav et al., 2025) offers an elegant solution to this problem: combine the weights of multiple fine-tuned models into a single model that inherits capabilities from all source models, without any *

Equal contribution. Author order determined by quantum random number generator over Google Meet. † Corresponding authors. Work done during Aaryan’s internship at ServiceNow.

additional training data or computation. Methods such as Task Arithmetic (Ilharco et al., 2023), TIES-Merging (Yadav et al., 2023), and DARE (Yu et al., 2024) have shown that simple arithmetic operations in weight space (such as adding scaled task vectors, resolving sign conflicts, or dropping redundant parameters) can produce surprisingly effective multi-task models. This has made model merging a standard tool in the open-source LLM ecosystem (Goddard et al., 2024), where practitioners routinely merge instruction-tuned, code-specialized, and math-specialized models into unified checkpoints. Despite this progress, a substantial performance gap persists between merged models and their individual fine-tuned counterparts. Merged models frequently degrade on one or more constituent tasks (Ilharco et al., 2023; Yadav et al., 2023; Yang et al., 2024b), and the severity of this degradation varies unpredictably across tasks, merging methods, and model scales. Understanding why merging hurts performance, and finding principled ways to close the gap, remains an open challenge. Yang et al. (2024a) recently offered a compelling explanation for this degradation. By visualizing the hidden representations of merged vision models (ViT-B/32, ViT-B/16, ViT-L/14 on eight classification tasks), they demonstrated that merged models suffer from representation bias: a systematic discrepancy between the feature representations extracted by the merged model and those of each individual fine-tuned model. Crucially, they showed that representation bias correlates with task performance: merging methods that produce smaller representation bias (e.g., AdaMerging (Yang et al., 2024b)) consistently outperform those with larger bias (e.g., Weight Averaging). Building on this insight, they proposed representation surgery: a lightweight low-rank module trained with L1 loss to predict and subtract the representation bias, achieving significant performance gains across all merging methods. However, the original surgery framework was designed exclusively for encoder-based vision models. In ViT architectures (Dosovitskiy et al., 2021), bidirectional self-attention ensures that each token (image patch) is contextualized by all other tokens equally, producing representation bias that is roughly uniform across token positions. A positionagnostic surgery module, which applies the same correction at every position, is well-suited to this setting. Likewise, the uniform L1 loss used for

training treats all token positions as equally important, which is reasonable when every patch contributes similarly to the classification decision. Decoder-based LLMs are fundamentally different. The autoregressive transformer architecture (Vaswani et al., 2017; Brown et al., 2020) processes tokens left-to-right under a causal attention mask: token t can only attend to tokens 0, 1, . . . , t−1. This architectural constraint has two consequences for representation bias that render the original surgery framework inadequate: First, representation bias in decoder models is position-dependent. Because each token’s representation is computed from a strictly causal context, any bias introduced by weight-space merging at earlier positions propagates forward through the attention mechanism. Later tokens aggregate not only the original merging error at their own position but also the accumulated errors from all preceding positions. Figure 1 empirically shows this effect: plotting per-token L1 distance between merged and individual model representations reveals a clear increasing trend, with later positions exhibiting 2–3× higher bias than early positions. A positionagnostic surgery module, which applies identical corrections at every position, is structurally incapable of capturing this pattern, forcing it to compromise between under-correcting later tokens and over-correcting earlier ones. Second, not all token positions are equally important for generation quality. In autoregressive LLMs, the model’s prediction at each position determines the next token in the generated sequence. At some positions, the model is highly confident and assigns most probability mass to a single token (e.g., closing brackets, deterministic syntax, common phrases). At other positions, the model distributes probability across many plausible continuations; these are the decision-critical positions where the choice of next token meaningfully affects the quality and correctness of the generated output. Small hidden-state errors at high-entropy positions can flip the argmax prediction and cascade through subsequent tokens, while comparable errors at low-entropy positions are absorbed without consequence. The standard L1 loss (Yang et al., 2024a) treats all positions uniformly, wasting the surgery module’s limited correction capacity on positions that do not affect downstream generation. Figure 2 illustrates this idea. To address these decoder-specific challenges, we propose Decoder-Aware Representation Tuning

via Surgery (DARTS), which adapts representation surgery to the autoregressive setting through two complementary innovations. First, we introduce entropy-weighted L1 loss (EWL1), which weights the per-position correction penalty by the target model’s prediction entropy, focusing the surgery module’s capacity on decision-critical positions where errors most affect generation quality. Second, we augment the low-rank content correction with a position correction module, a direct per-position additive bias table that learns independent correction vectors for each token position, capturing the structured position-dependent bias without network overparameterization. We evaluate DARTS on Llama-2-7B (Touvron et al., 2023) models merged from three specialized fine-tunes (code generation, mathematical reasoning, and instruction following), showing that it achieves significant improvement over the original surgery approach. The main contributions of this paper are as follows: 1. We analyze the problem of representation bias in decoder models and empirically demonstrate two challenges absent in encoder models: position-dependent bias accumulation under the causal attention mask, and nonuniform position importance arising from the autoregressive generation process. 2. To address these challenges, we propose DARTS, which uses a novel entropy-weighted L1 loss and a position correction module, to upweight correction at decision-critical positions, and perform position-dependent bias corrections. 3. We perform extensive experiments to show that DARTS reduces causal representation bias and significantly improves MTL performance on a variety of different tasks while adding merely 0.1% of parameters to the model.

Related Work

Model Merging for Multi-Task Learning. Model merging has emerged as a practical alternative to traditional multi-task learning (Crawshaw, 2020; Zhang and Yang, 2022), which requires centralized data collection and joint training. Early work showed that simply averaging model weights can improve generalization for models trained on

the same task (Wortsman et al., 2022), and subsequent methods extended this idea to merging models trained on different tasks. Task Arithmetic (Ilharco et al., 2023) introduced the concept of task vectors, defined as the difference τi = θi − θpre between fine-tuned and pretrained weights, and showed that adding scaled task vectors to the pretrained model produces effective multi-task models. TIES-Merging (Yadav et al., 2023) addressed parameter sign conflicts and redundancy by trimming, electing signs, and merging only aligned components. DARE (Yu et al., 2024) randomly drops delta parameters and rescales survivors to reduce interference. Fisher-Merging (Matena and Raffel, 2022) uses Fisher information to weight parameter importance, while RegMean (Jin et al., 2023) reweights based on training statistics. More recently, AdaMerging (Yang et al., 2024b) learns task-wise or layer-wise merging coefficients, Concrete (Tang et al., 2026) finds shared subspaces, and ZipIt (Stoica et al., 2024) merges models by aligning intermediate features. A comprehensive survey of these methods can be found in Yadav et al. (2025). All of the above methods operate exclusively in weight space and do not explicitly address the resulting representation-level discrepancies. Representation Surgery. Yang et al. (2024a) identified representation bias as a key obstacle in model merging: the hidden representations of merged ViT models (Dosovitskiy et al., 2021; Radford et al., 2021) diverge systematically from those of individual fine-tuned models, and this divergence correlates with task performance degradation. They proposed a lightweight adapterlike (Houlsby et al., 2019) correction module Φ(Z) = Wup · ReLU(Wdown · Z) trained with L1 loss to subtract the bias from merged representations. This approach was validated on eight image classification tasks across three ViT architectures, achieving consistent gains when applied on top of Weight Averaging, Task Arithmetic, TIESMerging, and AdaMerging. However, the original framework assumes position-agnostic correction and uniform loss weighting. These assumptions are well-motivated for bidirectional encoder models but, as we demonstrate, inadequate for autoregressive decoder models. Inference-Time Interventions and Routing. Beyond static weight-space merging, recent work has explored inference-time methods for LLM capability consolidation. Activation steering (Turner

Not all token positions are equally important: high-entropy (decision-critical) positions matter far more than low-entropy ones. Per-position Metrics

Example Generation (LLaMA-2-7B)

+ − × ÷

Mathematical Reasoning (GSM8K)

Q: If a train travels 60 km in 1.5 hours, what is its speed in km/h?

Inject bias at different positions: At low-importance position (“The”)

The Token-level Entropy (H)

1.0

is 0.9

2.8

1.5

2.6

2.9

40 1.1

km/h

3.2

1.1

3.3

Low

The speed is 60 / 1.5 = 40 km / h . Output quality: Unchanged At high-importance position (“40”)

High

Inject bias at different positions: At low-importance position (“def”)

def 1.2

is_prime 1.4

0.9

( 0.8

) 0.6

if 0.7

3.9

3.6

return 3.3

False

1.1

3.7

def is_prime ( n ) : if n < 2 : return False ...

... 3.8

Metric (Pass @ 1) High

At high-importance position (“return”) def is_prime ( n ) : if n < 2 : retrun False ...

Output quality: Fails test cases Low

Low

High

Inject bias at different positions:

Model output:

At low-importance position (“Regular”)

Regular exercise improves physical health , boosts mood , and reduces stress ...

Position Importance

High-imp. pos

Output quality: Unchanged ...

Q: Explain the benefits of regular exercise.

Token-level Entropy (H)

Metric (Accuracy / EM) High

The speed is 60 / 1.5 = 41 km / h . Low Low-imp. Output quality: Incorrect pos

Model output:

Position Importance

Instruction Following (AlpacaEval)

1.2

speed

Q: Write a function to check if a number is prime.

Code Generation Token-level (HumanEval) Entropy (H)

...

Inject same L1 bias magnitude at different positions

Model output:

Position Importance

</>

Effect of Equal-Magnitude Representation Bias

Low Importance (low entropy) High Importance (high entropy)

Regular exercise improves physical health boosts mood and reduces stress ...

Low-imp. pos

High-imp. pos

Metric (Win Rate) High

Output quality: Unchanged 1.2

1.4

2.6

0.8

0.6

0.7

3.0

1.2

0.3

1.1

2.3

1.2

...

At high-importance position (“reduces”) Regular exercise improves physical health boosts mood and increases stress ...

Low

High

Output quality: Worse response

Low

Low-imp. pos

High-imp. pos

Figure 2: Non-uniform importance of token positions in decoder models. We show three representative domains (math, code, instruction). For each example, we illustrate token-level entropy (H) and visualize position importance (higher entropy = higher importance). Injecting the same-magnitude representation bias δ at low-importance positions has negligible impact, while injecting at high-importance positions causes significant performance degradation. This motivates entropy-weighted losses that focus corrections where they matter most.

et al., 2023; Li et al., 2023) dynamically injects or subtracts steering vectors during the generation loop to modulate model behavior at test time. Mixture-of-Experts (MoE) architectures (Shazeer et al., 2017; Fedus et al., 2022) and dynamic routing mechanisms (Muqeeth et al., 2023) maintain multiple expert subnetworks and employ learned routers to select among them on a per-token or per-layer basis. Expert composition methods (Tang et al., 2024) combine multiple specialized adapters or modules at inference time. While these approaches offer flexible capability control, they differ fundamentally from DARTS in several practical dimensions. First, routing and MoE methods require loading multiple model blocks or expanded parameter sets in VRAM, whereas DARTS maintains a single model footprint with only ≈0.1% additional parameters. Second, inference-time steering and routing add dynamic computation branches that increase serving latency, while DARTS applies a static correction with latency identical to the base model. Third, DARTS produces a standard single-checkpoint model that serves as a drop-in replacement, requiring no specialized inference engine. We view these approaches as complementary: DARTS is best suited for low-overhead, permanent

consolidation of merged models, while inferencetime methods offer greater flexibility at the cost of deployment complexity.

Causal Representation Bias in Model Merging

We begin by establishing notation and defining the problem (Section 3.1), then empirically characterize the position-dependent structure of representation bias in decoder models (Section 3.3). We then present the two core components of DARTS: the position correction module (Section 3.4) and entropy-weighted L1 loss (Section 3.5). 3.1

Preliminaries

3.1.1

Notation

We consider K task-specific models obtained by fine-tuning a shared pretrained model on tasks T1 , . . . , TK . Let M denote the merged model and Mk∗ denote the k-th individual fine-tune (the pertask “target”). For an input sequence x1:T tokenized to length T , we denote: H • hM t ∈ R : the merged model’s final-layer hidden state at position t

H • h∗k t ∈ R : target model k’s corresponding hidden state

• WU∗k ∈ RV ×H : target model k’s LM head (unembedding matrix) with H = 4096 (hidden size, Llama-2-7B), V = 32,000 (vocabulary size), and Tmax = 512 (calibration max sequence length). For each task k, we learn a surgery module Sk that produces a corrected hidden state ĥt = Sk (hM )t . We drop the k subscript where clear from context. 3.1.2 Task Vectors and Merging Given a pretrained model θpre and a model finetuned on task k, θk , the task vector is τk = θk −θpre . Task Arithmetic (Ilharco et al., 2023) merges K tasks via: θmerged = θpre + λ

K X

τk

(1)

k=1

where λ is a scaling coefficient. Other merging methods (TIES-Merging, DARE, AdaMerging) modify how task vectors are combined but share the same general framework (Yadav et al., 2025). 3.1.3 Representation Bias Following Yang et al. (2024a), we define the perposition representation bias as the normalized L1 distance between the merged and target model’s hidden states: 1 δt = ∥hM − h∗k (2) t ∥1 H t In encoder models, Yang et al. showed that this bias is approximately uniform across token positions and correlates with downstream task performance. We extend this analysis to decoder models in the next section. 3.2

The original representation surgery (Yang et al., 2024a) employs a low-rank bottleneck with rank r and content correction function c(h) = Wup ReLU(Wdown h): (3)

where Wdown ∈ Rr×H and Wup ∈ RH×r . Following Yang et al. (2024a), Wup is zero-initialized so that the surgery module starts as the identity map, meaning an untrained module produces no correction. Parameter count: 2rH. Crucially, this module is position-agnostic: the same parameters are applied identically at every token position t.

Position-Dependent Bias in Decoders

To examine whether the uniform-bias assumption of encoder-based surgery transfers to decoder models, we compute the per-token L1 bias δt (Eq. 2) across a calibration set of prompts from each domain using Llama-2-7B merged models. The results (Figure 1) reveal a striking pattern absent in encoder models: representation bias exhibits a clear increasing trend with token position. Tokens at early positions (0–50) show relatively low bias, while tokens at later positions (100–256) exhibit significantly higher bias, often 2–3× the magnitude of early tokens. This pattern is consistent across all three domains (code, math, instruction following). The explanation follows directly from the causal attention mechanism (Vaswani et al., 2017). In encoder models (e.g., ViT), bidirectional attention distributes merging errors uniformly across all tokens. In decoder models, token t attends only to the preceding context {0, 1, . . . , t−1}. If earlier tokens carry representation bias from merging, token t aggregates these biased representations through attention, causing the bias to compound. Each successive position accumulates not only its own merging error but also the errors of all preceding positions, producing the observed monotonically increasing pattern. This has a direct implication for surgery module design: a position-agnostic correction, which applies the same magnitude of correction at every position, must compromise between undercorrecting later tokens (where bias is largest) and over-correcting earlier tokens (where bias is smallest). This motivates the position-aware correction described below. 3.4

Standard Surgery (Baseline)

M ĥt = hM t − c(ht )

3.3

Position Correction Module

To address the position-dependent structure of representation bias, we augment the content correction with a direct per-position additive bias b ∈ RTmax ×H , zero-initialized: ĥt = hM t −

c(hM ) | {zt }

content (input-dep.)

−

bt |{z}

position (input-indep.)

(4) The design decouples two sources of representation bias. The content correction c(hM t ) handles input-dependent errors, specifically the bias that varies with the specific input at position t. The position bias bt captures the systematic per-position offset that is consistent across inputs, reflecting

the cumulative effect of causal error accumulation documented in Section 3.3. A key design choice is that bt is a free parameter per position, rather than being parameterized by a network. Each position’s correction is independent of other positions and cannot compose. This provides structural regularization that prevents the position correction from overfitting to the small calibration set. We explored networkparameterized alternatives (viz., parallel adapters; see Appendix E), but found that the direct additive bias consistently matched or outperformed these more complex architectures while requiring no additional hyperparameters. Parameter count. 2rH + Tmax H. For Llama2-7B (H=4096, r=16, Tmax =512): ≈2.2M per domain, ≈6.6M total for 3 domains (≈0.1% of 7B). 3.5

Entropy-Weighted L1 Loss

The standard surgery objective (Yang et al., 2024a) minimizes the L1 distance between corrected and target representations, treating all token positions equally: LL1 =

1 X ∥ĥt − h∗t ∥1 H |T |

(5)

t∈T

where T is the set of non-padded positions. This uniform weighting is well-suited to classification tasks where all patch positions contribute similarly to the final prediction. In autoregressive generation, however, not all positions matter equally. We propose weighting each position by the target model’s prediction entropy. Let p∗t,v = softmax(WU∗ h∗t )v be the target model’s predicted P distribution at position t, and let Ht = − v p∗t,v log p∗t,v denote its entropy. The entropyweighted L1 loss is: LEWL1 =

1 X Ht ∥ĥt − h∗t ∥1 H |T | H̄

(6)

t∈T

P where H̄ = |T1 | t′ Ht′ is the mean entropy. The unit-mean normalization ensures that the overall loss scale remains comparable to standard L1 , allowing the same learning rate without adjustment. The intuition is as follows. At high-entropy positions, where the target model distributes probability across many plausible next tokens, small hidden-state errors can flip the argmax prediction, changing the token that would be generated and cascading through subsequent autoregressive

steps. At low-entropy positions (e.g., deterministic syntax, common phrases, punctuation), the model’s prediction is robust to perturbation: even a moderate hidden-state error leaves the argmax unchanged. By upweighting high-entropy positions, EWL1 focuses the surgery module’s limited correction capacity on the positions that actually affect generation quality, analogous to how importanceweighted distillation (Hinton et al., 2015) focuses on informative training examples.

Experimental Setup

4.1

Models and Merging

We use Llama-2-7B (Touvron et al., 2023) (H=4096, V =32,000) as the pretrained base and merge three publicly available fine-tunes: Llama2-7B-Instruct (Meta), WizardMath-7B-v1.0 (Luo et al., 2025), and Llama-2-7B-Code (Rozière et al., 2024). We evaluate four families of merging methods across an extensive hyperparameter sweep totaling over 124 configurations. Task Arithmetic (Ilharco et al., 2023) merges task vectors with a scaling coefficient λ ∈ {0.3, 0.5, 0.7, 0.9, 1.0}. Weight Averaging spans 22 weight distributions, including equal weighting, single-model dominant (40–80% weight on one model), and pairwiseheavy configurations. TIES-Merging (Yadav et al., 2023) is swept over 42 configurations combining weight ∈ {0.3, 0.5, 0.7, 1.0, 1.2, 1.5} and density ∈ {0.1, 0.2, 0.3, 0.5, 0.7, 0.9}, including both symmetric and asymmetric per-model weight assignments. DARE (Yu et al., 2024) covers 60 configurations across both dare_ties and dare_linear variants with weight ∈ {0.3, 0.5, 0.7, 1.0, 1.2} and density ∈ {0.1, 0.2, 0.3, 0.5, 0.7, 0.9}. All merged models are produced using mergekit (Goddard et al., 2024). For each merging method, we select the best-performing configuration by aggregate benchmark score as the representative baseline for that method in subsequent surgery experiments. 4.2

Evaluation Metrics

We evaluate on three benchmarks spanning complementary capabilities. For code generation, we use HumanEval (Chen et al., 2021) (164 problems) and report Pass@1, which measures the fraction of problems solved correctly on the first attempt. For mathematical reasoning, we use the test set of GSM8K (Cobbe et al., 2021) (1,319 problems) with chain-of-thought zero-shot prompting (Wei

et al., 2022) and report Accuracy. For instruction following, we use AlpacaEval (Dubois et al., 2024) (805 instructions) and report the Win Rate against GPT-3.5-Turbo as judged by GPT-4.1. In addition to these task-level metrics, we report L1 Reduction (ρL1 ) as an indicator of how effectively surgery corrects representation bias in ∗ hidden-state space. Let δtbefore = H1 ∥hM t − ht ∥1 and δtafter = H1 ∥ĥt − h∗t ∥1 denote the per-position L1 residuals before and after surgery. The L1 Reduction is: P δtafter ρL1 = 1 − P t∈T before t∈T δt

(7)

Higher ρL1 indicates that a larger fraction of the representation bias has been removed. As shown by Yang et al. (2024a), this metric correlates with downstream task performance in encoder models; we examine whether the same relationship holds for decoder models in Section 6. 4.3

Implementation Details

All merged models are produced using mergekit (Goddard et al., 2024) from three Llama-2-7B fine-tunes: Llama-2-7B-Instruct, WizardMath-7B-v1.0, and Llama-2-7B-Code. Surgery modules are trained for 500 iterations with the Adam optimizer (β1 =0.9, β2 =0.999, η = 10−3 ) on 50 domain-specific calibration samples (HumanEval prompts for code, GSM8K questions for math, AlpacaEval instructions for instruct), tokenized to a maximum length of 512. The calibration samples are completely excluded from the final evaluation sets to ensure that the main results represent true zero-shot performance evaluated solely on unseen test samples (see Table 7 for split details). Hidden states are precomputed from both frozen models in bfloat16 and cached on GPU; only the surgery parameters are updated. For evaluation, HumanEval and GSM8K are run through lm-eval-harness (Gao et al., 2024) with greedy decoding (temperature 0.0) and the HuggingFace backend; the maximum number of generated tokens is 512 for HumanEval and 512 for GSM8K (chain-of-thought, zero-shot). AlpacaEval responses are generated with sampling (temperature 0.7, top-p 0.9, max 1024 tokens) and judged pairwise against GPT-3.5-Turbo using GPT-4.1 as judge. All experiments are conducted on NVIDIA A100-SXM4-80GB GPUs.

4.4

Baselines

We compare four settings. Individual models: each fine-tuned model evaluated on all three benchmarks, establishing the upper bound for domainspecific performance. Merged (no surgery): the merged model evaluated directly. Standard + L1 (Yang et al., 2024a): position-agnostic surgery with standard L1 loss, representing the original approach. DARTS (ours): position correction module with entropy-weighted L1 loss.

Results

5.1

Main Results

Table 1 presents the main results, comparing the original surgery approach (Standard + L1 ) against DARTS across both merged model configurations and all three benchmarks.

Analysis

6.1

L1 Bias Reduction vs Task Performance

A central question is whether reducing representation bias in hidden-state space translates to downstream task performance. Table 3 reports the mean L1 bias before and after DARTS surgery alongside the corresponding task performance changes. 6.2

Statistically Significant Improvements

To verify that the improvements of DARTS over the standard surgery baseline are not due to random variation in calibration data selection, we train both methods across 6 different random seeds (controlling the calibration data shuffle order) and compare their average scores. Table 4 reports the perseed average performance (mean of HumanEval, GSM8K, and AlpacaEval) for each method. We first test each group for normality using the Shapiro-Wilk test (α = 0.05). The DARTS scores pass the normality test (W = 0.80, p = 0.059), while the Standard surgery scores do not (W = 0.77, p = 0.034). Since one group is nonnormal, we apply the Mann-Whitney U test rather than a parametric t-test. The result (U = 36.0, p = 0.002) indicates that DARTS significantly outperforms Standard surgery at the α = 0.05 level. Despite a modest absolute improvement of 2.0 percentage points in average score, the consistency of the gain across all 6 seeds (with DARTS outperforming Standard in every run) yields a highly significant p-value, confirming that the improvement is

Merging Method

Weight Averaging

Task Arithmetic

TIES-Merging

DARE

Surgery Method

HumanEval

GSM8K

AlpacaEval

Avg.

None (baseline) Yang et al. (2024a) DARTS (ours)

27.4 26.2 26.8

31.2 28.7 28.9

37.3 39.5 41.3

32.0 31.5 32.3

None (baseline) Yang et al. (2024a) DARTS (ours)

23.8 23.2 25.0

41.2 42.5 43.4

49.5 47.0 58.6

38.2 37.6 42.3

None (baseline) Yang et al. (2024a) DARTS (ours)

17.1 23.2 21.9

12.6 7.3 8.3

8.9 12.8 13.0

12.9 14.4 14.4

None (baseline) Yang et al. (2024a) DARTS (ours)

19.5 18.3 21.3

34.9 35.4 34.9

47.7 47.6 48.7

34.0 33.8 35.0

Table 1: Main results on three benchmarks. Comparing the original surgery approach (Standard + L1 ) against our decoder-specific adaptations. Best results per merged model in bold.

GSM

Llama-2-7B-Base Llama-2-7B-Instruct WizardMath-7B-v1.0 Llama-2-7B-Code

12.8 15.2 12.8 28.1

4.5 24.5 39.7 5.2

1.5 58.2 7.0 0.2

Table 2: Individual model baselines. HE = HumanEval (pass@1), GSM = GSM8K (exact match), AE = AlpacaEval (win rate %).

60 55 50

Score (%)

Model

45 40 35

HumanEval GSM8K AlpacaEval Average

30 25

robust to calibration data variation. A more granular statistical analysis is given in Appendix I

Ablation Studies

7.1

Loss Function Ablation

Table 5 compares four loss functions using Standard (position-agnostic) surgery on TA (λ = 0.5), holding architecture constant to isolate the effect of the loss. From the results, we find that loss function choice matters more than surgery module architecture for decoder-based surgery. 7.2

Surgery Module Ablation

Table 6 compares Standard surgery against DARTS (with position correction), both using EWL1 loss, isolating the complementary contribution of the position correction module. From the results, we find that adding the position correction module leads to modest but significant improvements. 7.3

Learning Rate Sensitivity

Figure 3 shows DARTS performance across learning rates spanning two orders of magnitude (10−4 to 5 × 10−3 ). Performance is remarkably stable: the average score varies by less than 2 percentage points across the entire range, and no single

1e-4

5e-4

1e-3

Learning Rate

2e-3

5e-3

Figure 3: Learning rate sensitivity of DARTS. Performance is stable across two orders of magnitude, with the default 10−3 achieving the best average.

learning rate causes catastrophic degradation on any benchmark. The default learning rate of 10−3 achieves the best AlpacaEval score and the highest average, but nearby values (5 × 10−4 , 5 × 10−3 ) perform competitively. This robustness suggests that DARTS does not require careful learning rate tuning in practice.

Conclusion

We have extended representation surgery from encoder-based vision models to decoder-based language models and identified two fundamental challenges specific to the autoregressive setting: position-dependent bias accumulation under the causal attention mask, and non-uniform position importance arising from the structure of language generation. To address these challenges, we proposed DARTS, which combines entropy-weighted L1 loss (focusing correction on decision-critical

Merged Model

Domain

ρL1

Task Score

(%)

No Surg.

DARTS

Change

TA (λ = 0.5)

Code Math Instruct

8.2 11.3 42.9

23.8 41.2 49.5

25.0 43.4 58.6

+1.2 +2.2 +9.1

TA (λ = 0.7)

Code Math Instruct

14.2 14.3 51.0

21.9 46.2 49.3

26.8 42.9 59.2

+4.9 −3.3 +9.9

Table 3: Correlation between L1 bias reduction and task performance. pp = percentage point change in the task metric.

Seed

DARTS (Avg.)

Standard (Avg.)

42.3 42.2 42.3 43.3 42.2 43.0

37.6 41.7 40.4 41.8 41.2 41.1

42.6 ± 0.5

40.6 ± 1.6

default 7 29 71 164 190 Mean ± Std

Mann-Whitney U test: U = 36.0, p = 0.002

Table 4: Statistical significance of DARTS vs. Standard surgery (L1 loss). Average score = mean of HumanEval, GSM8K, and AlpacaEval across 6 seeds. Loss Function

GSM

Avg.

L1 (original) KL on logits Cross-entropy distill. EWL1 (ours)

23.2 20.1 18.3 25.6

42.5 43.4 33.1 41.7

47.0 54.5 40.0 57.0

37.6 39.3 30.5 41.4

Table 5: Loss function ablation with Standard surgery. EWL1 dominates, confirming that loss choice is the primary driver of improvement.

positions) with a structurally regularized position correction module that captures the monotonically increasing bias pattern. Our experiments on Llama2-7B merged models across three diverse domains demonstrate that loss function design is the dominant factor: entropy-weighted L1 alone accounts for the majority of performance gains, with position correction providing complementary improvements. On Llama-2-7B merged models across code, math, and instruction-following domains, DARTS achieves significant improvement over the original surgery approach.

Limitations 1. Sequence length generalization. The position correction bias table b ∈ RTmax ×H cannot extrapolate beyond Tmax = 512. Positions exceeding Tmax receive zero position correction

Module

GSM

Standard + Position corr. (ours)

25.6 25.0

41.7 43.4

57.0 58.6

∆

−0.6

+1.7

+1.6

Table 6: Surgery module ablation with EWL1 loss. Position correction provides complementary gains on top of the improved loss.

(content correction still applies). Future work could explore continuous position functions or relative position encodings. 2. Scale. We evaluate exclusively on Llama-27B. Whether DARTS transfers to larger models (13B, 70B) or different architectures (Mistral (Jiang et al., 2023)) remains untested. 3. Domain asymmetry. The entropy weighting may deprioritize positions critical for code generation (e.g., syntactically deterministic but semantically critical tokens), potentially explaining domain-dependent gains.

References Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias

Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. Michael Crawshaw. 2020. Multi-Task Learning with Deep Neural Networks: A Survey. arXiv preprint arXiv:2009.09796. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The Language Model Evaluation Harness. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s Mergekit: A Toolkit for Merging Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477–485. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning (ICML).

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations (ICLR). Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inferencetime intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, JianGuang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct. In The Thirteenth International Conference on Learning Representations. Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems (NeurIPS). Mohammed Muqeeth, Haokun Liu, and Colin Raffel. 2023. Soft merging of experts with adaptive routing. arXiv preprint arXiv:2306.03745. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS). Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML).

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, and 7 others. 2024. Code Llama: Open Foundation Models for Code. Preprint, arXiv:2308.12950.

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR).

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. 2024. ZipIt! Merging Models from Different Tasks without Training. In International Conference on Learning Representations, volume 2024, pages 29215–29237.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. TIES-Merging: Resolving Interference When Merging Models. Advances in neural information processing systems, 36:7093–7115.

Anke Tang, Xianglin Luo, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. 2026. Concrete subspace learning based interference elimination for multi-task model fusion. Preprint, arXiv:2312.06173.

Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. 2024a. Representation Surgery for Multi-Task Model Merging. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 56332–56356. PMLR.

Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. 2024. Merging multi-task models via weight-ensembling mixture of experts. arXiv preprint arXiv:2402.00433. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35:24824– 24837. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965–23998. PMLR. Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. 2025. A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning. Transactions on Machine Learning Research. Survey Certification.

Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024b. AdaMerging: Adaptive Model Merging for MultiTask Learning. In The Twelfth International Conference on Learning Representations. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. In Forty-first International Conference on Machine Learning. Yu Zhang and Qiang Yang. 2022. A Survey on MultiTask Learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609.

Calibration vs. Test split

Benchmark HumanEval (Code) GSM8K (Math) AlpacaEval (Instruct)

Total Samples Calibration Split Evaluation Split 164 1,319 805

50 50 50

114 1,269 755

Table 7: Dataset partitions detailing the calibration (training) and evaluation (test) splits for each benchmark. The splits are mutually exclusive to prevent any potential leakage.

Calibration Cost

Although DARTS introduces negligible inference overhead (≈0.1% additional parameters), the offline calibration stage requires forward passes through both the merged and target models to precompute hidden states and prediction entropy weights. To make the practical resource requirements fully transparent, we measure the end-to-end calibration cost on a single NVIDIA A100-SXM480GB GPU using Llama-2-7B-Code as the target model, with the standard protocol: 50 unlabeled calibration prompts, Tmax = 512, batch size 8, 500 training iterations, rank-16 surgery bottleneck. Table 8 reports the per-configuration cost (one merged model × one domain × one surgery variant). The actual surgery computation (precompute

+ training) takes only ≈4–5 seconds; loading the two 7B models accounts for ≈7 seconds and dominates the wall-clock time. Peak GPU memory remains well within A100 capacity at ≈30–31.5 GB (under 40% utilization). The hidden-state cache occupies 0.42 GB, and the trained surgery module is 265 KB–4.46 MB on disk (<0.04% of backbone parameters). Since each domain is calibrated independently, the total cost for all three domains is ≈36 seconds, making the entire calibration pipeline lightweight enough to run on a single commodity GPU. Phase Load merged model Load target model Precompute (2× forward) Training loop (500 iter) Total wall-clock Peak GPU memory Hidden-state cache Surgery parameters Checkpoint size on disk

Yang et al. (2024a)

DARTS

3.37 s 3.43 s 1.81 s 2.40 s

3.46 s 3.37 s 1.87 s 3.15 s

11.01 s 11.85 s 29.91 GB 31.49 GB 0.42 GB 0.42 GB 131,072 2,228,224 265 KB 4.46 MB

Table 8: Per-domain calibration cost on a single NVIDIA A100 (80 GB). Model: Llama-2-7B-Code (≈7B params), 50 calibration prompts, Tmax = 512, rank 16, 500 iterations. The surgery computation (precompute + training) is ≈4–5 s; model loading (≈7 s) dominates.

Effect of Calibration Data Size

The surgery module is trained on a small set of N domain-specific calibration samples. Table 9 examines how performance varies as N ranges from 10 to the full test set. Performance is relatively stable across calibration sizes, with as few as 20 samples achieving results competitive with larger budgets. This suggests that the surgery module learns a generalizable bias correction rather than memorizing calibration-specific patterns. Using the full test set for calibration yields comparable results, confirming that the module does not overfit even with abundant data. Note that calibrating on the full test set is strictly an exploratory analysis intended to examine data scaling limits and establish a theoretical upper bound; these test-adapted results are kept isolated to this appendix and are not used for any of the main model claims or baseline comparisons in the paper.

Calibration N

GSM

10 20 30 50 100 Full Test

24.4 24.4 26.2 25.0 26.2 23.17

42.5 44.8 43.8 43.4 42.8 43.52

54.0 58.4 56.8 58.6 55.4 59.45

Table 9: Effect of calibration data size.

Full Loss Function Comparison

In the main paper (Table 5), we compare loss functions using Standard surgery only. Here, Table 10 presents the full factorial comparison of loss functions crossed with surgery module type (Standard vs. position correction). This allows us to disentangle the contribution of the loss function from the contribution of the position correction module. Two patterns emerge. First, switching from L1 to entropy-weighted L1 consistently improves average performance regardless of the surgery module, confirming that the loss function is the primary driver. Second, adding position correction on top of each loss function provides a further boost in average score, with the largest gain observed for EWL1 (+0.9 average), indicating that the two innovations are complementary.

Position Correction Module Variants

Beyond the position correction module used in DARTS, we explored several alternative architectures for incorporating position information into the surgery module. Table 11 compares these variants, all trained with EWL1 loss at rank 16. Standard applies only the content bottleneck with no position awareness. Add adds a learned position embedding to the hidden state before the downprojection, allowing position to modulate the ReLU activation pattern. Parallel uses two independent bottleneck paths (one for content, one for position) whose outputs are summed; however, we observe that the position path causes severe degradation at inference, likely due to overfitting. Pos. Corr. (Ours) adds a free per-position bias vector that is independent across positions and zero-initialized, providing structural regularization that prevents overfitting while still capturing systematic positiondependent offsets. Among all variants, our position correction module achieves the highest average score while maintaining stable HumanEval performance, unlike the Parallel variant which collapses on code generation.

Loss Function

Surgery Module

L1 L1 Entropy-Weighted L1 Entropy-Weighted L1

Standard + Pos. corr. Standard + Pos. corr.

HumanEval

GSM8K

AlpacaEval

Avg.

23.2 21.3 25.6 25.0

42.5 43.1 41.7 43.4

47.0 48.8 57.0 58.6

37.6 37.7 41.4 42.3

Table 10: Full loss function × surgery module grid.

Variant

GSM

Extra Params

Rank r

GSM

Params/domain

Standard Add Parallel Pos. Corr. (Ours)

25.6 21.9 12.2 25.0

41.7 44.2 45.0 43.4

57.0 57.9 55.5 58.6

0 2.1M 2.2M 2.1M

16 32 64 128

25.0 25.0 25.0 23.8

43.4 44.3 42.8 42.6

58.6 56.4 56.0 58.1

131K + 2.1M 262K + 2.1M 524K + 2.1M 1.04M + 2.1M

Table 11: Position correction module variant comparison.

Effect of Training Steps

Table 12 varies the number of training iterations from 50 to 1000. Performance is stable across this range, with even 50 iterations producing competitive results. Longer training (1000 steps) yields a modest GSM8K improvement without degrading other benchmarks. Steps

GSM

50 100 200 400 500 (default) 1000

25.6 25.0 25.0 24.4 25.0 25.6

42.3 42.9 43.4 43.0 43.4 44.3

57.9 56.5 57.9 57.0 58.6 58.4

Table 12: Effect of training iterations.

Effect of Surgery Rank

Table 13 varies the bottleneck rank r of the surgery module. Rank 16 (the default) achieves the best overall balance. Increasing rank to 32 or 64 does not improve performance and slightly hurts AlpacaEval, while rank 128 degrades HumanEval, suggesting that higher-rank modules overfit on the small calibration set.

Table 13: Effect of surgery rank on DARTS.

Seed Sensitivity

Table 14 reports DARTS performance across 6 random seeds that control the calibration data shuffle order. The low standard deviation (HE: ±0.2, GSM: ±0.8, AE: ±1.0) confirms that DARTS is robust to the choice of calibration samples.

Seed

GSM

default 7 29 71 164 190

25.0 24.4 25.0 25.0 25.0 25.0

43.4 45.0 43.6 45.0 44.4 45.0

58.6 57.3 58.3 59.9 57.1 59.0

Mean ± Std

24.9 ± 0.2

44.4 ± 0.8

58.4 ± 1.0

Table 14: Reproducibility across random seeds. Seed controls the calibration data shuffle order.

Granular Statistical Analysis of Seed Variation

To provide a more granular statistical analysis, we compare the task-level performance of DARTS against the Standard surgery baseline across the 6 varied calibration seeds (including the default seed) using both parametric and non-parametric tests. We perform paired t-tests and Wilcoxon signedrank tests on the raw scores for each task. The results, summarized in Table 15, show that DARTS provides statistically significant improvements on GSM8K and AlpacaEval using the non-parametric Wilcoxon test, and significant to marginally significant improvements using the parametric t-test. Performance on HumanEval remains statistically indistinguishable from the baseline. Furthermore, we fit a linear mixed-effects model (Score ∼ Model + Task) with Seed as a random intercept across all tasks. The coefficient for the DARTS model is positive and highly significant (β = 0.019, z = 3.224, p = 0.001), confirming that DARTS provides a consistent and statistically significant improvement over the baseline across tasks and seeds.

Task HumanEval GSM8K AlpacaEval

Mean Diff t-stat t-test p-val W -stat Wilcoxon p-val +0.30 0.690 +1.56 2.849 +3.95 2.459

0.521 0.036 0.057

7.500 0.000 0.000

0.688 0.031 0.031

Table 15: Paired t-test and Wilcoxon signed-rank test comparing DARTS vs. Standard surgery across 6 random seeds for each task. Positive Mean Diff indicates DARTS outperforms the baseline.

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

Analyze & explore related research in Conceptio

Related documents