arXiv CSopen access

Locking Pretrained Weights via Deep Low-Rank Residual Distillation

Locking Pretrained Weights via Deep Low-Rank Residual Distillation Keitaro Sakamoto∗ , Pierre Ablin, Federico Danieli, Marco Cuturi

arXiv:2605.10777v1 [cs.LG] 11 May 2026

Apple The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardware and software platforms. They also allow for more open research and testing, to the extent that users can use them as checkpoints, fine-tune them according to their needs, and potentially redistribute them. In some cases, however, concerns on modifying these weights towards unauthorized uses may outweigh the pros of giving users such a freedom. Defending against such adaptation is non-trivial: since an adaptive attacker can observe all weights and architectures by definition, they can reverse simple structural defenses, and use optimization to defeat the simplest locking mechanisms. In this work, we exploit the inference–training asymmetry of automatic differentiation as a novel defense axis. We propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network (DLR-Net) of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation. DLR-Nets are efficiently trained via module-wise distillation. We show that, beyond this memory overhead, DLR-Lock results in architectural mismatches that complicate the optimization landscape of standard fine-tuning, and a backward pass that incurs disproportionately more overhead than the forward pass. Our defense succeeds in withstanding adaptive attackers with full knowledge of the defense strategy while preserving the original model’s capabilities. Experiments on LLM validate these claims. Correspondence: KS: [email protected]; PA: [email protected]; FD: [email protected]; MC: m_cuturi@apple. com Contributions: Work done while KS was an intern at Apple. Date: May 12, 2026

1

Introduction

Pretrained large language models represent a substantial investment of compute, data curation, and engineering effort. When released as open weights, the model provider loses control over how these weights are used: any party can adapt the model to new domains or remove usage restrictions, often at a fraction of the original training cost (Kapoor et al., 2024; Bommasani et al., 2024; Casper et al., 2026). Open-weight licenses provide legal protection against such misuse, but no technical enforcement. These concerns motivate a fundamental scientific question: can a neural network be locked, so that adaptation is inherently more expensive than adapting the original model, while retaining its inference time performance? This pressing question is far from straightforward: simple structural defenses that exploit weight symmetries can be trivially reverted by an adaptive attacker who inspects the model. One might hope to use optimization to embed the defense more deeply into the loss landscape, but this strategy is inherently self-defeating: the defender must use optimization to find a solution that resists optimization, yet the same gradient information available to the defender is equally available to the attacker. Recent model locking methods (Rosati et al., 2025; Wang et al., 2026b; Rosati et al., 2026) face this fundamental difficulty. In this work, we exploit a different defense axis: the inference–training asymmetry of automatic differentiation (Griewank and Walther, 2008). Inference requires only a forward pass through the model, discarding all intermediate activations; training, by contrast, must store those intermediates for backpropagation. To exploit this weakness, we propose to replace each pretrained MLP in a large network with a deep low-rank residual

1

k

-Loc

DLR

Attention Module

RMS Norm

MLP

Module-wise Distillation

DLR-Net RMS Norm

×N

×L σ

💸 Cheap & reliable adaptation 💸 Fine-tuning loss

Attention Module

×N

FFN Module

Fine-tuning budget

🔒 Expensive & brittle adaptation 🔒 Fine-tuning loss

el

Mod

e Bas

Fine-tuning budget

Figure 1 Overview of DLR-Lock. Each MLP in the original model (top) is replaced with a functionally equivalent

deep low-rank residual network (DLR-Net, bottom) via module-wise distillation. The replacement introduces three locking mechanisms: (1) activation memory growth, (2) optimization instability from architectural mismatch, and (3) disproportionate backward-pass overhead.

network (DLR-Net) that matches the original input–output mapping via distillation (vector regression) with comparable parameter count. The replacement introduces three locking mechanisms: (1) activation memory that grows linearly with depth; (2) architectural mismatch between the deep residual networks and the original MLPs destabilizes standard fine-tuning recipes; and (3) the backward pass is disproportionately slowed compared to the forward pass due to heavier memory access and more complex computation. The locked model thus preserves inference quality while imposing substantial additional cost on any party attempting to adapt it. Our locking mechanism is fine-tuning-domain agnostic: it prevents fine-tuning the model on any target domain, not only on potentially harmful target domains (Huang et al., 2024; Tamirisa et al., 2025). On the one hand, users cannot fine-tune the model for genuinely inoffensive applications. On the other hand, we are guaranteed to be locked against any malicious fine-tuning dataset. Figure 1 illustrates the overall idea; related work is discussed in Appendix A. Our contributions are: • We formalize the problem of model locking as adaptation cost amplification and establish an adaptive threat model grounded in Kerckhoffs’s principle, which requires that security rely on structural properties rather than secrecy of the defense mechanism. • We categorize existing locking approaches as symmetry-based, barrier-based, and penalty-based, and identify inherent limitations of each through analysis and controlled experiments. • We show that the inference–training asymmetry of automatic differentiation provides a novel, optimizeragnostic defense axis, and propose DLR-Lock, a deep low-rank residual locking with a two-phase distillation pipeline locking modules without quality loss. • We validate the approach on the Qwen3-0.6B model (Yang et al., 2025), demonstrating that the locked models preserve the original capabilities while imposing additional cost on fine-tuning: beyond the increased activation memory expected by design, architectural mismatch with the original MLPs destabilizes standard fine-tuning recipes, and the backward pass is disproportionately slowed compared to the forward pass.

2

Model Locking: Formulation and Challenges

Modern large language models are built from a stack of N transformer blocks (Vaswani et al., 2017). Each block consists of a multi-head self-attention module Attn(·) followed by a feed-forward network (FFN) module, with a residual connection and RMSNorm (Zhang and Sennrich, 2019) applied before each module, following the pre-norm convention:

Z = X + Attn RMSNorm(X) , X ′ = Z + FFN RMSNorm(Z) . (2.1) 2

Here X ∈ Rn×d is a sequence of n token representations of dimension d, Z and X ′ are intermediate and output sequences of the same shape, and both FFN and RMSNorm act row-wise, i.e., independently on each token. In recent architectures such as LLaMA (Grattafiori et al., 2024) and Qwen (Yang et al., 2025), the FFN is a SwiGLU MLP (Shazeer, 2020):
FFN(x) = Wdown (Wup x) ⊙ σ(Wgate x) , where ⊙ denotes element-wise multiplication, σ(z) = z · sigmoid(z) is the SiLU activation (Elfwing et al., 2018), Wgate , Wup ∈ Rdff ×d , and Wdown ∈ Rd×dff . Our method replaces each FFN residual sub-block with a functionally equivalent architecture, leaving all other components unchanged.

2.1

Problem formulation

The term model locking was introduced by Rosati et al. (2025) to describe making pretrained weights resistant to fine-tuning; a related concept of non-fine-tunability was independently proposed by Wang et al. (2026b). We formalize this setting below. Definition 2.1 (Locking Mechanism). Let f (·; θ0 ) be a model obtained by standard pretraining. A locking mechanism M with private information s produces a locked model g(·; θ′ ) = M(f, θ0 , s) that is expected to be harder to fine-tune than f (·; θ0 ). Assumption 2.2 (Attacker Capabilities). The attacker has: (i) full access to the published weights θ′ , but not to s; (ii) knowledge of the locking algorithm M; (iii) a compute budget Cattack ≪ Cpretrain . Condition (ii) follows Kerckhoffs’s principle (Kerckhoffs, 1883): the defense must remain effective even when the attacker knows exactly how it was constructed. The defender’s advantage comes from structural properties of g, not secrecy of M. Condition (iii) is motivated by the observation that the attacker can always query g on arbitrary inputs and train a fresh model from the resulting logits, even in the black-box setting. Since this logit distillation cost is of the order Cpretrain (Busbridge et al., 2025), no defense can withstand an attacker whose budget exceeds this threshold. The attacker can employ various strategies to adapt the locked model g to a new, potentially harmful, task T under its budget constraint: fine-tuning the entire model, distilling individual modules, applying analytical transforms to preserve functional equivalence, or any combination thereof. Let A denote the set of all feasible attack strategies available to the attacker. Each strategy A ∈ A incurs a cost C(A) and yields a model A(g) with performance perf T (A(g)) on task T . The adaptation cost for reaching a target performance level p is the minimum cost over all strategies that achieve it:
CT (g → p) = min C(A) s.t. perf T A(g) ≥ p. (2.2) A∈A

We can now state the two requirements for model locking. Definition 2.3 (Model Locking). A defense mechanism M achieves model locking with locking factor κ > 1 and tolerance ε > 0 if the published model g = M(f, θ0 , s) satisfies: (i) Utility preservation. L(g) < L(f ) + ε for every downstream task loss L. (ii) Adaptation cost amplification. For any downstream task T and target performance level p, CT (g → p) ≥ κ · CT (f → p), where CT (· → p) is measured in wall-clock time, FLOPs, or GPU-hours. Condition (i) requires that the locked model retain general capabilities, not merely specific benchmark scores. Definition 2.3 places no constraint on the locking cost Clock , since the defender can afford a one-time cost that is amortized over all deployments.

3

Figure 2 Locking solutions exist but gradient descent cannot reach them. We optimize min∆θ MSE − λ Ω on a two-layer ReLU MLP trained on MNIST, sweeping over 29 values of λ and three learning rates with SGD and gradient norm clipping. Each curve is one trajectory; color encodes log10 λ. Red stars: weight-symmetry solutions (a−1 W1 , aW2 ). Left: Ω = ∥∆θ∥2 ; y-axis is ∥∆θ∥/∥θ0 ∥. Right: Ω = Tr(HCE ); y-axis is relative Hessian trace exp Ex [log Tr(Hi (θ))/Tr(Hi (θ0 ))] (Hi : per-sample CE Hessian). x-axis (both): relative MSE, exp Ex [log ∥f (θ, x)−f (θ0 , x)∥2 /∥f (θ0 , x)∥2 ].

2.2

Why model locking is fundamentally difficult

Before introducing our method, let us consider what approaches to model locking are available and why they face fundamental difficulties. We provide a detailed comparison with existing methods in Appendix A, in light of the discussion below. A natural starting point for model locking is to exploit weight symmetries: transformations of the parameters that preserve the network’s input–output mapping while potentially disrupting optimization (Godfrey et al., 2022; Zhao et al., 2026). The ReLU activation is positively homogeneous, satisfying ReLU(az) = a ReLU(z) for any a > 0. For any network layer of the form W2 ReLU(W1 x), this yields the identity (aW2 ) ReLU(a−1 W1 x). The reparameterization preserves the function exactly, but creates an imbalance: the output becomes highly sensitive to small perturbations of a−1 W1 , since these are amplified by aW2 . This slows the optimization, but the defense is trivially broken: the attacker can inspect per-layer weight norms, detect the imbalance, and rebalance them. Example 1: activation homogeneity.

Between two consecutive linear layers, one can insert an invertible matrix A and its inverse: (W2 A−1 )(AW1 ). In a transformer, this applies to consecutive linear maps such as the key–query or value–output projections, where an adversarial choice of A can significantly disturb the individual factors. A concrete example of matrix factorization is given in Appendix C. However, even in this case, the adaptive attacker can compute the singular value decomposition (SVD) of the product (W2 A−1 )(AW1 ) = W2 W1 = U ΣV ⊤ and reparameterize as (U Σ1/2 )(Σ1/2 V ⊤ ), a canonical factorization entirely independent of the defender’s choice of A. Example 2: invertible matrix insertion.

These examples illustrate a general difficulty: known symmetries are limited to simple patterns (scaling, permutation, invertible linear transforms) that an informed attacker can detect and reverse, which is negligible additional cost in terms of equation 2.2. This motivates optimization-based approaches, where the defense is embedded through training and the configuration serves as secret information—even with full knowledge, exact unlearning requires solving a hard optimization problem. We discuss two possible families below. From symmetry-based to optimization-based approaches.

(i) Barrier-based locking: disturbing the input Jacobian. The idea is to insert a barrier module B between layers whose input Jacobian ∂B(h)/∂h causes gradients to explode or vanish, making fine-tuning difficult. Two constraints arise: the barrier must be inseparable from the model, and its form must be learned rather than hand-crafted, since an adaptive attacker can otherwise remove or reverse it. Replacing entire modules is a natural approach—replacing individual activations would be hand-crafted and removable—but standard dis-

4

tillation produces smooth approximations, so achieving pathological Jacobians without affecting downstream layers remains difficult. Limitation: well-hidden barriers that sufficiently block gradient flow also make it more difficult for the defender to maintain model quality, as required by condition (i) of Definition 2.3. An alternative approach is to directly search for illconditioned solutions via optimization, rather than relying on known symmetries. One can formulate this as a constrained optimization problem: (ii) Penalty-based locking: optimizing for ill-conditioning.

max Ω(θ) θ

s.t. L(θ) < L(θ0 ) + ε,

(2.3)

where Ω measures difficulty of fine-tuning. However, this formulation is self-defeating: Limitation: the defender seeks to make optimization hard for the attacker, yet must itself rely on optimization to find such a solution, caught in a self-made trap. To confirm this, we test a simplified instance of equation 2.3 on a two-layer ReLU MLP trained on MNIST (LeCun et al., 2010). We optimize min∆θ MSE(fθ0 , fθ0 +∆θ ) − λ Ω(θ0 + ∆θ), where the MSE term preserves the full input–output mapping to maintain the original performance, sweeping over λ and learning rates with two proxy penalties: Ω = ∥∆θ∥22 (distance from initialization) and Ω = Tr(∇2θ LCE ) (sharpness via Hutchinson trace (Hutchinson, 1990)). Neither directly measures fine-tuning difficulty; if gradient descent cannot achieve even these simpler objectives under the constraint, the full problem equation 2.3 is unlikely to be easier. As shown in Figure 2, no trajectory reaches the high-Ω corner while maintaining the original function. Yet such solutions do exist: the weight symmetry in Example 1 preserves the function exactly while increasing both quantities. Gradient descent simply cannot reach them from the pretrained initialization (see also Appendix D.1).

3

DLR-Lock: Deep Low-Rank Residual Locking

The approaches discussed in Section 2.2 all attempt to modify the loss landscape or gradient flow. We take a different route: we replace each SwiGLU MLP with a DLR-Net that computes the same function but is structurally expensive to differentiate. The key observation is that inference and training have fundamentally different memory requirements: users who run the model are unaffected, while those who update its weights face additional cost. • Inference evaluates the model in a single forward pass, discarding intermediates. • Training must store all intermediate activations for backpropagation. This is not a property of the loss landscape or the optimizer but a structural property of automatic differentiation. An architecture that is cheap to evaluate but expensive to differentiate creates an inherent, optimizer-agnostic defense—one that does not depend on finding particular weights. While the defender replaces each MLP independently without touching other components, the attacker optimizes all parameters jointly through N × L layers. Beyond the memory cost, two additional optimization challenges may arise: the effective depth may cause training instability, and standard fine-tuning recipes developed for the original architecture are unlikely to transfer, since the deep residual MLPs differ substantially from the MLP blocks they replace.

3.1

Deep low-rank residual network

A practical defense must not disadvantage legitimate users: the replacement must (i) match the original module’s input–output behavior and (ii) not significantly increase the total parameter count or inference cost. We use low-rank weight matrices in each residual block: the bottleneck rank r ≪ d keeps the per-layer parameter count small, allowing the total budget to match (or only slightly exceed) that of the original SwiGLU MLP while distributing the parameters across many layers.

5

We replace the MLP residual block x′ = z + FFN(RMSNorm(z)) with hidden dimension d and intermediate dimension dff by a depth-L low-rank residual network:
h0 = z, hi+1 = hi + αi Ui σ Vi RMSNorm(hi ) , x′ = hL , (3.1) where σ is a pointwise nonlinearity (we use SiLU for consistency with the SwiGLU activation), Vi ∈ Rr×d projects to the bottleneck, Ui ∈ Rd×r projects back, and αi is a learnable ReZero (Bachlechner et al., 2021) scalar. Each layer costs 2dr parameters (plus normalization parameters), so at a fixed budget P the depth is L = ⌊P/(2dr + cnorm )⌋. Smaller rank yields a deeper network at the same parameter cost. RMSNorm pre-normalization is essential for stable training at L > 100. These per-layer costs accumulate across the full network. We quantify the cost amplification for the simplest attack strategy; stronger attacks are analyzed in Section 3.3: Proposition 3.1 (Cost amplification under naive fine-tuning). Consider an attacker who uses naive finetuning A = {Aft }, i.e., gradient descent with full backpropagation through all layers, and assume the number of tokens required to reach target performance p on task T is the same for f and g. Then the GPU-hour cost of fine-tuning the locked model satisfies κ ≥ Θ(d/r) when MLP activations dominate the memory budget. The bound is conservative: it assumes equal sample efficiency between f and g; in practice, the architectural mismatch introduced by the deep residual MLPs may further slow convergence, increasing the true κ. The detailed derivation, including per-layer memory analysis, is given in Appendix B.

3.2

Distillation procedure

We distill the pretrained MLP into the DLR-Net in two phases: a module-wise phase that approximates each module’s input–output mapping independently, followed by a logits distillation phase that corrects the accumulated error by matching the full model’s output distribution. In both phases, the objective is to preserve the original function of the pretrained model. By construction, this satisfies the utility preservation condition (i) of Definition 2.3 for every downstream task, unlike methods that optimize a task-specific loss and only guarantee preservation on that particular objective. For each transformer layer l, we collect hidden states zl (after the attention sub-layer) from the pretrained model. The DLR-Net g (l) is trained to replicate the full residual sub-block fl (z) = z +FFNl (RMSNorm(z)) by minimizing the relative MSE, normalized by ∥fl ∥2 because output norms vary across layers. Phase 1: Module-wise distillation.

After all MLP layers are replaced, the small per-layer distillation errors compound across the full forward pass, degrading language modeling performance. We correct this by fine-tuning the DLR-Net parameters (MLP only; attention and embeddings are frozen) using the pretrained model as teacher. The alignment loss is the top-k KL divergence between teacher and student output distributions, restricting the sum to the k tokens with highest teacher probability. Phase 2: Logits distillation.

The full procedure is summarized in Algorithm 1. Phase 2 requires back-propagating through the full locked model, facing the same activation-memory barrier as the attacker; however, Phase 1 provides a good initialization so convergence is fast, and the defender pays this cost only once.

3.3

Attack strategies

We consider strategies an attacker may use to fine-tune the locked model. Gradient checkpointing (Chen et al., 2016) is the most direct countermeasure to activation memory overhead. Rather than naively checkpointing at fixed intervals, an adaptive attacker can selectively recompute only the DLR-Net blocks during the backward pass. This reduces peak memory from O(N Ld) to O(Ld), at the cost of recomputing each √ MLP forward pass. The optimal tradeoff between recomputation and saved activations is achieved at O( N L d) via evenly spaced checkpoints. If the attacker Gradient checkpointing.

6

Algorithm 1 DLR-Lock: model locking via deep low-rank residual distillation Require: Pretrained model M with N transformer layers, each containing a normalization–MLP residual sub-block; bottleneck rank r; per-MLP parameter budget P ; training data D Ensure: Locked model M̂ with MLP sub-blocks replaced by DLR-Nets 1: Phase 1: Module-wise distillation 2: for each layer l = 1, . . . , N in parallel do 3: Initialize DLR-Net g (l) with rank P r and depth L = ⌊P/(2dr + cnorm )⌋ 4: Train g (l) to minimize LMSE = n1 i ∥g (l) (zi ) − fl (zi )∥2 /∥fl (zi )∥2 5: Replace normalization, MLP, and skip connection with g (l) in M̂ 6: end for 7: Phase 2: Logits distillation 8: Freeze all parameters in M̂ except DLR-Net weights P teacher ) using M as teacher /pstudent log(pteacher 9: Train M̂ to minimize LKD = j j j∈top-k pj cannot afford even this memory, they must resort to hierarchical checkpointing, additional GPUs, or compromises such as shorter sequence lengths or reduced precision. This recomputation cost is unavoidable for every attacker. The attacker may freeze the DLR-Net parameters and train only the remaining model components. However, even with frozen weights, the RMSNorm backward pass at every layer must store d-dimensional intermediate vectors to compute its input Jacobian. If the attacker further applies stop_grad to avoid backpropagation entirely, gradients to earlier layers omit the main branch’s contribution, producing a biased signal that degrades training. Partial weight updates.

LoRA (Hu et al., 2022) reduces memory by freezing the base weights and training low-rank adapters. However, LoRA does not shorten the backward pass: backpropagation still traverses all L layers to compute gradients for the adapters. Moreover, it is non-trivial where to apply LoRA in the DLR-Net: the weight matrices Ui ∈ Rd×r and Vi ∈ Rr×d are already low-rank by construction, so adding a further decomposition adds little expressiveness. Low-rank adaptation (LoRA).

Since we distill SwiGLU → DLR-Net, the attacker can perform the exact same procedure in reverse. However, the attacker must reverse all locked layers to fully remove the defense, and the cost scales with the number of layers. The defender pays this cost once to protect the model; the attacker must pay it again just to enable fine-tuning. Module-wise distillation.

4

Experiments

4.1

Experimental setup

We use the Qwen3 (Yang et al., 2025) as our testbed. For distillation, we use the Nemotron-CC dataset (Su √ et al., 2025), a curated web text corpus. Weights are initialized as Ui ∼ N (0, 1/(d L)) and Vi ∼ N (0, 1/r), and αi = 0. For training, both phases use AdamW (Loshchilov and Hutter, 2019) with weight decay 10−5 , cosine learning-rate schedule with linear warmup (5% of total steps), and gradient clipping at norm 1. For Phase 1, each layer is trained independently for 500k steps with learning rate 3×10−3 , batch size 2, and sequence length 2048. The rank is chosen so that the depth falls in the range L ∈ [140, 150] (e.g., r = 32 for 0.6B). For Phase 2, we train all DLR-Net parameters jointly for 100k steps with top-k=1000, learning rate 10−3 , batch size 4 and sequence length 2048.

4.2

Quality preservation

Since our algorithm replaces every MLP module, we first verify that the locked model retains the original model’s capabilities. We evaluate locked models on the Nemotron-CC validation set for perplexity on the

7

Perplexity ↓

Accuracy (%) ↑

Nemo

WT

MMLU

ARC-E

ARC-C

HSw

WiGr

BoolQ

PIQA

Pretrained

14.9

21.5

40.3

55.9

33.7

47.3

56.4

63.8

67.3

Module-wise dist. +Logits dist.

21.8 14.9

182 23.4

31.1 36.8

50.6 50.7

26.5 29.0

40.5 43.5

54.9 55.2

62.6 63.8

63.4 65.3

Table 1 Quality preservation of Qwen3-0.6B after model locking. “Module-wise dist.” replaces every SwiGLU MLP with a DLR-Net in Phase 1. “+Logits dist.” applies logits distillation in Phase 2 with k=1000 and learning rate 10−3 . Nemo = Nemotron, WT = WikiText-103, ARC-E/C = ARC Easy/Challenge, HSw = HellaSwag, WiGr = WinoGrande.

Qwen3-8B, fixed n=32k tokens, prefill Depth L Seq. len

143

71

35

17

1024 8192 32768

3.37× 2.07× 1.39×

2.03× 1.44× 1.18×

1.45× 1.18× 1.09×

1.23× 1.08× 1.04×

Figure 3 Left: MLP-level overhead ratio (DLR-Net / SwiGLU) vs. batch size at the deepest configuration with

reduce-overhead compilation. For 0.6B (r=32), arithmetic intensity remains below the roofline threshold, so the overhead stays memory-bound. Right: Full-model prefill overhead on Qwen3-8B. Each column corresponds to a different depth L (rank r = 128, 256, 512, 1024).

distillation data, WikiText-103 (Merity et al., 2016) for perplexity on unseen text, and seven benchmarks from the LM Evaluation Harness (Gao et al., 2023) for downstream task performance: MMLU, ARC (Easy and Challenge), HellaSwag, WinoGrande, BoolQ, and PIQA. Table 1 shows the results. Module-wise distillation alone leaves a noticeable gap in several metrics. Logits distillation recovers Nemotron perplexity almost completely, as expected for in-distribution data, and WikiText perplexity is also largely recovered since both are natural text corpora. On the remaining benchmarks, logits distillation closes much of the gap, bringing most metrics within a few points of the pretrained model. To further close this gap, one can distill on a more diverse corpus beyond Nemotron, analogous to standard pretraining practice, given the strong perplexity recovery on the natural language datasets.

4.3

Inference overhead

Replacing each MLP with a depth-L DLR-Net preserves total theoretical inference FLOPs (Section 3.1), but the sequential chain of small matrix multiplications incurs wall-clock overhead from kernel launch costs. Since forward-pass benchmarking requires only the architecture, we evaluate at three scales (0.6B, 8B, 32B) in bfloat16 on H100 GPUs using PyTorch 2.9 (Ansel et al., 2024), comparing eager execution and torch.compile with CUDA graph replay (reduce-overhead). Because our architecture issues many small kernel launches per MLP, it benefits disproportionately from CUDA graph replay. For instance, an isolated DLR-Net at the 8B scale sees its overhead ratio drop from 144× (eager) to 11× with reduce-overhead compilation (see Figure 7 in Appendix for full comparison). Compilation eliminates kernel-launch overhead.

8

Figure 4 Fine-tuning convergence comparison between the pretrained baseline and the locked model on WikiText-103. Top: loss vs. number of training tokens (same batch size and sequence length for both); Bottom: loss vs. wall-clock time. Columns correspond to fine-tuning learning rates. Shaded regions indicate ±1 standard deviation across 3 seeds. In wall-clock terms, the backward pass accounts for 83% of total training time for the baseline and 89% for the locked model, confirming that training-specific computation dominates the overhead.

As the batch size grows, both architectures become compute-bound and the per-kernel launch cost is amortized. With reduce-overhead compilation, the 32B-scale ratio drops from 7.2× at batch size 1 to 2.9× at batch size 218 , converging toward the theoretical FLOP ratio. Larger models benefit more because each residual block performs proportionally larger matrix multiplications (Figure 3, left). Batch-size scaling saturates the overhead.

Defense–speed trade-off. The defender controls the operating point by choosing the rank r and depth L (recall 2drL ≈ 3d · dff at fixed parameter count). Figure 3 (right) shows the full-model prefill overhead on Qwen3-8B: at longer sequences where attention dominates, even the strongest configuration (r=128, L=143) reaches 1.4×, while a moderate depth (L=35) is nearly transparent at 1.09×. Detailed latency tables are provided in Appendix D.2.

4.4

Resistance to fine-tuning

We now evaluate the core claim: whether the locked model resists fine-tuning by an attacker. We use the same locked model as in Section 4.2, with gradient checkpointing enabled to reduce memory overhead. In principle, the defender could further strengthen the defense by evaluating multiple distillation configurations on public datasets and selecting the most resistant one. The attacker sweeps the fine-tuning learning rate over {3×10−6 , 10−5 , 3×10−5 , 10−4 }, each compared against fine-tuning the baseline unlocked model at the same rate. Figure 4 shows convergence curves at three fine-tuning learning rates; we omit 3×10−6 as it is too small and shows the same trend as 10−5 . The locked model starts at a similar loss to the baseline, confirming that the final loss difference cannot be explained by the initial loss difference alone. At lower rates (10−5 , 3×10−5 ), the baseline steadily improves while the locked model converges more slowly and plateaus at a higher loss, leaving a persistent gap even at the attacker’s best rate. At the largest rate (10−4 ), the locked model diverges entirely while the baseline trains normally, illustrating that the optimization itself becomes more unstable by our defense and narrowing the attacker’s effective learning rate range. The bottom row Results.

9

shows that in wall-clock terms, this gap is widely amplified, including by the recomputation cost of gradient checkpointing. One may ask whether the wall-clock delay stems from forward-pass overhead rather than genuine training difficulty. As shown in Figure 4, the backward pass accounts for 83% of total training time for the baseline and 89% for the locked model; the ratio not only dominates but further increases for the locked model, confirming that training-related computation grows disproportionately. Expressed relative to inference cost, the backward and optimizer overhead amounts to 4.9× the forward time for the baseline versus 8.1× for the locked model, where the increase is driven by recomputation necessitated by the higher memory footprint. Crucially, the same compilaTable 2 Single DLR-Net compilation ratio tion gains from Section 4.3 do not transfer to training. Let (same setting as §4.3). t′b /tb > 1 means compilation hurts training. tf denote inference forward time, te one training step, and tb = te − tf the training overhead; primed quantities denote the compiled setting. Table 2 isolates a single DLR-Net and 0.6B 8B measures how reduce-overhead compilation affects tf and tb . n t′f /tf t′b /tb t′f /tf t′b /tb Compilation significantly speeds up inference, but the backward 4 0.07 1.24 0.09 1.28 pass, which involves gradient computation and memory access 256 0.07 1.23 0.11 1.92 patterns that resist graph capture, shows that training overhead 16384 0.33 9.09 0.34 1.47 becomes worse than eager execution. This means the defender can offer normal users a fully optimized model via compilation or custom kernels, since the sequential forward pass is amenable to such optimizations. The attacker, who must compute gradients, cannot benefit from these same optimizations: the backward pass cannot simply propagate a single hidden state like the forward. Our deep architecture amplifies this asymmetry. Backward pass resists optimization.

5

Conclusion

We introduced a model locking approach that replaces MLP layers with deep low-rank residual networks, exploiting the inference–training asymmetry of automatic differentiation. The locked model imposes three complementary barriers: activation memory that grows with depth, architectural mismatch destabilizing fine-tuning, and a backward pass disproportionately costlier than the forward pass. A two-phase distillation procedure preserves the original model’s capabilities while achieving a measurable locking effect against adaptive attackers. The natural next step is reducing the forward-pass overhead through custom fused kernels or batched computation across residual layers.

References Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’24, page 929–947, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703850. doi: 10.1145/3620665.3640366. URL https://doi.org/10.1145/3620665.3640366. Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in artificial intelligence, pages 1352–1361. PMLR, 2021. Rishi Bommasani, Sayash Kapoor, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Daniel Zhang, Marietje Schaake, Daniel E Ho, Arvind Narayanan, and Percy Liang. Considerations for governing open foundation models. Science, 386(6718):151–153, 2024. Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.

10

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08606, 2025. Stephen Casper, Kyle O’Brien, Shayne Longpre, Elizabeth Seger, Kevin Klyman, Rishi Bommasani, Aniruddha Nrusimha, Ilia Shumailov, Sören Mindermann, Steven Basart, Frank Rudzicz, Kellin Pelrine, Avijit Ghosh, Andrew Strait, Robert Kirk, Dan Hendrycks, Peter Henderson, J Zico Kolter, Geoffrey Irving, Yarin Gal, Yoshua Bengio, and Dylan Hadfield-Menell. Open technical problems in open-weight AI model risk management. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=8QyGLnFkzc. Survey Certification. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang, and Jin-Long Li. Attention editing: A versatile framework for crossarchitecture attention conversion. arXiv preprint arXiv:2604.05688, 2026. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec. Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3–11, 2018. Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards LLM unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=zZjLv6F0Ks. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836. Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:11893–11905, 2022. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. arXiv:1503.02531, 2015.

Distilling the knowledge in a neural network.

arXiv preprint

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned LLMs via benign relearning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=fMNRYBvcQN. Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=lpXDZKiAnt. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tTPHgb0EtV. M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics - Simulation and Computation, 19(2):433–450, 1990. doi: 10.1080/03610919008812866. URL https://doi.org/10.1080/03610919008812866. Wei Junhao, Yu Zhe, and Jun Sakuma. Disrupting model merging: A parameter-level defense without sacrificing accuracy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17698–17707, 2025.

11

Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, et al. On the societal impact of open foundation models. arXiv preprint arXiv:2403.07918, 2024. Auguste Kerckhoffs. La cryptographie militaire. J. Sci. Militaires, 9(4):5–38, 1883. Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010. Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish K Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, and Alexander Matt Turner. Distillation robustifies unlearning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=UTGjik64IK. Chen Liang, Jiahui Yu, Ming-Hsuan Yang, Matthew Brown, Yin Cui, Tuo Zhao, Boqing Gong, and Tianyi Zhou. Module-wise adaptive distillation for multimodality foundation models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openreview.net/forum?id=JhQP33aMx2. Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852–20867. PMLR, 2023b. Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security, 2025. Ka Man Lo, Yiming Liang, Wenyu Du, Yuantao Fan, Zili Wang, Wenhao Huang, Lei Ma, and Jie Fu. m2mkd: Module-to-module knowledge distillation for modular transformers. arXiv preprint arXiv:2402.16918, 2024. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. Quoc Minh Nguyen, Trung Le, Jing Wu, Anh Tuan Bui, and Mehrtash Harandi. Antibody: Strengthening defense against harmful fine-tuning for large language models via attenuating harmful gradient influence. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=qur2ef8MqQ. Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open-weight LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= fXJCqdUSVG. Yeonju Ro, Zhenyu Zhang, Souvik Kundu, Zhangyang Wang, and Aditya Akella. On-the-fly adaptive distillation of transformer to dual-state linear attention for long-context LLM serving. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=pqHWzviKKN. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=eP9auEJqFg. Domenic Rosati, Sebastian Dionicio, Xijie Zeng, Subhabrata Majumdar, Frank Rudzicz, and Hassan Sajjad. Locking open weight models with spectral deformation. In ICML Workshop on Technical AI Governance (TAIG), 2025. URL https://openreview.net/forum?id=cjrm7bo6Eg. Domenic Rosati, Xijie Zeng, Hong Huang, Sebastian Dionicio, Subhabrata Majumdar, Frank Rudzicz, and Hassan Sajjad. Limits of convergence-rate control for open-weight safety. arXiv preprint arXiv:2602.18868, 2026. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2459–2475, 2025.

12

Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Tamper-resistant safeguards for open-weight LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FIjRodbW6. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Yuhui Wang, Rongyi Zhu, and Ting Wang. Self-destructive language models. In The Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview.net/forum?id=ERNpUGr8M5. Zihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, and Li Shen. Model unmerging: Making your models unmergeable for secure model sharing. arXiv preprint arXiv:2509.01548, 2025. Ziyao Wang, Nizhang Li, Pingzhi Li, Guoheng Sun, Tianlong Chen, and Ang Li. Towards building non-fine-tunable foundation models. arXiv preprint arXiv:2602.00446, 2026b. Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing bert by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859–7869, 2020. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Shuo Yang, Qihui Zhang, Yuyang Liu, Yue Huang, Xiaojun Jia, Kun-Peng Ning, Jia-Yu Yao, Jigang Wang, Dai Hailiang, Yibing Song, et al. Asft: Anchoring safety during llm fine-tuning within narrow safety basin. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34322–34330, 2026. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019. Bo Zhao, Robin Walters, and Rose Yu. Symmetry in neural network parameter spaces. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=jLpWq5QY6I. Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=yR47RmND1m. Amber Yijia Zheng, Site Bai, Brian Bullins, and Raymond A. Yeh. Model immunization from a condition number perspective. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum? id=uitj69FqD5. Wangchunshu Zhou, Ronan Le Bras, and Yejin Choi. Modular transformers: Compressing transformers into modularized layers for flexible efficient inference. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10452–10465, 2023.

13

A

Related Work

A comprehensive survey of security risks in open-weight models is provided by Casper et al. (2026). The first direction is preventing fine-tuning to harmful content (Huang et al., 2024; Tamirisa et al., 2025; Zhao et al., 2025; Huang et al., 2025; Zheng et al., 2025; Liu et al., 2025; Wang et al., 2026a; Yang et al., 2026; Nguyen et al., 2026) or robustifying unlearned models against relearning attacks (Lynch et al., 2024; Hu et al., 2025; Fan et al., 2025; Lee et al., 2025). The major line of work takes a meta-learning approach, optimizing a bilevel objective or training the model under simulated tampering.Qi et al. (2025) show that existing objective-level defenses are broken by simple hyperparameter changes, motivating structural alternatives. Another approach modifies the model’s internal representations (Rosati et al., 2024), which can be naturally combined with our method: representation modification alters the input– output relationship for safety, whereas our architectural replacement preserves it. The two mechanisms are complementary and can be applied jointly. A separate line of work targets model merging prevention (Junhao et al., 2025; Wang et al., 2025): these methods reparameterize weight matrices using the symmetries discussed in Section 2.2, moving the model into a different loss basin to disrupt merging. However, such reparameterizations can be detected and reversed by an attacker who knows the defense strategy. Preventing harmful model editing.

Building on knowledge distillation (Hinton et al., 2015), several works replace individual transformer modules with compact alternatives for model compression (Xu et al., 2020; Zhou et al., 2023; Liang et al., 2023a,b; Lo et al., 2024; Ro et al., 2025). Closest to our approach, Cheng et al. (2026) replace attention modules with alternatives such as multi-head latent attention via module-wise distillation without retraining from scratch. Module replacement via distillation.

Model locking aims to make pretrained weights resistant to any form of adaptation—not only harmful fine-tuning. While methods preventing harmful editing apply to both the open-weight and finetuning-as-a-service (FTaaS, where the provider performs the fine-tuning) settings, model locking specifically targets the open-weight case, where the attacker controls the full training pipeline. SpecDef (Rosati et al., 2025) and Wang et al. (2026b) are the closest works to ours; we discuss them in detail in the following paragraphs. Concurrently, Rosati et al. (2026) study spectral weight decomposition as a defense but acknowledge that adaptive attackers can trivially revert it. Additionally, they show that any spectral deformation can be diluted by decomposing it across consecutive layers, motivating a shift from weight-space defenses to our structural approach. Our work differs in that we formalize the problem (Definitions 2.1–2.3) and propose an architectural defense with a quantifiable locking factor κ. Model locking.

SpecDef P Pk maximizes the top-k singular values of weight matrices as an instance of equation 2.3 with Ω(θ) = l i=1 σi (Wl (θ)). This is a post-hoc method: it can be applied to any pretrained model without retraining from scratch. We note several open challenges: maximizing Ω while minimizing the task loss L is a multi-objective problem subject to the self-defeating structure discussed in Section 2.2 (Figure 2); moreover, using the task loss as the faithfulness constraint risks overfitting to that specific task, with no guarantee that general capabilities are preserved—in contrast, distillation-based approaches maintain the input-output relationship of the original model and thus preserve performance across all tasks in principle. Additionally, the experimental evaluation is conducted at moderate scale, leaving scaling behavior as an open question. Spectral Deformation (SpecDef; Rosati et al. 2025).

This method embeds a lottery-ticket-style binary mask during pretraining: a subset of weights is designated as “locked” (masked) and the remaining weights are trained to compensate. An attacker unaware of the mask structure optimizes all weights, entering a different optimization space from the one the model was trained in. We note several open challenges: the binary mask reduces effective model capacity since masked weights do not contribute to the forward computation; the mask structure may be detectable via statistics such as weight magnitude or second-order derivatives, in addition to the gradient norm they investigated; the experimental evaluation is conducted at small scale; and most importantly, the comparison between masked and unmasked models reports similar benchmark performance, but both achieve scores near chance level on binary tasks (e.g., ∼50%), making it difficult to conclude that the proposed locked method does not sacrifice the original performance. Non-Fine-Tunable Foundation Models (Wang et al., 2026b).

14

B

Activation memory analysis

This section provides the detailed activation memory analysis underlying Proposition 3.1. Proposition B.1 (DLR-Net activation memory). For a depth-L network of the form equation 3.1: (a) Full training. Each layer stores: the RMSNorm input hi (d), the RMSNorm output (d), the SiLU pre-activation (r), the SiLU output (r), and the block output (d) for the αi gradient. Per-layer cost: 3d + 2r ≈ 3d. Total: Θ(dL). (b) Frozen parameters. Each layer stores: the RMSNorm input hi (d, needed because the backward Jacobian depends on hi ) and the SiLU pre-activation (r). Per-layer cost: d + r ≈ d. Total: Θ(dL). Case (a) is the standard training cost. Importantly, case (b) shows that this overhead cannot be avoided even when the attacker freezes all DLR-Net parameters and trains only other components. Using this result, we now prove the cost amplification bound stated in the main text. Proof of Proposition 3.1. Recall that the pretrained transformer has N layers, hidden dimension d, and FFN intermediate dimension dff . The locked model replaces all N SwiGLU MLPs with depth-L DLR-Nets of rank r, where L = ⌊3dff /(2r)⌋ matches the original parameter count. By Proposition B.1(a), the total per-token activation memory for the MLP layers increases from Θ(N dff ) (one SwiGLU per layer) to Θ(N dL) (depth-L DLR-Net per layer), an amplification factor of Θ(d/r) at matched parameter count. On a GPU with fixed memory M , the maximum batch size is B = ⌊(M − Mfixed )/m⌋, where m is the pertoken activation memory. Let aattn denote the per-token per-layer activation memory for the attention blocks (unchanged), and let Bf , Bg denote the maximum batch sizes for the original and locked models respectively. The batch size ratio satisfies aattn + dff Bg . ≲ Bf aattn + dL The attacker must process the same number of tokens T (by assumption). Whether the reduced batch size is compensated by more gradient steps or more GPUs via data parallelism, the GPU-hour cost increases by at least aattn + dL κ ≳ . aattn + dff At matched parameter count, the numerator grows as Θ(d·dff /r), yielding κ = Θ(d/r) when MLP activations dominate (dL ≫ aattn ). □ Remark. The proof assumes naive full training without gradient checkpointing. In practice, the attacker may use checkpointing to trade compute for memory (as in Section 3.3), but the peak memory of this technique still grows with L: recomputation materializes activations one segment at a time, reducing the dependence to O(log L) with recursive checkpointing. In practice this makes OOM unlikely, but since M is treated as a constant in the asymptotic analysis, focusing on standard training without checkpointing is justified.

C

An example of ill-conditioning from weight symmetries

We illustrate the ill-conditioning caused by the weight symmetries discussed in Section 2.2 on the matrix factorization problem min ℓ(W1 , W2 ) = ∥W1 W2 − M ∥2F , (C.1) W1 ,W2 ∈Rd×d

where M ∈ Rd×d is a fixed target. This simple setting captures the structure of Example 2: for any invertible A ∈ Rd×d , the reparameterization (W1 , W2 ) 7→ (W1 A−1 , AW2 ) preserves the product W1 W2 exactly while changing the individual factors and hence the optimization landscape. We study the scalar case A = aI with a > 0, giving (W1 , W2 ) 7→ (a−1 W1 , aW2 ). Before presenting the experimental results, we show that such a scaling worsens the condition number of the Hessian, leading to slower convergence.

15

Proposition C.1 (Ill-conditioning under scaling). Let ℓ(W1 , W2 ) = ∥W1 W2 − M ∥2F with W̃1 = a−1 W1 and W̃2 = aW2 , and suppose that W1 and W2 are full-rank. Then, the condition number of the Hessian of ℓ with respect to the parameters (W̃1 , W̃2 ) satisfies κ ≥ a4 ·

2 σmax (W2 ) , 2 σmin (W1 )

(C.2)

where σmax (·) and σmin (·) denote the largest and smallest singular values, so that κ = Ω(a4 ). Proof. Since ℓ = ∥W̃1 W̃2 − M ∥2F , the gradients with respect to each weight matrix are given by ∇W̃1 ℓ = 2(W̃1 W̃2 − M )W̃2⊤ ,

∇W̃2 ℓ = 2W̃1⊤ (W̃1 W̃2 − M ).

(C.3)

Let vec(·) denote column-major vectorization of a matrix. Differentiating again gives H̃1,1 =

∂2ℓ = 2 (W̃2 W̃2⊤ ) ⊗ Id = 2a2 (W2 W2⊤ ) ⊗ Id , ∂ vec(W̃1 )2

(C.4)

where ⊗ denotes the Kronecker product. Similarly, we have H̃2,2 =

∂2ℓ = 2 Id ⊗ (W̃1⊤ W̃1 ) = a22 Id ⊗ (W1⊤ W1 ). ∂ vec(W̃2 )2

Therefore, the largest singular value of the Hessian H̃ with respect to (W̃1 , W̃2 ) is given by
σmax (H̃) = max v ⊤ H̃v ≥ max v ⊤ H̃1,1 v = 2a2 σmax (W2 W2⊤ ) ⊗ Id , ∥v∥2 =1

∥v∥2 =1

(C.5)

(C.6)

where the inequality follows by restricting the last d2 entries to zero, and the last equality is from equation C.4. Using the property of the Kronecker product, we have
2 σmax (H̃) ≥ 2a2 σmax W2 W2⊤ σmax (Id ) = 2a2 σmax (W2 ) . (C.7) Similarly, equation C.5 leads to

2 2 σmin (W1 ) . a2 Combining with equation C.7 gives the desired result. σmin (H̃) ≤

(C.8)

This result shows that the condition number is worsened at the rate of a4 . The full-rank assumption in Proposition C.1 is satisfied almost surely under standard random initialization. Remark C.2 (Connection to classical convergence bounds). Proposition C.1 only shows that the reparameterization worsens conditioning in this matrix-factorization example. Independently of this example, classical first-order optimization theory provides several conditions in which larger condition number leads to slower convergence (Boyd and Vandenberghe, 2004). As one representative case, we focus on the classical µ-strongly convex and L-smooth setting; the gradient descent with stepsize smaller than 1/L has the linear convergence rate 1 − µ/L, yielding the ϵ-iteration complexity t = O(L/µ log(1/ϵ)) = O(κ log(1/ϵ)), where κ is the condition number. Therefore, the larger condition number increases the complexity bound, which implies a slower convergence. We verify the above discussion empirically by training the matrix factorization problem equation C.1 with and without the scaling reparameterization. We set d = 64 with entries of M , W1 , W2 drawn i.i.d. from N (0, 1/d), apply the scaling with a = 100, and train with both SGD and Adam across three learning rates each. Experimental verification.

Figure 5 shows that the bad condition number actually leads to a slower convergence. For SGD at a = 100, the only smallest learning rate is stable; and the convergence is extremely slow because the gradient of the enlarged factor W̃2 = aW2 scales as a−1 , so it barely updates while W̃1 converges to its conditionally optimal value. Larger learning rates immediately diverge. Adam does not exhibit the same divergence because its element-wise normalization makes the effective step size approximately scale-invariant. However, it still converges slower than at a = 1, confirming that the ill-conditioning affects optimization even with adaptive methods. 16

Figure 5 Training loss for matrix factorization ∥W1 W2 − M ∥2F from balanced (a = 1, dashed) and scaled (a = 100,

solid) initialization. Left: SGD. Under the scaling, the two larger learning rates diverge (not shown). Only the smallest learning rate remains stable, yet it plateaus orders of magnitude above the a = 1 optimum. Right: Adam. All learning rates converge at a = 100, albeit slower than at a = 1. Shaded regions show the interquartile range over five seeds.

D

Additional experimental results

D.1

Perturbation limit: Hessian scope comparison

Figure 2 (right panel) computes the Hessian trace over all parameters. A natural question is whether restricting the penalty to a subset of parameters makes the optimization easier—for example, concentrating the sharpness penalty on a single layer or a random subset might reduce interference between the MSE and Ω objectives. This setting directly corresponds to increasing the condition number of the Hessian: high curvature in a subset of directions makes gradient-based fine-tuning over the full parameter space difficult. Figure 6 tests four scopes: all parameters, the first layer only, a random 10% mask, and a random 50% mask. The result is consistent across all scopes: trajectories move rightward (increasing MSE) as λ grows, but fail to reach the high-Hessian-trace region occupied by scaling-symmetry solutions. This confirms that the optimization difficulty observed in the main text is not an artifact of the penalty scope.

D.2

Inference latency

All latency measurements use bfloat16 with FlashAttention (Dao, 2024) on a single H100 GPU, compiled with torch.compile (reduce-overhead mode), and averaged over 200 iterations after 50 warmup steps. Figure 7 shows the single-MLP overhead ratio (DLR-Net / SwiGLU) as a function of batch size. Compilation eliminates kernel-launch overhead, reducing the ratio by roughly 10× compared to eager execution. For 8B and 32B the ratio further decreases at large batch sizes as GPU utilization improves; for 0.6B the small rank (r=32) keeps arithmetic intensity below the roofline, so the overhead remains memory-bound. Table 3 provides the complete overhead across all rank/depth configurations. The results above measure a single MLP in isolation. In a full transformer, attention, embeddings, and layer normalization are identical between the baseline and DLR-Net, and attention increasingly dominates at longer sequences, diluting the per-MLP overhead. Tables 4–5 show the full-model prefill overhead ratio for Qwen3-0.6B and 8B. The ratio decreases with sequence length: at seq=4096 the 8B overhead is 2.9× (vs. 4.6× at seq=1024), and at seq ≥ 32768 both models approach 1.2–1.4×.

Apple and the Apple logo are trademarks of Apple Inc., registered in the U.S. and other countries and regions.

17

Figure 6 Hessian-trace penalty across four parameter scopes. Same setup as Figure 2 (right panel), but each panel

restricts the Hessian trace Ω = Tr(HCE ) to a different parameter subset. Each curve is one trajectory; color encodes log10 λ (dark = small, bright = large). Circles: start; squares: end. Red stars: weight-symmetry solutions (a−1 W1 , aW2 ). x-axis: relative MSE; y-axis: relative Hessian trace over the corresponding subset. Across all scopes, no trajectory reaches the low-MSE, high-Hessian-trace corner occupied by scaling solutions.

Figure 7 Single-MLP overhead ratio (DLR-Net / SwiGLU) as a function of batch size, across three model scales and

three compilation modes. Each panel uses the deepest configuration for that scale (0.6B: r=32, L=141; 8B: r=128, L=143; 32B: r=256, L=149).

18

Table 3 Complete overhead ratio (DLR-Net / SwiGLU) for all rank/depth configurations at four representative batch

sizes, with reduce-overhead compilation on an H100. Column headers list the depth L for each scale (0.6B / 8B / 32B). Configurations with ratio < 1 indicate that the DLR-Net is faster than SwiGLU due to smaller working set and better cache behavior. Depth L (0.6B / 8B / 32B) n

Scale 141/143/149 71/71/74

1

0.6B 8B 32B

11.0× 11.2× 7.1×

0.6B 212 8B 32B

35/35/37

17/17/18

8/8/9

4/4/4

2/2/2

5.9× 5.9× 3.9×

3.3× 3.3× 2.3×

2.0× 2.1× 1.6×

1.3× 1.4× 1.3×

1.1× 1.2× 1.0×

1.0× 1.0× 0.9×

12.4× 6.4× 3.5×

6.5× 3.7× 2.1×

3.9× 2.1× 1.5×

2.4× 1.6× 1.2×

1.5× 1.1× 1.1×

1.4× 1.1× 0.9×

1.3× 1.0× 0.9×

0.6B 216 8B 32B

17.6× 5.4× 3.0×

8.9× 2.9× 2.0×

4.9× 1.9× 1.5×

2.9× 1.5× 1.2×

1.7× 1.1× 1.1×

1.5× 1.0× 0.9×

1.4× 1.0× 0.9×

0.6B 218 8B 32B

17.5× 5.1× 3.0×

8.9× 2.8× 2.0×

4.9× 1.8× 1.5×

2.8× 1.4× 1.2×

1.6× 1.1× 1.1×

1.5× 1.0× 0.9×

1.4× 0.9× 0.9×

Table 4 Full-model prefill overhead ratio (DLR-Net / baseline) for Qwen3-0.6B with reduce-overhead compilation on

a single H100. Rows: batch size. Columns: sequence length. “–” denotes configurations that exceeded GPU memory. Sequence length B

128

256

512

1024

2048

4096

8192

16384

32768

65536

1 2 4 8 16 32 64 128 256 512

18.7 16.4 14.4 11.0 8.5 6.4 6.1 6.1 6.0 5.7

16.7 13.9 10.3 7.9 5.9 5.7 5.8 5.7 5.4 5.3

13.5 9.4 7.1 5.5 5.3 5.3 5.2 4.9 4.8 –

7.8 6.1 4.5 4.5 4.5 4.4 4.2 4.2 – –

5.0 3.6 3.5 3.6 3.4 3.3 3.3 – – –

2.8 2.7 2.6 2.6 2.5 2.5 – – – –

2.0 2.0 1.9 1.9 1.8 – – – – –

1.6 1.5 1.5 – – – – – – –

1.3 1.3 – – – – – – – –

1.1 – – – – – – – – –

Table 5 Full-model prefill overhead ratio (DLR-Net / baseline) for Qwen3-8B with reduce-overhead compilation on a

single H100. Sequence length B

128

256

512

1024

2048

4096

8192

16384

32768

65536

1 2 4 8 16 32 64 128 256 512

10.3 9.1 6.9 5.3 4.3 4.2 4.0 3.9 3.8 3.7

8.9 6.6 5.1 4.3 4.1 3.9 3.8 3.7 3.7 –

6.5 4.9 4.1 4.0 3.8 3.7 3.6 3.6 – –

4.6 3.9 3.8 3.6 3.4 3.4 – – – –

3.5 3.4 3.3 3.1 3.0 – – – – –

2.9 2.8 2.7 2.6 – – – – – –

2.3 2.2 2.1 – – – – – – –

1.7 1.7 – – – – – – – –

1.4 1.4 – – – – – – – –

1.2 – – – – – – – – –

19