Can Watermarking Techniques Help Prevent LLM Model Stealing? Elette Boyle1 , MohammadTaghi Hajiaghayi2 , Keivan Rezaei2 , Suho Shin2 and Amos Stern1 1 Reichman University 2 University of Maryland
arXiv:2607.10794v1 [cs.CR] 12 Jul 2026
Abstract Model stealing attacks have recently been introduced, enabling the extraction of precise information from black-box commercial language models. In this work, we propose defense methods against a recent attack of [Carlini et al., 2024b] and extensions for extracting the hidden layer dimension of production language models. Our methods are inspired by watermarking techniques that perturb the logits layer of these models to prevent such attacks. We provide empirical experiments demonstrating the effectiveness of the proposed defense versus model quality degradation across various configurations, and propose an effective defense against such attacks while preserving model utility.
1
Introduction
Commercial language models (LMs) such as Gemini, GPT4, and Claude [Achiam et al., 2023; Team et al., 2023; Anthropic, 2024], which are publicly accessible, require substantial time and resources for training. Consequently, details about these models, including training data, methodologies, architectures, and parameters, are typically not disclosed. However, recent reports highlight the risks of model stealing, where unauthorized actors attempt to extract proprietary knowledge. For example, recent article [Reuters, 2025] indicate the possibility of knowledge being extracted from OpenAI’s API output to produce a competing product. These models operate through API access, allowing users to input prompts and receive predictions. [Carlini et al., 2024b] proposed techniques to extract precise information from various black-box LMs, such as their hidden state dimensions and final projection layers. This raises serious concerns that extension of these attacks could potentially reveal even more sensitive information. Specifically, [Carlini et al., 2024b] present a scalable approach targeting production LMs. Unlike earlier model stealing methods [Carlini et al., 2020; Carlini et al., 2024a; Rolnick and Kording, 2020], it does not depend on specific activation functions in the model or network structures, making it both versatile and a significant concern. This highlights the need for robust defenses against such attacks. The method described in [Carlini et al., 2024b] leverages a model’s API
Figure 1: Our framework of perturbing logits to prevent dimensionextraction model stealing attacks.
to extract logit vectors of next-token-prediction on a set of prompts. These logits are then used to construct a matrix, whose properties are used for extracting hidden dimension, and later the last projection layer in a black-box LMs. While the specific API queries that enable logit vector reconstruction in the [Carlini et al., 2024b] attack have since been removed from major models,1 their attack indicates a clear danger and potential vulnerability. In particular, the hidden latent dimension of the target model is a critical component for [Carlini et al., 2024b], as it serves as the foundation for stealing the model’s last layer. In this work, we propose a structured perturbation of logits to counter rank-estimation attacks, inspired by watermarking techniques [Kirchenbauer et al., 2023]. Unlike prior methods that sample and add independent noise, we seed targeted perturbations using a private hash of the model’s hidden latents. Our approach significantly increases the complexity of attacks attempting to break the defense, while preserving the model’s quality. Our overall framework is given in Figure 1. We remark that watermarking has been used for post-hoc detection of behavioral cloning [Zhao et al., 2023]. Our method takes a very different approach, proactively disrupting rank-based extraction attacks of architectural details such as hidden dimension. (See Related Work in Appendix A.) 1 Still, one may recover the entire logits using a binary search-like approach [Morris et al., 2023].
1.1
Summary of results
We study rank-estimation attacks based on principal component analysis (PCA) and its variants, and develop defense mechanisms that are robust to any such attack. We first summarize the main attack methods considered, then describe our defenses. Further details appear in Sections 3 and 4. Attack methods We briefly describe the main classes of attacks, noting that we additionally consider stylized variants tailored to specific defenses. PCA. Following [Carlini et al., 2024b], the attacker constructs a matrix whose columns are logit vectors obtained from controllable prompts, and applies PCA to estimate the numerical rank, corresponding to the hidden dimension of the LLM. PCA is robust to small additive perturbations: if the observed matrix is A + E with ∥E∥op small, Weyl’s inequality guarantees recoverability of the true rank. PCA with averaging. In practice, PCA can be sensitive to noise with large spectral norm, such as i.i.d. Gaussian noise. In the additive-noise setting, an attacker can mitigate this by querying the same prompt multiple times and averaging the resulting matrices, thereby reducing the variance and spectral norm of the noise and enabling rank recovery. Robust PCA. If the defender injects sparse but largemagnitude noise, classical PCA may fail. However, robust PCA [Candès et al., 2011] shows that exact rank recovery remains possible under reasonable conditions via convex optimization, even with gross but sparse corruptions. Defense mechanisms A naive defense is to add independent noise to each logit coordinate. This approach is undesirable due to nonreproducibility, and is fundamentally vulnerable to averaging attacks, which can effectively remove independent noise. To counter this, the injected noise should be promptdependent yet deterministic: identical prompts should always induce the same perturbation, preventing averaging, while different prompts induce different noise. Prompt-dependent noise via pseudorandom functions. We generate noise deterministically using a cryptographic pseudorandom function (PRF) applied to the model’s last hidden states, with a secret key held by the model owner. This ensures reproducibility while appearing random across prompts. This idea is inspired by LLM watermarking [Kirchenbauer et al., 2023], but unlike token-based seeding, our construction avoids trivial attacks where an adversary forces identical seeds across prompts. Additive versus nonadditive noise. Given promptdependent noise, a natural approach is additive perturbation of logits. We empirically study additive noise across distributions, finding it often effective but potentially vulnerable to sophisticated averaging-style attacks. We therefore also consider nonadditive alternatives. Simple multiplicative noise significantly degrades model quality and still fails to prevent rank recovery (Appendix 4.1). Preserving the relative ordering of logits is critical for maintaining LLM quality. This motivates an order-preserving
additive variant that reorders sampled noise to match the logit ranking before addition. Despite its appeal, this method is ineffective: the reordered noise matrix itself lies in a low-dimensional subspace, enabling rank recovery (Appendix 4.1). Softplus-then-perturb mechanism. We therefore propose the softplus-then-perturb (STP) mechanism. It first applies an order-preserving softplus transformation to skew the logits, then adds prompt-dependent perturbations. The initial skewing helps preserve token ordering under perturbation, while the subsequent noise prevents rank-estimation attacks. We evaluate multiple perturbation schemes and characterize their trade-offs between quality and robustness. Gaussian perturbations combined with STP are robust to all considered attacks while maintaining high output quality.
2
Problem Setup
We investigate model stealing attacks [Carlini et al., 2024b] that exploit the logits generated by models. We assume the adversary is able to make adaptive blackbox prompt queries to the model and receives the corresponding logit vectors. By collecting logit vectors corresponding to the model’s predictions for next-token generation across multiple prompts, the attackers construct a logits matrix and estimate its rank to infer the hidden dimension. This serves as a stepping stone toward further stealing of the model’s last layer’s parameters. Our defense, inspired by watermarking techniques of [Kirchenbauer et al., 2023], aims to perturb the logits in a way that prevents accurate rank estimation, thereby mitigating rank-based model stealing attacks. Importantly, these perturbations must not degrade the language model’s performance or render it unusable. Thus, evaluating the language model’s utility is essential to validate the practicality of the defense.
2.1
Preliminaries
Let [n] = {1, 2, . . . , n}. Let T be the set of tokens in a LLM and T be its cardinality. For a set X, we write X ∗ to denote the family of every finite sequence over X. To describe the overall behavior of the LLM token generation process, once a user writes a prompt p ∈ T ∗ into the LLM, the LLM generates the corresponding hidden state h ∈ RH . Then it maps each hidden state to vocabulary logits ℓ ∈ RV by matrix multiplication ℓ = W ⊤ h where W ∈ RH×T . Finally, it (possibly randomly) samples a vocabulary based on the logit vector ℓ according to the LLM’s decoding algorithm, then the user sees the next token written by LLM. [Carlini et al., 2024b] propose a model stealing attack where an adversary tries to infer the hidden state dimensionality H from N controllable prompts. Let such prompts be {pi }i∈[N ] , and corresponding hidden states be {hi }i∈[N ] with hi ∈ RH . For each yi , the LLM generates the logit vector ℓi = W ⊤ hi , where W refers to the projection matrix that maps the latent space to the vocabulary space. The adversary can access the logit vector {ℓi }i∈[N ] , and use this information to infer H. [Carlini et al., 2024b] suggest that H
can simply be obtained by computing the rank of the matrix constructed by stacking the ℓi vectors and compute its rank, i.e., H = rank ([ℓ1 , ℓ2 , . . . , ℓN ]) . Indeed, H intrinsically serves as an upper bound on this rank; in practice, the hidden latent generates a full-rank matrix (and W itself is full rank), resulting in equality. We aim to design a perturbation mechanism that perturbs each logit vector ℓi to ℓ′i to prevent such attacks, while preserving the model’s quality.
3
Rank Estimation of Noisy Matrices
A crucial step in the attack of [Carlini et al., 2024b] is computing the rank of the generated logits matrix (ℓ1 , . . . , ℓN ). We next provide a deeper treatment of the rank estimation procedure used in [Carlini et al., 2024b] and extensions. Baseline: Numerical rank from [Carlini et al., 2024b]. Practically, since one can only access every numeric values over floating-point numbers that prevents us from computing the exact number of linearly independent rows, one can instead compute the numerical rank by principal component analysis (PCA) to obtain a sorted list of singular values σ1 ≥ σ2 ≥ . . . ≥ 0, and identify the index i with the largest multiplicative gap σi /σi+1 (alternatively, logarithmic difference). This index i ∈ [N ] constitutes the output of their attack: namely their prediction for the hidden dimension. Beyond [Carlini et al., 2024b]. Consider instead a perturbed logit matrix, e.g. where each logit vector is offset by additive noise. Intuitively, since the magnitude of the introduced noise should be sufficiently small to guarantee that the perturbed model does not deviate too much from the original, one may ask whether the adversary can still recover the rank of the matrix by conducting PCA and investigating the singular values. Whether this approach could successfully recover the rank, however, largely depends on the density and magnitude of the noise. In what follows, we introduce some seminal results on matrix perturbation theory to overview possible approaches to recover the rank of a noised matrix beyond the model (rank) stealing attack by [Carlini et al., 2024b].
3.1
PCA: small but possibly dense noise
Given a matrix M , let ∥M ∥2 = inf{c ≥ 0 : P ∥Av∥2 ≤ c ∥v∥2 , ∀v ∈ Rn } be operatorPnorm, ∥M ∥∗ = i σi (M ) be nuclear norm, and ∥M ∥1 = i,j |Mi,j | be ℓ1 -norm of M , where σi (M ) denotes i-th largest singular value of M . Suppose we have a matrix A ∈ Rm×n of rank r. Then it possesses r nonzero singular values. We consider a noise matrix E and the perturbed matrix A + E. We want to understand how large E (in operator norm) must be in order to make A + E change rank. Weyl’s inequality. A core fact from matrix perturbation theory is that if ∥E∥2 < σmin (A), then A + E must still have rank r. Intuitively, the operator norm ∥E∥2 cannot push a nonzero singular value of A to zero unless it is at least as large as that singular value. Equivalently, to modify a low-rank matrix into a higher-rank one, the noise must have operator norm exceeding the relevant singular values. Formally, a classical result is the following version of Weyl’s inequality:
Theorem 1 ([Weyl, 1912]). Let A, E ∈ Rn×m , and let σi (·) denote the i-th largest singular value. Then, for each i ≥ 1, σi (A + E) − σi (A) ≤ ∥E∥2 . In particular, no singular value of A can increase by more than ∥E∥2 , implying that substantial noise is required to alter the numerical rank. Given these stability guarantees, PCA (or SVD) is a natural tool for estimating the rank of a matrix contaminated by moderate noise. In practice, one computes the SVD of A + E and identifies a spectral gap: a large drop between σr and σr+1 typically signals rank r.
3.2
PCA & averaging: reducing spectral norms
From Theorem 1, analyzing the operator norm of the noise matrix is a key step in understanding when PCA-based rank recovery succeeds. For sub-Gaussian noise matrices (including Bernoulli and Gaussian), the operator norm scales with dimension: Theorem 2 (Chapter 4, [Vershynin, 2018]). Let E be an m×n random matrix √ with √independent sub-Gaussian entries. Then ∥E∥2 = Ω( m + n). This follows since the operator norm is bounded below by the norm of any row or column. As a result, Weyl’s inequality alone provides only weak guarantees for rank recovery in high dimensions. However, an adversary can significantly reduce the effective noise magnitude if multiple copies of the same logit matrix are available with independent noise. The key idea is averaging: querying each prompt multiple times and averaging the resulting matrices reduces noise variance. Formally, let A + E denote the N × T logit matrix for prompts p1 , . . . , pN , and suppose independent noise is added across k repeated queries. This yields matrices A′j = A + Ej for j ∈ [k], where the Ej are independent. Averaging gives Ã′ =
k k 1X ′ 1X Aj = A + Ej , k j=1 k j=1
so the operator norm of the noise decreases as k grows. This argument relies critically on independence of the Ej .2 Discussion. Due to the structured nature of logits, averaging can be even more effective than this analysis suggests. Each noise-free logit vector lies in the low-dimensional column space induced by the projection matrix W , so averaged matrices remain close to a low-rank subspace. If the noise Ej is weakly correlated with Aj , the averaged matrix can again be expressed as a low-rank signal plus reduced-variance noise. Thus, the effectiveness of averaging crucially depends on the dependence structure between Aj and Ej . Empirically, we observe that independent noise degrades under averaging around k = 40, while dependent (embedding-based) noise remains robust beyond k = 50 (Appendix D.1). 2 In contrast, our defenses use deterministic pseudorandom noise, so repeated queries yield identical perturbations.
3.3
Robust-PCA: sparse but possibly gross noise
What if the noise is large enough and the adversary cannot rely on the averaging argument due to potential dependency between the noise matrices? In this case, one might employ robust PCA [Candès et al., 2011] (RPCA) instead of the standard PCA to estimate the numerical rank of matrices. RPCA seeks to robustly estimate the original matrix given a perturbed matrix, when the noise matrix is sparse yet could have large values in element-wise. Surprisingly, such estimation can be done by solving the following convex optimization problem, termed principal component pursuit (PCP), given an observed matrix A′ : minimize ∥A∥∗ + λ ∥E∥1 subject to A + E = A′ . [Candès et al., 2011] show that this exactly recovers the low-rank matrix A and sparse noise E even if the rank of A grows near-linearly in the dimension of the matrix and the noise in each element of E is up to a constant factor of the original matrix A. See Appendix B for more details.
4
Defense against dimension extraction
In this section, we propose defense strategies against rank estimation attacks. Let p represent a prompt input to the LLM, where the model’s standard forward pass generates the logit vector ℓ, used for sampling the next token. Our methodology involves perturbing the logits layer, transforming ℓ into a perturbed vector ℓ′ . These perturbations alter the properties of the logits matrix, that the adversary constructs by stacking logits corresponding to different prompts. Formally, inspired by [Kirchenbauer et al., 2023], given an input prompt p ∈ T ∗ and corresponding logit vector ℓ ∈ RV , our method finds a bias vector bias(p) ∈ RT as a function of the prompt p and computes ℓ′ = f (ℓ, bias(p)) for perturbation function f . Then it decodes the next token based on the perturbed logit vector ℓ′ .
4.1
Figure 2: Dimension-extraction attack on Mistral-7B. Each row of the matrix was constructed by sampling Gaussian noise ∼ N (0, 1), then reordering the noise values to match the rank order of the original logits. Singular Value Decomposition (SVD) was performed on this sorted noise matrix—without using the original logits—to estimate the hidden dimension.
it robust is to impose nonadditive noise. We consider a multiplicative noise such that we sample noises ε i.i.d. from uniform distribution over [0, 10],3 and then multiply the logits by 1 + ε. Interestingly, however, we observe that this scheme is still significantly vulnerable to simply performing PCA over the final matrix, as shown in Figure 3. This suggests that more complicated noise structure is necessary to make it robust against PCA.
Various noises and their vulnerability
We first discuss why simple noise structures are vulnerable to attack, providing insights behind our proposed mechanism. Warm-up: fixed noise. We first explain why the noises should be varying with respect to the prompt. Suppose we run a perturbation scheme that deterministically uses a fixed bias function bias(p) = ⃗b ∈ RT . Then, the adversary can input prompts {p1 , p2 , ..., pN }, stack perturbed logits, and recover the rank of the hidden layer as the rank remains the same. Namely, rank([ℓ1 , . . . , ℓN ]) = rank([ℓ1 + ⃗b, . . . , ℓN + ⃗b]). Thus, a reasonable defense method should vary the noise added to the logit vector ℓ to avoid such scenario. Sorted Gaussian noise. We constructed a matrix E by sampling Gaussian noise as follows: for each row, we sampled T independent values from N (0, 1) and then sorted these values according to the ordering of the original logits. Remarkably, we discovered that the matrix E ′ of reordered Gaussian samples alone exhibits similar low-dimensional structure as the LLM’s hidden representations. Singular value decomposition of E ′ reveals a signal corresponding to the actual hidden dimension, as demonstrated in Figure 2. Multiplicative noise. Given that additive noise is vulnerable to PCA with averaging attack, a promising approach to make
Figure 3: Dimension-extraction attack on Mistral-7B. Each row of the matrix was generated by sampling ϵ ∼ U(0, 10) and multiplying the logits by 1 + ϵ. While this row-wise multiplicative noise significantly degrades the model’s output quality, it does not impede the attack. Thus, this type of noise is ineffective as a defense.
4.2
Proposed Method
We consider a defense mechanism that introduces a structured pseudorandom noise to the logit vector for each query prompt, generalizing the additive noise in the watermarking method of [Kirchenbauer et al., 2023]. More concretely, the noise values are derived from a random sample from a given noise distribution D, with the sampling process determined by a seed. Importantly, this seed is computed as a function of the input prompt, ensuring that 3
This is an extremely strong version of the defense mechanism. Any weaker perturbation would still be vulnerable to the dimension extraction attack.
Algorithm 1: Embedding-Dependent Noise Addition Input: LLM forward function F , prompt p, noise distribution D, PRF, and perturbation function f . Output: Biased logits 1: ℓ, h ← F (p) // logits ℓ and last hidden state h 2: s ← PRF(h) // seed s as a function of h 3: noise(p) ← D(seed = s) // sample noise vector 4: return: f (ℓ, noise(p)) the perturbation process is deterministic. We design the seed as a cryptographic pseudorandom function (PRF) applied to the model’s last hidden states. A PRF is an efficiently computable keyed function family such that for a random and secret key k, the function PRFk (k, ·) is computationally indistinguishable from a truly random function. In particular, any set of evaluations PRF(k, xi ) on different inputs xi appear as i.i.d. sampled outputs; however, on the same input x, PRF(k, x) will always return the same output. This design ensures that the added noise depends on the prompt-specific hidden states, which encode the features the language model extracts based on the input tokens to predict the next token. Finally, the sampled noise and the logit vector outputs a final perturbed logit vector with respect to a perturbation function f . The pseudo-code for generic defense mechanism is provided in Algorithm 1. In the system setup, we assume the model owner samples and maintains a secret PRF key k once for all time, and omit k from PRF notation. Additive noise. We first consider additive noise, analogous to [Kirchenbauer et al., 2023], i.e., f (ℓ, noise(p)) = ℓ + noise(p). We investigate two types of noise distribution families. Discrete noise of magnitude δ and sparsity γ corresponds to a random vector in which γ fraction of the coordinates are set to δ, and the remaining are 0. Gaussian noise with standard deviation (std) σ and sparsity γ will correspond to a random vector in which γ fraction of coordinates are sampled as i.i.d. Gaussian with mean 0 and std σ, and the remaining are 0. We note that for constant choices of magnitude and sparsity, the N × T matrix formed by N samples from either distribution will have high row√norm (approximately γδT in the case of discrete noise, and γT for Gaussian noise), and in turn high spectral norm. This means they stand as valid candidates for increasing the rank of the LLM logits matrix. Softplus noise. We next consider further approaches toward noise perturbation, with the aim of improving the effect on the model output. We observe that maintaining the ordering of each token within the logit vector plays a significant role in model quality, which is likely to be reordered given additive noise to individual logit vector components. We thus propose a generic mechanism, dubbed softplusthen-perturb (STP) mechanism, that first applies the softplus transformation to the logit vector to make it more skewed which would help preserve the original order after further modification, then apply random perturbation to prevent attacks. More concretely, in the above notation, f (ℓ, noise(p)) = softplus(αℓ) + noise(p), where α is either a fixed or per-query randomized scale factor.
We remark that while softplus(αℓ) itself already yields a matrix with high rank, this alone is insufficient for defense, as an attacker can simply invert the softplus computation to retrieve a matrix whose rows lie within the original low-rank subspace. In turn, we investigate variations of the discrete and Gaussian noise families above for noise(p) distribution. Other options for random seed. For the embeddingdependent noise, we select the seed for noise to be a function of the model’s hidden state when prompted with an input p. We emphasize that this design choice crucially differs from that of [Kirchenbauer et al., 2023], who added noise seeded by a sliding window of the last few tokens. The reason for this modification is that it is critical in our setting for an adversary to be unable to generate several different prompts yielding the same seed. Otherwise, this enables producing identical noise that fails to alter the rank of the logits matrix. For example, if the noise is seeded by the last m tokens, an adversary could simply append m fixed tokens to a collection of prompts, resulting in identical noise for each and effectively compromising the defense. To remove this vulnerability, we design our noise addition to be dependent on the model’s inner states. Relying on hidden states (together with the security of the PRF) ensures that the noise generation becomes significantly harder to replicate, thereby making the defense more robust. If one wishes to remove dependence on the model, the seed can instead rely on a hash of the complete input prompt.
5
Experiments
In this section, we provide empirical evidence of the effect of various distributions of additive noise in preventing the hidden dimension extraction attack of [Carlini et al., 2024b], in the case of Mistral-7B. Complimentarily, we demonstrate that our noise modifications do not significantly degrade the quality of the underlying models by providing evaluations of resulting model perplexity. In Appendix D.3, we provide analogous experiments for GPT-2. Implementation. All experiments were performed in the Google Colab environment, using the A100 GPU runtime type. We instantiate the PRF output for sampling noise vectors via BLAKE3 [O’Connor et al., 2020] in keyed-hash mode. In some cases, the resulting pseudorandom output is extended further using the numpy python package RNG, seeded by the BLAKE3 output.
5.1
Model Stealing Prevention
We consider the effectiveness of the [Carlini et al., 2024b] attack on extracting hidden model dimension for an assortment of noise parameters, including discrete noise and i.i.d. Gaussian with varying magnitude and sparsity. In each experiment with Mistral-7B (token vocabulary size T = 32, 000, hidden dimension 4,096), we query N = 10, 000 prompts, consisting of randomly sampled token pairs from the model vocabulary (following [Carlini et al., 2024b]). Each resulting output logit vector is additively perturbed by a noise vector sampled using randomness that is computed by applying a PRF to the last hidden state (see Algorithm 1). The resulting N perturbed logit vectors are appended into an
(a) Discrete noise
(b) Gaussian noise
Figure 4: Summary of attack success on Mistral-7B given additive discrete or Gaussian noise as a function of noise magnitude (x axis) and sparsity (curves). Hidden dimension difference (y axis) indicates distance between the attack’s predicted hidden dimension and ground truth, 4,096. Reaching the top value y = 4, 096 indicates complete failure of the attack.
N ×T matrix A = (ℓ̃1 , . . . , ℓ̃N ). We perform SVD decomposition of the matrix A; sort the resulting N singular values in decreasing order; compute the difference between each pair of adjacent singular values; and output index i ∈ [N ] for which this distance is maximal as the predicted dimension. RPCA attack. We implement an extension of the SVDbased attack, using the Robust PCA’s PCP algorithmic procedure from [Candès et al., 2011]. Interestingly, the RPCA process did not appear to significantly enhance the attack. In each of our tested experimental noise parameter settings, the standard PCA attack and the RPCA algorithm for rank estimation either both succeeded or both failed to provide a realistic estimate of the matrix rank. See Appendix C for additional information. Softplus inversion attacks. We explore tailored extensions of the [Carlini et al., 2024b] attack against variations of the softplus-then-perturb mechanism (Figure 5(b)). As mentioned, applying softplus(αℓ) alone to (even randomly α-scaled) logit vectors does not suffice as a defense mechanism, as this operation can be inverted by the adversary to yield α-scaled copies of the original logit vectors, (yielding the same original matrix rank). When further perturbation is added, however, inversion does not work directly. We identified new attacks for softplus with discrete noise. Under positive bias +b, we recovered rank approximations (e.g., 3,600 for +b = 1.0, sparsity 0.5) by applying PCA to the N × T matrix constructed via attack I: (1) shifting each perturbed logit vector ℓ̃i (for each row i ∈ [N ]) componentwise by the difference between its mean and the global mean; (2) replacing non-positive elements with ϵ = 10−10 ; and (3) softplus inversion. For negative bias −b, many elements became negative, invalidating softplus inversion. Instead, attack II exploits the constant bias: the most negative postperturbation value estimates −b, which we subtract from all non-positive elements, effectively removing the bias. Softplus plus Gaussian noise, on the other hand, withstood all attack attempts. The varying Gaussian offsets disrupted the softplus-inversion without evident means of removal. Further, the nonlinear skew via softplus disrupted attack methods on pure additive noise such as PCA with averaging. We leave
further investigation of the attack space on the softplus-thenperturb class (including Gaussian and beyond) as an important direction of study.
5.2
Quality Evaluation
Overall, we validate various quality metrics for LLMs including perplexity (PPL), massive multitask language understanding (MMLU) as well as several distributional shift metrics between the original logit vector and the final perturbed logit vector that capture how much our defense mechanism changes the logit vector. These distances include total variation distance (TVD), Kullback–Leibler divergence (KL), Jensen-Shannon divergence (JS), and top 50 token’s agreement percentage (Top-50) between two logit vectors. Perplexity evaluation. By design, our discrete noise perturbation (specifically, our relatively conservative magnitude 2.0 and sparsity 0.5) corresponds identically to the perturbation induced by the soft watermarking procedure in [Kirchenbauer et al., 2023]. In turn, we directly inherit the corresponding conclusions regarding benign impact on quality of generated text. We use Meta-Llama-3-8B [Dubey et al., 2024] as larger oracle model to evaluate perplexity. To generate samples, we took a random selection of 500 texts each of length 500 tokens from the news-like subset of the C4 dataset [Raffel et al., 2023], and removed the final 100 tokens. Each remaining 400 tokens served as a prompt to the model being tested, where completions of length 100 were generated using greedy sampling with a temperature of 0.1. These prompts were additionally given to the oracle MetaLlama-3-8B model, and perplexity was computed as the exponential of the average token-wise loss with respect to the predicted next-token distribution of the oracle model for each token output. The resulting calculated perplexity measurements are given in Table 1. We observe that while some noise-perturbed models exhibit only modest increases in perplexity compared to the original model, others show more significant degradation, especially with stronger perturbation settings. These results indicate that the perturbations do not uniformly affect text quality, with some configurations maintaining near-original performance. More sophisticated sampling procedures, such as
(a) Constant bias comparison
(b) STP comparison
Figure 5: Dimension-extraction attack on Mistral-7B with (a) Discrete additive noise for a progression of noise magnitudes (0, 0.1, 0.5, 1.0, and 2.0) and sparsity 0.5. Each plot depicts the sorted singular values of the corresponding noisy logits matrix, demonstrating a significant jump at correct position 4,096 for no noise and low noise 0.1; an inflection point at 4,096 for mid-level noise 0.5; and no identifiable anomaly for the case of larger noise 1.0 and 2.0. (b) STP comparison showing different noise perturbations and attacks. The pure softplus inversion (blue) shows a sharp drop at position 4,096, clearly revealing the model dimension. The STP with bias 1 attack I (orange) shows a notable drop at around position 3,600, while the STP with bias -1 attack II (green) shows more gradual degradation. When using the attacks on STP with bias ∼ N (0, 1), there is no identifiable anomaly. We provide analogous plots for further settings in Appendix D.2.
beam search [Kirchenbauer et al., 2023], can result in further improvements in perplexity. Defense Mechanism
Additive
Softplus α ∼ Uni[1, 4]
γ
Bias
MMLU
PPL
TVD
KL
JS
Top-50
Breaks attacks
0.25 0.1 0.5 0.1
1 2 2 3
54.61 49.52 49.28 44.75
3.73 4.47 4.49 6.43
0.18 0.27 0.34 0.47
0.09 0.23 0.38 0.59
0.02 0.06 0.08 0.15
0.81 0.71 0.66 0.51
Yes Yes Yes Yes
0.25 0.5
∼ N (0, 1) ∼ N (0, 1)
55.28 53.65
3.58 3.74
0.14 0.23
0.12 0.22
0.03 0.05
0.85 0.76
Yes Yes
No Perturbation 0.25 1 0.25 -1 0.25 ∼ N (0, 1)
57.78 56.67 56.87 56.55
3.26 3.35 3.39 3.37
0.48 0.50 0.49 0.50
3.22 3.25 3.26 3.27
0.20 0.21 0.21 0.21
1.00 0.92 0.93 0.94
No: Inv+SVD No: attack I No: attack II Yes
57.78
3.40
0.0
0.0
0.0
1.0
No: SVD
Original
Table 1: Evaluation scores for Mistral-7B. Original refers to the original Mistral-7B model. γ denotes the sparsity parameter of the noise, and bias denotes the magnitude of the bias (e.g., 1 denotes constant bias). The last four rows denote the softplus-then-perturb mechanisms with corresponding perturbations, where α is sampled randomly for each prompt. Distributional shift metrics are calculated as the average from 10,000 distinct prompts.
MMLU Evaluation. We evaluate the performance of both the unmodified and perturbed Mistral-7B models on the Massive Multitask Language Understanding (MMLU) benchmark [Hendrycks et al., 2021]. MMLU is a widely adopted benchmark comprising 57 diverse tasks that test language model’s reasoning ability across subjects such as mathematics, science, and the humanities. Under the zero-shot setting, we assess the extent to which noise perturbations degrade performance in knowledge-intensive tasks. The original model achieves an MMLU accuracy of 57.78%. The additive noise shows significant degradation (44.75-55.28%). Softplus-based noise functions, while inducing high distributional divergence, preserve the top-k logit ordering and thus maintain strong MMLU performance. The pure Softplus variant matches the original model’s accuracy (57.78%), and even with additional constant or Gaussian noise, the performance drop remains modest (around 56-57%). This suggests that preserving the relative ranking of high-confidence log-
its can mitigate the impact of perturbations on task accuracy, even with significant shifts in the probability distribution. These results highlight that the impact of noise perturbations on model performance is not solely determined by the magnitude of the perturbation. Instead, the functional characteristics of the noise play a significant role. Specifically, Softplus-based perturbations demonstrate that it is possible to introduce substantial distributional shifts while still preserving strong performance on benchmarks like MMLU, due to their ability to maintain the relative ranking of logits. Distributional Shift Metrics. As expected, stronger constant bias noise creates larger divergence, with the most aggressive configuration (γ = 0.1, bias=3) exhibiting highest KL (0.59) and JS divergence (0.15), and a significant drop in top-50 agreement (0.51). Gaussian perturbations result in moderate divergence and more stable output agreement. Softplus-based perturbations behave differently. Despite inducing comparatively high divergence values (e.g., KL > 3.2, JS > 0.21), they maintain high top-50 accuracy agreement (≥ 0.92) and negligible degradation in perplexity or MMLU accuracy. This behavior arises because the Softplus function primarily affects lower-magnitude logits while preserving the relative ordering of top-scoring logits. As a result, the model’s most confident predictions remain largely unchanged, which explains the stability in accuracy despite measurable distributional shifts.
6
Conclusion
Our work proposes watermarking-inspired defenses against model stealing attacks targeting hidden dimension extraction via rank estimation [Carlini et al., 2024b], achieved by perturbing the logits layer of large language models. Guided by theoretical analysis and validated through extensive experiments, our technique proves robust against adversaries seeking to extract hidden latent dimensions. The results show that the proposed approach remains effective across different configurations, highlighting its potential to bolster the security and integrity of commercial language models.
References [Abdelnabi and Fritz, 2021] Sahar Abdelnabi and Mario Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pages 121–140. IEEE, 2021. [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [Anthropic, 2024] Anthropic. Claude 3 model family: Opus, sonnet, haiku, 2024. [Atallah et al., 2001] Mikhail J. Atallah, Victor Raskin, Michael Crogan, Christian F. Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-ofconcept implementation. In Information Hiding, 2001. [Candès et al., 2011] Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37, 2011. [Carlini et al., 2020] Nicholas Carlini, Matthew Jagielski, and Ilya Mironov. Cryptanalytic extraction of neural network models. In Annual international cryptology conference, pages 189–218. Springer, 2020. [Carlini et al., 2024a] Nicholas Carlini, Jorge Chávez-Saab, Anna Hambitzer, Francisco Rodrı́guez-Henrı́quez, and Adi Shamir. Polynomial time cryptanalytic extraction of deep neural networks in the hard-label setting. arXiv preprint arXiv:2410.05750, 2024. [Carlini et al., 2024b] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of a production language model. arXiv preprint arXiv:2403.06634, 2024. [Dubey et al., 2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. [Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. Int. J. Comput. Vision, 129(6):1789–1819, June 2021. [Hendrycks et al., 2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. [Jagielski et al., 2020] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In 29th USENIX security symposium (USENIX Security 20), pages 1345–1362, 2020.
[Kirchenbauer et al., 2023] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pages 17061– 17084. PMLR, 2023. [Konstantinides and Yao, 1988] Konstantinos Konstantinides and Kung Yao. Statistical analysis of effective singular values in matrix rank determination. IEEE Transactions on Acoustics, Speech, and Signal Processing, 36(5):757–763, 1988. [Lau et al., 2024] Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Waterfall: Framework for robust and scalable text watermarking and provenance for llms. arXiv preprint arXiv:2407.04411, 2024. [Meral et al., 2009] Hasan Mesut Meral, Bülent Sankur, A. Sumru Özsoy, Tunga Güngör, and Emre Sevinç. Natural language watermarking via morphosyntactic alterations. Comput. Speech Lang., 23:107–125, 2009. [Milli et al., 2019] Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 1– 9, 2019. [Morris et al., 2023] John X Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. arXiv preprint arXiv:2311.13647, 2023. [Munyer et al., 2023] Travis Munyer, Abdullah Tanvir, Arjon Das, and Xin Zhong. Deeptextmark: A deep learning-driven text watermarking approach for identifying large language model generated text. arXiv preprint arXiv:2305.05773, 2023. [O’Connor et al., 2020] Jack O’Connor, Jean-Philippe Aumasson, Samuel Neves, and Zooko Wilcox-O’Hearn. BLAKE3: A highly parallel cryptographic hash function, 2020. [Panaitescu-Liess et al., 2025] Michael-Andrei PanaitescuLiess, Zora Che, Bang An, Yuancheng Xu, Pankayaraj Pathmanathan, Souradip Chakraborty, Sicheng Zhu, Tom Goldstein, and Furong Huang. Can watermarking large language models prevent copyrighted text generation and hide training data? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25002–25009, 2025. [Por et al., 2012] Lip Yee Por, KokSheik Wong, and Kok Onn Chee. Unispach: A text-based data hiding method using unicode space characters. J. Syst. Softw., 85(5):1075–1082, May 2012. [Raffel et al., 2023] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
[Reuters, 2025] Reuters. Microsoft probing if deepseeklinked group improperly obtained openai data. Reuters, 2025. Accessed: 2025-01-30. [Rizzo et al., 2016] Stefano Giovanni Rizzo, Flavio Bertini, and Danilo Montesi. Content-preserving text watermarking through unicode homoglyph substitution. Proceedings of the 20th International Database Engineering & Applications Symposium, 2016. [Rolnick and Kording, 2020] David Rolnick and Konrad Kording. Reverse-engineering deep relu networks. In International conference on machine learning, pages 8178– 8187. PMLR, 2020. [Stewart, 1998] Gilbert W Stewart. Perturbation theory for the singular value decomposition. Citeseer, 1998. [Team et al., 2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [Topkara et al., 2006a] Mercan Topkara, Umut Topkara, and Mikhail J. Atallah. Words are not enough: sentence level natural language watermarking. In Proceedings of the 4th ACM International Workshop on Contents Protection and Security, MCPS ’06, page 37–46, New York, NY, USA, 2006. Association for Computing Machinery. [Topkara et al., 2006b] Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In Workshop on Multimedia & Security, 2006. [Tramèr et al., 2016] Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In Thorsten Holz and Stefan Savage, editors, 25th USENIX Security Symposium, pages 601–618. USENIX Association, 2016. [Vershynin, 2018] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018. [Weyl, 1912] Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgleichungen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathematische Annalen, 71(4):441–479, 1912. [Xu et al., 2024] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models, 2024. [Yang et al., 2023] Xi Yang, Kejiang Chen, Weiming Zhang, Chang Liu, Yuang Qi, Jie Zhang, Han Fang, and Nenghai Yu. Watermarking text generated by black-box language models. arXiv preprint arXiv:2305.08883, 2023. [Zhang et al., 2024] Ruisi Zhang, Shehzeen Samarah Hussain, Paarth Neekhara, and Farinaz Koushanfar. {REMARK-LLM}: A robust and efficient watermarking framework for generative large language models. In 33rd
USENIX Security Symposium (USENIX Security 24), pages 1813–1830, 2024. [Zhao et al., 2023] Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning, pages 42187–42199. PMLR, 2023. [Zhou et al., 2010] Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candes, and Yi Ma. Stable principal component pursuit. In 2010 IEEE international symposium on information theory, pages 1518–1522. IEEE, 2010.
Appendix A Related Work Model Stealing. We consider the setting of model stealing [Tramèr et al., 2016] (or model extraction) attacks, where an adversary who has black-box access to a model aims to recover information about the model’s internal architecture or parameters, or tries to recover the functionality of that model. Prior work has primarily studied in ReLU-based networks, [Milli et al., 2019] showed that gradient access enables near-exact model recovery, while [Jagielski et al., 2020] used finite differences for gradient approximation. Later efforts extended these attacks to deeper models [Carlini et al., 2020; Rolnick and Kording, 2020; Carlini et al., 2024a]. However, these methods do not scale to production language models due to token-based inputs, nonReLU activations, complex architectures. [Carlini et al., 2024b] proposes a method to extract the hidden latent dimensions and last-layer projection of commercial language models using API access, enabling model stealing even for large and expensive models. Watermarking. Broadly, watermarking is a form of embedding signature in a hidden way. It is categorized into formatbased [Rizzo et al., 2016; Por et al., 2012], lexical-based [Topkara et al., 2006b; Yang et al., 2023; Munyer et al., 2023], syntatic-based [Topkara et al., 2006a; Meral et al., 2009; Atallah et al., 2001], and generation-based [Lau et al., 2024; Zhang et al., 2024; Abdelnabi and Fritz, 2021] watermarking. Recently, [Kirchenbauer et al., 2023] propose a watermarking scheme where watermark is efficiently injected into the logits layer of a language model. [Panaitescu-Liess et al., 2025] find that watermarking can reduce the likelihood of generating copyrighted content. The most relevant to ours is [Zhao et al., 2023], which perturbs logits to detect model distillation. While similarly based on logit perturbation, our method targets a different threat model. Rather than detecting behavioral cloning after the fact, we proactively disrupt rank-based attacks to prevent extraction of architectural details, such as the hidden dimension. Rank estimation. The theory of matrix perturbation [Stewart, 1998] (see also [Konstantinides and Yao, 1988]) indicate that the smallest noise needed to reduce a full-column-rank matrix’s rank is precisely its smallest nonzero eigenvalue. In its simplest form, Weyl’s inequality ensures that the singular value decomposition (SVD) can reliably recover the numerical rank4 of a matrix under small perturbations. In particular, one may truncate at the r-th largest singular value when it exhibits a significant drop, ensuring accurate rank detection. On the other hand, if the noise’s magnitude becomes larger, one cannot rely on PCA to recover the exact rank. [Candès et al., 2011] suggests the robust PCA (RPCA) that exactly recovers the numerical rank if the noise is sparse but possibly gross by constructing a convex programming, dubbed principal component pursuit (PCP). [Zhou et al., 2010] present a variant of PCP which is simultaneously stable against small entry-wise noise and gross sparse error. Distillation. While not directly related to model stealing, knowledge distillation techniques have been widely studied as a method for transferring knowledge from large models to smaller ones. Distillation is typically used to improve efficiency, enabling smaller models to retain much of the performance of their larger counterparts while reducing computational costs. However, these techniques also pose risks to model security and intellectual property, as they can be used to replicate proprietary models without direct access to their architectures or training data. Several surveys, e.g. [Gou et al., 2021; Xu et al., 2024] provide an overview of distillation methods, covering strategies, evaluation metrics, and ethical considerations, along with their benefits and potential risks in AI development.
B
Omitted Details in Section 3.3 for RPCA
Pr We here provide the technical result for RPCA proven by [Candès et al., 2011]. Let A = U ΣV ⊤ = i=1 σi ui vi⊤ be the SVD of m × n dimensional matrix A with rank r. Then define the following two conditions parameterized by µ: µr µr 2 max U ⊤ ei ≤ , max V ⊤ ei ≤ (1) i i m n r µr , (2) UV ⊤ ∞ ≤ mn where ei has one in i-th coordinate and otherwise zero. Formally, [Candès et al., 2011] prove the following theorem: Theorem 3 ([Candès et al., 2011]). Let A be m×n matrix obeying (1) and (2). Let the support set, e.g., the nonzero elements, of the noise matrix E is uniformly distributed among all sets √ of cardinality S. Then, there is a constant c such that with probability at least 1 − c max(m, n)−10 , solving (PCP) with λ = 1/ m exactly returns the original matrix A and the noise E, provided that rank(A) ≤ ρr min(m, n)µ−1 (log max(m, n))−2 and S ≤ ρs mn, where ρr and ρs are absolute constants. They also conduct numerical experiments to validate that RPCA succeeds in recovering the exact rank up to the sparsity of 10% with gross errors. 4
See Section 3 for detailed definitions.
C
Experimental results for RPCA
We further tested our perturbed Mistral-7B instances against an extension of the SVD-based dimension-extraction attack, using the Robust PCA Principal Component Pursuit (PCP) procedure from [Candès et al., 2011]. Overall, our observation was that the RPCA process did not appear to significantly enhance the attack. In each of our tested experimental noise parameter settings, the standard PCA attack and the RPCA algorithm for rank estimate either both succeeded or both failed to extract the matrix rank. We provide an example comparison between SVD and RPCA resulting outputs in Figure 6 for the case of discrete noise of sparsity 0.5 and magnitude 2.0.
(a) SVD-based attack
(b) Robust PCA attack
Figure 6: (Mistral-7B RPCA). Comparison of SVD-based attack versus extended Robust PCA attack on Mistral-7B with discrete additive noise, for the case of magnitude 2.0 and sparsity 0.5. In particular, there is no significant distinction.
D
Further Experimental Results
D.1
PCA with averaging
Recall the standard SVD-based attack forms a single matrix A of size N × T from N = 10, 000 prompts. In the averaging attack, k such matrices A, . . . , Ak are independently generated, via k · N prompt queries, and then rank estimation is performed Pk on the average of the matrices, k1 i=1 Ai . We tested the effectiveness of the averaging attack up to k = 50 instances, focusing on the case of discrete noise with sparsity 0.5 and magnitude 2.0. Each matrix was generated in the same manner as the standard attack, using an independent random token pair for k · N total prompts. We also simulated the same experiment in a modified setting where every prompt is given independently sampled noise, as opposed to deterministically determined (via PRF) by the hidden layer as in our construction. This was recreated by first generating N noise-free logit vectors via random token pair prompts as above. Then, Ai was formed by adding independently sampled noise to each row. Our experiments indicate our (embedding-based) dependent noise withstood the averaging attack up through k = 50; see Figure 7. In contrast, the independent noise version appeared to break down around k = 40. We conjecture that this distinction comes from loss in signal precision within the dependent noise setting given k different matrices as opposed to k identical matrices as the starting point. We leave further investigation of this subject, including identifying if or when there is a breakdown point for the dependent noise setting, to future work.
D.2
Mistral-7B with additional noise parameters
We present experimental results for Mistral-7B with a range of noise types and parameters. In Figure 8 we give our findings for applying discrete noise with varying sparsity γ ∈ {0.1, 0.25, 0.5} and magnitude ∈ {0.1, 0.5, 1.0, 2.0}. In Figure 9, we provide the comparable plots for Mistral-7B with Gaussian noise.
Figure 7: Sorted singular values as a result of averaging attack for k = 50 instances on Mistral-7B with discrete noise magnitude b = 2.0, γ = 0.5. Smoothness of the curve indicates failure of the attack to extract hidden dimension.
Bias/Std dev Sparsity Perplexity
Original
Constant
Constant
Constant
Constant
Gaussian
Gaussian
19.34
1 0.25 20.59
2 0.1 40.16
2 0.5 38.8
3 0.1 66.72
1 0.25 31.63
1 0.5 39.85
Table 2: Perplexity results for GPT2. Original refers to the original GPT2 model, Constant denotes constant noise, and Gaussian refers to Gaussian noise added to the original model. Bias/Std dev denotes the magnitude of the noise bias for discrete noise and standard deviation for Gaussian noise. Sparsity denotes the sparsity parameter γ of the noise, and Perplexity is evaluated against Meta-Llama-3-8B as the larger oracle model.
We observe a clear trend where increased noise parameters lead to decreased success of the attack, in a gradual fashion. In particular, we see that noise magnitude (or standard deviation, in the case of Gaussian noise) plays a seemingly stronger role than sparsity.
D.3
Results for GPT-2
For completeness, we performed comparable measurements on GPT-2, with vocabulary size T = 50, 257 and hidden dimension 768. Experiments were run with N = 2, 300 prompts for each N × T logits matrix. Figure 10 provides the results for GPT-2 with discrete noise for varying sparsity γ ∈ {0.1, 0.25, 0.5} and magnitude ∈ {0.1, 0.25, 0.5, 1.0}. Figure 11 provides the comparable results for GPT-2 with Gaussian noise, for the same sparsity values, and standard deviations std ∈ {0.1, 0.25, 0.5, 1.0}. Finally, we perform an evaluation of the perplexity of GPT-2 with and without noise perturbation, again using Meta-Llama3-8B as the larger oracle model. The corresponding perplexity values are given in Table 2.
(a) Noise 0.1, Sparsity 0.1
(b) Noise 0.5, Sparsity 0.1
(c) Noise 1.0, Sparsity 0.1
(d) Noise 2.0, Sparsity 0.1
(e) Noise 0.1, Sparsity 0.25
(f) Noise 0.5, Sparsity 0.25
(g) Noise 1.0, Sparsity 0.25
(h) Noise 2.0, Sparsity 0.25
(i) Noise 0.1, Sparsity 0.5
(j) Noise 0.5, Sparsity 0.5
(k) Noise 1.0, Sparsity 0.5
(l) Noise 2.0, Sparsity 0.5
Figure 8: (Mistral-7B Discrete Noise). Dimension extraction attack on Mistral-7B by SVD singular values for discrete noise addition with varying sparsity (0.1, 0.25, 0.5) and magnitude (0.1, 0.5, 1.0, 2.0). Each top plot depicts the sorted singular values of the corresponding noisy logits matrix, where a jump at correct position 4,096 yields a successful dimension extraction. Each bottom plot provides a zoomed in picture of the difference between consecutive singular values; the attack outputs the index for which this difference is maximized.
(c) Standard Deviation (a) std = 0.1, γ = 0.1
(b) std = 0.25, γ = 0.1
(d) std = 0.5, γ = 0.1
(e) std = 1.0, γ = 0.1
(f) std = 0.1, γ = 0.25
(g) std = 0.25, γ = 0.25
(h) std = 0.5, γ = 0.25
(i) std = 1.0, γ = 0.25
(j) std = 0.1, γ = 0.5
(k) std = 0.25, γ = 0.5
(l) std = 0.5, γ = 0.5
(m) std = 1.0, γ = 0.5
Figure 9: (Mistral-7B Gaussian Noise). Dimension extraction attack on Mistral-7B by SVD singular values for Gaussian noise addition with varying sparsity γ ∈ {0.1, 0.25, 0.5} and standard deviation std ∈ {0.1, 0.25, 0.5, 1.0}. Each top plot depicts the sorted singular values of the corresponding noisy logits matrix, where a jump at correct position 4,096 yields a successful dimension extraction. Each bottom plot provides a zoomed in picture of the difference between consecutive singular values; the attack outputs the index for which this difference is maximized.
(a) Noise 0.1, Sparsity 0.1
(b) Noise 0.5, Sparsity 0.1
(c) Noise 1.0, Sparsity 0.1
(d) Noise 2.0, Sparsity 0.1
(e) Noise 0.1, Sparsity 0.25
(f) Noise 0.5, Sparsity 0.25
(g) Noise 1.0, Sparsity 0.25
(h) Noise 2.0, Sparsity 0.25
(i) Noise 0.1, Sparsity 0.5
(j) Noise 0.5, Sparsity 0.5
(k) Noise 1.0, Sparsity 0.5
(l) Noise 2.0, Sparsity 0.5
Figure 10: (GPT-2 Discrete Noise). Dimension extraction attack on GPT-2 by SVD singular values for discrete noise addition with varying sparsity (0.1, 0.25, 0.5) and magnitude (0.1, 0.5, 1.0, 2.0). Each top plot depicts the sorted singular values of the corresponding noisy logits matrix, where a jump at correct position 768 yields a successful dimension extraction. Each bottom plot provides a zoomed in picture of the difference between consecutive singular values; the attack outputs the index for which this difference is maximized.
(a) std = 0.1, γ = 0.1
(b) std = 0.25, γ = 0.1
(c) std = 0.5, γ = 0.1
(d) std = 1.0, γ = 0.1
(e) std = 0.1, γ = 0.25
(f) std = 0.25, γ = 0.25
(g) std = 0.5, γ = 0.25
(h) std = 1.0, γ = 0.25
(i) std = 0.1, γ = 0.5
(j) std = 0.25, γ = 0.5
(k) std = 0.5, γ = 0.5
(l) std = 1.0, γ = 0.5
Figure 11: (GPT-2 Gaussian Noise). Dimension extraction attack on GPT-2 by SVD singular values for Gaussian noise addition with varying sparsity γ ∈ {0.1, 0.25, 0.5} and standard deviation std ∈ {0.1, 0.25, 0.5, 1.0}. Each top plot depicts the sorted singular values of the corresponding noisy logits matrix, where a jump at correct position 768 yields a successful dimension extraction. Each bottom plot provides a zoomed in picture of the difference between consecutive singular values; the attack outputs the index for which this difference is maximized.