What Does It Mean to Break a Distillation Defense?
Lena Libon Pura Peetathawatchai Michael Aerni Daniel Paleka Florian Tramèr ETH Zurich
arXiv:2606.25059v1 [cs.CR] 23 Jun 2026
Abstract
become widely accessible via APIs, however, distillation has also become a vector for extracting proprietary capabilities without authorization (Anthropic, 2026a; OpenAI, 2026b). From the perspective of frontier model providers, APIs that return rich outputs, including reasoning traces, token probabilities, or detailed generations, represent a potential forfeiture of intellectual property (IP), allowing competitors to replicate a model’s capabilities without bearing the development costs. These concerns are reflected in provider terms of service and public communications (OpenAI, 2026a; Google Threat Intelligence Group, 2026).
Black-box LLMs (accessible only via API) are vulnerable to distillation attacks, in which an attacker queries the model and trains a student on its outputs. A recent line of work proposes output perturbation defenses that modify the teacher’s output to reduce student performance while preserving utility for legitimate users. As a relatively new family of approaches, output perturbation defenses lack a shared threat model, making it difficult to compare them, reason about composing them with other attacks, or evaluate their robustness against realistic adversaries. This underspecification matters beyond technical evaluation: when defenses are deployed to protect intellectual property or justify regulatory compliance, an imprecise threat model can create a false sense of security. We propose a threat model framework that describes attackers along three dimensions: a query budget, a data budget, and an interface profile that captures how attackers interact with the API. Using antidistillation sampling as a case study, we show that whether the defense is considered effective depends on the assumed threat model. We argue that future work on distillation defenses, along with any governance or policy frameworks built around them, should explicitly specify and stress-test attacker capabilities along our three dimensions.
Existing defenses operate at three levels. At the system level, providers implement rate limiting, coordinated activity monitoring, and legal enforcement through terms of service (Anthropic, 2026a; Google Threat Intelligence Group, 2026; Hulse et al., 2025). These mechanisms operate outside the model and are independent of individual outputs. At the detection level, statistical watermarks are embedded in outputs to enable post-hoc identification of distilled model content (Kirchenbauer et al., 2023; Xu et al., 2026). More recent output perturbation defenses (Jiang, 2026) modify model outputs to degrade student performance while preserving utility for legitimate users. Examples include fine-tuning the teacher (Li et al., 2025), rewriting reasoning traces after generation (Ding et al., 2025; Ma et al., 2026; Hartman et al., 2026), and corrupting the output distribution at inference time, either via a proxy model (Savani et al., 2025) or a learned logit transformation (Fang et al., 2026). Existing defenses at all three levels lack a unified threat model, making it difficult to compare them, reason about composition, or evaluate robustness against realistic adversaries. The problem is most acute for output perturbation defenses, where even the basic question of whether a defense works is undefined without specifying attacker capabilities. Existing work evaluates only against a simple implicit baseline: the attacker queries each prompt exactly once, collects outputs, and trains a student without further intervention (Savani et al., 2025; Fang et al., 2026; Li et al., 2025; Ding et al., 2025; Ma et al., 2026; Jiang, 2026; Hartman et al., 2026). This ignores how defenses perform against adversaries who allocate queries strategically, apply local post-processing, or interact through channels beyond the plain prompt.
1. Introduction Distillation is a process by which a student model replicates the capabilities of a more powerful teacher model by training on its outputs (Hinton et al., 2015; Taori et al., 2023; Peng et al., 2023), enabling smaller, cheaper models to achieve strong performance without the compute cost of training from scratch. As large language models have Correspondence to: Lena Libon <[email protected]>. Second Workshop on Technical AI Governance Research (TAIGR) @ ICML 2026, Seoul, South Korea. 2026. Copyright 2026 by the author(s).
1
What Does It Mean to Break a Distillation Defense? Query & Data Budget
Interface Profile
D at aQ ue
ry -
lim
ite d
+
0
∞
0
T HIS WORK
Query budget
lim
ite d
G ENERIC
∞ I NPUT CHANNELS
P ROCESSING
O UTPUT CHANNELS
Input
API
Response
e.g., prompt, prefill, sampling params
e.g., filtering, summarization
e.g., text, logprobs
Black-box Prompt
Prefill Prompt API
Text
Prefix
API
Text
Data budget Figure 1. Our proposed threat model space for evaluating distillation defenses. A complete threat model specifies a query budget, a data budget (left), and an interface profile (right). The interface profile is a collection of interface components spanning the input side (e.g. prefill), provider-side processing (e.g., output filtering, reasoning-trace summarization), and the output side (e.g., logprobs). In this paper we compare two interface profiles: a pure black-box API, and an API that permits response prefilling.
This pattern recurs throughout adversarial machine learning: defenses proposed and tested against fixed, non-adaptive attackers were later broken by stronger adaptive ones. Examples span adversarial-example defenses (Athalye et al., 2018; Tramer et al., 2020), protective perturbations for artists (Hönig et al., 2025), and LLM jailbreak and promptinjection defenses (Nasr et al., 2025). Defenses against distillation are at an analogous early stage. Yet the immediate obstacle is not only the absence of a stronger attack but the absence of any shared way to specify how strong an adversary is assumed to be.
defenses (e.g., watermarking), the setup is identical, except the attacker must also evade detection, while system-level defenses can be captured through budget constraints. We also argue that the lack of a shared threat model has consequences that extend beyond technical evaluation. As output perturbation defenses move from research into production (Kim, 2026), policymakers, legal teams, and IP holders (OpenAI, 2026b; Anthropic, 2026a) may rely on them as meaningful protection against distillation. But without an explicit threat model, neither deployers nor external auditors can know whether a defense that passed its evaluation would survive a more capable adversary. This ambiguity is not merely an academic limitation: it creates false assurances that may shape IP enforcement strategies, regulatory frameworks, and investment in better defenses. By substituting the appearance of protection for its substance and crowding out more robust alternatives, a deployed defense that overstates its coverage may ultimately set back IP protection efforts more than no defense at all.
Our contribution is a framework for reasoning about attacker strength that makes these dimensions explicit. We characterize threat models along two quantitative dimensions: the attacker’s query budget (the total number of API calls to the teacher) and data budget (the number of distinct input prompts available). Additionally, an interface profile consisting of multiple interface components captures the structure of the attacker’s interaction with the model: available input channels, provider-side processing, and exposed output channels. Together, a query budget, a data budget, and an interface profile fully specify a threat model. Figure 1 visualizes this space.
2. Background and Related Work 2.1. Knowledge Distillation
We use antidistillation sampling (ADS) (Savani et al., 2025) as a concrete case-study defense. ADS modifies the teacher’s sampling distribution at inference time to reduce the performance of a distilled student while preserving output utility for legitimate users. To study the role of the interface profile, we vary a single component: whether the attacker can inject a prefix into the model’s response. By evaluating ADS under these different threat models, we show how different assumptions can change conclusions about a defense’s effectiveness. Although we focus on ADS, the framework itself is defense-agnostic: for detection-level
Knowledge distillation is a model compression technique in which a smaller student model is trained to mimic the behavior of a larger teacher model (Hinton et al., 2015). Rather than training the student from scratch on raw data, the student learns from the teacher’s outputs, which carry richer information than hard labels alone. For instance, the teacher’s full output distribution over tokens encodes relative similarities between classes that a one-hot label discards (Hinton et al., 2015).
2
What Does It Mean to Break a Distillation Defense?
In the context of large language models, distillation typically takes the form of supervised fine-tuning on teachergenerated text: input prompts are collected, the teacher generates responses, and the student is trained on the resulting pairs (Taori et al., 2023; Peng et al., 2023). When the teacher produces explicit reasoning traces, these provide an especially rich training signal, since the student learns not just what answer to produce but how to decompose the problem (Savani et al., 2025). Models distilled from frontier reasoning teachers can achieve strong performance on math and coding benchmarks at a fraction of the training cost (DeepSeek-AI, 2025).
yond the defense itself, and only the output text is returned. Both the query and data budget are bounded by the size of the benchmark dataset, each prompt is queried exactly once, and the collected text is used to train a student via supervised fine-tuning without filtering or post-processing. ADS (Savani et al., 2025), DOGe (Li et al., 2025), PART (Ding et al., 2025), Ma et al. (2026), TraceGuard (Hartman et al., 2026), and the benchmark study DistillGuard (Jiang, 2026) all follow this pattern, differing only in the choice of benchmark, teacher model, and student model. Fang et al. (2026) extends the interface profile to expose logprobs but otherwise keeps the same setup. DistillGuard explicitly acknowledges that stronger adversaries could reduce defense effectiveness, but does not formalize the capabilities of such adversaries. TraceGuard asks a different question, whether the defense can be recognized rather than whether it can be broken, but the recognizer it considers still operates within the minimal interface profile.
2.2. Antidistillation Sampling We use antidistillation sampling (ADS) (Savani et al., 2025) as a running example throughout this paper. ADS is a decoding-time defense that perturbs the teacher model’s next-token distribution to reduce distillation effectiveness while preserving output utility. Given a prompt and partial generation, ADS samples from a modified distribution that up-weights tokens estimated to increase the student’s downstream loss, using a finite-difference approximation through a proxy student model. The perturbation strength λ ≥ 0 controls the trade-off between teacher utility and distillation degradation. At λ = 0, ADS reduces to standard temperature sampling. We focus on the high-utility regime (λ ≤ 0.107), where the teacher retains at least 70% accuracy on the GSM8K dataset (Cobbe et al., 2021). In this regime, ADS achieves approximately 40% student accuracy degradation for a 20% teacher accuracy drop, substantially outperforming temperature sampling. Further details about ADS are provided in Appendix A.
Conflating query and data budgets. Even within this minimal threat model, an attacker can allocate a fixed query budget strategically, querying some prompts multiple times at the cost of skipping others, or selectively targeting the most informative prompts. Separating the query budget from the data budget captures exactly this flexibility: the query budget bounds total API calls, the data budget bounds the pool of available prompts, and the allocation strategy between them is left to the adversary. This distinction is particularly important in the setting of output perturbation defenses, because the cost of acquiring new high-quality prompts to extract useful reasoning traces and the cost of additional API calls scale very differently. To our knowledge, this separation is not made in the broader LLM model extraction literature either, where the attacker is consistently characterized by a single query budget (Zhao et al., 2025) with each available prompt queried exactly once.
2.3. Output Perturbation Defenses in Deployed Systems Output perturbation defenses are deployed by several major providers in practice. A recent source leak of Anthropic’s Claude Code exposed two of them (Kim, 2026): The first causes the API to silently inject fabricated tool definitions into the system prompt of first-party CLI sessions. The second one summarizes the assistant’s inter-tool-call reasoning. A similar trace-summarization strategy is adopted by OpenAI, which exposes only model-generated summaries of o-series reasoning rather than the raw chain of thought (OpenAI Developers, 2026).
3. Threat Model Framework A defense against a distillation attack is only meaningful relative to a concrete attacker. We propose a framework (see Figure 1) that characterizes threat models as tuples (Q, D, P ) where Q ∈ N is a query budget, D ∈ N is a data budget, and P is an interface profile: a structured, qualitative description of how the attacker can observe and interact with the API.
2.4. Limitations of Existing Evaluations A shared, narrow evaluation pattern. Output perturbation distillation defenses have, to our knowledge, been evaluated under narrow attacker profiles. In our framework (cf. Section 3), most evaluations implicitly assume a minimal interface profile: the attacker sends a prompt through a single input channel, no provider-side processing is applied be-
3.1. Query and Data Budgets The distillation pipeline and our budget model are illustrated in Figure 2. Only teacher API queries and input prompts constitute the attacker’s cost. All other computation, such as student training, filtering low-quality generations, or post3
What Does It Mean to Break a Distillation Defense? API
prompts
responses
train
responses Teacher API
Data
Attacker
Distilled model
Attacker’s capabilities Counted toward attacker budget e.g. prompt (re-)querying
Free (local compute) e.g. post-processing, student training
Figure 2. Distillation attack pipeline and attacker budget model. The attacker queries the teacher API and collects responses to build a training dataset. Only teacher API queries and input prompts count toward the attacker’s budget (red); all local computation, including filtering, post-processing, and student training, is free (green).
processing outputs, is treated as free. This modeling choice reflects how API access is governed by the provider, whereas local computation scales with the attacker’s own resources.
answer with a chosen prefix, or a multi-turn conversational state that allows the attacker to build context across calls. For provider-side processing, the provider may apply pipeline steps between the post-defense model output and the response the attacker sees. Examples include safety and copyright filters that redact or replace portions of the output, truncation of responses that exceed a maximum length, and reasoning-trace summarization. Some of these steps can also be counted as distillation defenses. For instance, summarizing reasoning traces is used both for latency and to limit reasoning extraction. From a threat-modeling perspective, what matters is the transformation’s effect on the attacker rather than the provider’s intent.
Within this cost model, we distinguish two dimensions: (1) the query budget Q, the total number of API calls the attacker can make, and (2) the data budget D, the number of distinct input prompts available to the attacker. When Q > D, the attacker can obtain multiple generations per prompt, enabling strategies such as requerying prompts that produced incorrect outputs. When Q < D, the attacker cannot query every available prompt and must prioritize a subset of the data. They may query many prompts once, query a smaller subset multiple times, or allocate queries adaptively. In either case, the attacker is free to allocate queries unevenly between prompts. These separate budgets thus capture attacker strategies that a single dataset size parameter would conflate.
On the output side, the generated text is the baseline signal, but providers can additionally return token-level logprobs (over the full vocabulary or a top-k truncation) or intermediate tool calls.
3.2. Interface Profile
3.3. Mapping to Practice
The interface profile P captures the structure of the attacker’s interaction with the API, orthogonal to the number of queries or prompts used. We decompose P into three stages, illustrated in Figure 1 (right): the input channels available to the attacker, the provider-side processing applied between the model’s post-defense output and the response returned to the attacker, and the output channels through which information is exposed. Each stage is a set of interface components, and each component is a discrete design choice made independently by the provider. A defense that is robust against one interface profile may collapse once a single component is changed, so specifying P is essential for any claim about a defense’s effectiveness.
Each dimension of our framework corresponds to concrete constraints that attackers face and providers impose. In previously observed attacks, carefully designed prompts have been used to elicit high-quality responses targeting specific capabilities, or to create broad task distributions for training (Anthropic, 2026a). However, producing such prompts at scale is costly, so an attacker cannot simply acquire new prompts on demand. The data budget reflects this difficulty in obtaining suitable prompts. Meanwhile, the query budget is bounded by the attacker’s financial resources and the inference costs, by provider-deployed mechanisms at the system level (usage quotas, rate limits, behavioral fingerprinting, coordinated activity monitoring, and terms-of-service enforcement) (Anthropic, 2026a; Google Threat Intelligence Group, 2026; Hulse et al., 2025), as well as by the attacker’s capacity to circumvent them. For instance, OpenAI alleges that DeepSeek developed programmatic access patterns and routed queries
On the input side, the most basic channel is the prompt itself, but APIs commonly admit additional signals like a hidden or attacker-controlled system prompt, sampling parameters (e.g. temperature, top-p, top-k, repetition penalty, and stop sequences), a response prefill forcing the model to begin its 4
What Does It Mean to Break a Distillation Defense?
Experimental setup. Throughout this section, we fix the data budget to a 70% subset of GSM8K (Cobbe et al., 2021) (D = 5,231 samples), matching the original ADS evaluation (Savani et al., 2025). We adopt the same teacher (DeepSeek-R1-Distill-Qwen-7B), proxy student (Qwen2.53B), and student model (Llama-3.2-3B), and use the same training setup, including fine-tuning the student via LoRA with identical hyperparameters. Our threat model is therefore the tuple (Q, D, P ) with D = 5,231 fixed, Q varied across experiments, and P taking one of the two values: a restrictive black-box interface profile (text-only output, no additional input channels beyond the prompt, no providerside processing), or the same profile extended to admit a response prefill on the input side.
through obfuscated third-party routers to evade detection (OpenAI, 2026b). The interface profile is determined by the provider’s API and varies across companies and model generations. Input channels differ: response prefill is supported by some providers like DeepSeek (DeepSeek, 2026) and previously by earlier Claude models (Anthropic, 2026c) but was removed from Claude Opus 4.6 and Sonnet 4.6 onwards (Anthropic, 2026b). Provider-side processing also varies: OpenAI summarizes reasoning traces for its o-series models (OpenAI Developers, 2026), Anthropic does the same for Claude 4 (Vercel, 2026), and all major providers apply content filtering. Output channels have changed over time, with logprobs being the most visible example, since they were widely available early on, but have been restricted by many providers due to extraction risks (Carlini et al., 2024). Each threat model in our framework thus corresponds to a realistic deployment scenario defined by a specific provider’s API at a specific point in time.
Evaluation. We define the high-utility regime as the settings where the teacher retains at least 70% accuracy on GSM8K (λ ≤ 0.107), and evaluate five values λ ∈ {0.0553, 0.0632, 0.0711, 0.0868, 0.107} within this range. See Appendix B for lower-utility results. To measure student performance, we follow Savani et al. (2025) and use answer forcing (AF): after free-form generation from the prompt, we concatenate “\n\n**Final Answer**\n[\boxed{” and generate 32 additional answer tokens. Appendix C provides examples of raw and ADS-attacked traces.
4. Case Study: Antidistillation Sampling To demonstrate the value of our framework, we present a case study on antidistillation sampling (ADS) that shows how varying the assumed threat model can change conclusions about a defense’s effectiveness. For direct comparison with Savani et al. (2025), we match their dataset, models, and training setup throughout.
4.1. Weak-Attacker Model
Our goal is not to find the strongest possible attack against ADS, but to show that even small, well-motivated changes to the threat model are sufficient to reverse conclusions about whether a defense works. This is precisely the risk that an underspecified threat model creates. Rather than sweeping the full threat model space, which would require a separate study for each combination of interface components, budget regimes, and student architectures, we therefore vary a small number of dimensions while holding others fixed. We fix every interface component except one: whether the attacker has prefill access.
We begin with the weakest attacker: black-box interface profile, with query budget equal to data budget (Q = D = 5,231). This is the threat model under which ADS was originally evaluated. Even here, since our framework treats any computation the attacker performs on collected outputs as free, a realistic attacker has access to free local postprocessing that the defense cannot prevent. ADS outputs often contain repetitive passages that are straightforward to detect and remove. Deleting these across all collected traces before training requires no additional queries and no knowledge of the defense mechanism. As shown by the “No Resampling” points of Figure 3, this single step already lifts student accuracy significantly above the originally reported baseline across the high-utility regime. Because this post-processing is within the capabilities of any attacker operating under ADS’s own assumptions, the utility gap reported by Savani et al. (2025) overestimates the defense’s effectiveness even in the original threat model.
We focus on prefill for two reasons. First, it is a design choice that some commercial APIs expose while others have explicitly removed. Second, unlike richer output channels such as logprobs, which would require changing the student training procedure to use soft targets, prefill can be studied within the exact evaluation setup of the ADS paper. This isolates the effect of a single interface component and shows that conclusions about defense effectiveness can depend on choices often treated as implementation details.
We additionally test query reallocation, a second strategy available within the same interface profile. With a fixed query budget, an attacker can requery prompts that produced incorrect answers rather than querying every prompt exactly once. We experiment with max-k resampling: a prompt that produces an incorrect answer is requeried up to k additional
We conclude the section with a discussion of threat model variations we do not instantiate.
5
What Does It Mean to Break a Distillation Defense?
Q = D = 5231; Black