arXiv CSopen access

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

arXiv:2604.18179v1 [cs.CR] 20 Apr 2026

Ziyang Liu Independent Researcher [email protected]

Abstract Hosted-LLM providers have a silent-substitution incentive: advertise a stronger model while serving cheaper replies. Probe-after-return schemes such as SVIP leave a parallel-serve side-channel, since a dishonest provider can route the verifier’s probe to the advertised model while serving ordinary users from a substitute. We propose a commit-open protocol that closes this gap. Before any opening request, the provider commits via a Merkle tree to a per-position sparse-autoencoder (SAE) feature-trace sketch of its served output at a published probe layer. A verifier opens random positions, scores them against a public named-circuit probe library calibrated with cross-backend noise, and decides with a fixed-threshold joint-consistency z-score rule. We instantiate the protocol on three backbones—Qwen3-1.7B, Gemma-2-2B, and a 4.5× scale-up to Gemma-2-9B with a 131k-feature SAE. Of 17 attackers spanning same-family lifts, cross-family substitutes, and rank-≤ 128 adaptive LoRA, all are rejected at a shared, scale-stable threshold; the same attackers all evade a matched SVIP-style parallel-serve baseline. A white-box end-to-end attack that backpropagates through the frozen SAE encoder does not close the margin, and a feature-forgery attacker that never runs M is bounded in closed form by an intrinsicdimension argument. Commitment adds ≤ 2.1% to forward-only wall-clock at batch 32.

1

Introduction

A growing share of large-language-model (LLM) inference is delivered through opaque hosted APIs, and the provider’s quality-for-cost substitution of served models is both profitable and difficult to detect: a user who pays for access to a 70B-parameter model cannot, from responses alone, distinguish it from a 7B substitute that happens to answer the specific queries the user sent. Three lines of work partially address this gap: determinism / verify-rollback re-runs inference bitwise but does not establish identity; proof-of-inference SNARKs [1] give soundness at 103 –104 × overhead; response-side fingerprints (LLM MAP [2], instructional fingerprinting [3]) lack a binding primitive. The closest prior binding primitive is SVIP [4], a secret linear projection of a hidden state returned on demand; because the projection is requested after service, SVIP admits a trivial parallel-serve side-channel — the provider serves users from M ′ while answering each verifier query from the advertised M . Our contribution. We study a commit-open protocol in which the provider commits, via a Merkle tree, to the per-position SAE-feature trace sketch of its served sequence at a published probe layer, before any verifier opening request. Verification checks random openings against a public namedcircuit probe library, calibrated with cross-backend noise, and decides with a fixed-threshold jointconsistency z-score. Within the scoped threat model (Appendix A), a dishonest provider faces three options: (a) commit from the substitute and be statistically detected, (b) run M to produce the Preprint.

Figure 1: Commit-open verification. Left: four provider strategies — (A) honest, (B) substitute + substitute’s trace (detected), (C) substitute + M -trace (pays honest compute), (D) adaptive φ (bounded by SAE capacity). Right: the provider commits a Merkle root of per-position SAE traces before the verifier’s probe, closing SVIP’s after-the-fact side-channel (dashed). commitment, which on our serving stack costs within ∼ 3% of honest wall-clock, or (c) adapt the substitute within a public adaptation budget. We characterize (c) empirically across a weak-to-strong attacker ladder and pair it with a white-box diagnostic that backpropagates through the frozen SAE encoder to the scoring rule. Paper map. Section 3 fixes the commit-open protocol. Section 4 states the strategy dichotomy (detected, or pays honest compute). Section 5 reports the main 17-attacker frontier across three backbones, the SVIP-gap comparison, the white-box diagnostic, and the feature-forgery bound. Section 6 scopes the claims. Evidence preview. We evaluate three attacker tiers. Weak non-adaptive attackers — three samefamily lifts and four cross-family substitutes combined with a public-corpus linear alignment φ [5] — all clear the deployment threshold by ≥ 12× on Qwen3-1.7B. Strong black-box adaptive attackers built on top of a library-aware joint-φ plus LoRA up to r=128 (1500–3000 steps) are detected with at least a 2.78× margin on Qwen3-1.7B, and this margin grows with SAE width, reaching 40−260× on Gemma-2-9B + Gemma-Scope-9B (L20, width-131k). A white-box end-to-end attacker that backpropagates through the frozen SAE encoder to the scoring rule does not close the blackbox margin. The detection threshold is scale-stable (τ9B =τGemma =1.09, agreement < 10−4 ), and commitment overhead is ≤ 2.1% at batch 32 with a 224-byte per-request payload. Contributions. 1. Protocol. The first hosted-LLM verification scheme that binds the provider to a per-position SAE-feature trace via a pre-query Merkle commitment (Section 3). 2. Strategy dichotomy. Every non-honest provider strategy we evaluate is either statistically detected at the deployment threshold or forced to run M at near-honest wall-clock (Section 4). 3. Empirical scale. A single 96-probe library detects every evaluated lifted, cross-family, and rank≤ 128 adaptive attacker across Qwen3-1.7B, Gemma-2-2B, and Gemma-2-9B, with a scale-stable threshold and detection margins that grow with SAE width (Section 5). Together these close an 11/11 gap vs. a matched SVIP baseline (Section 5.6). 4. Systems cost. Batched commit overhead drops from 18.5% at B=1 to ≤ 2.1% at B=32, with a 224-byte per-request payload (Section 5, Appendix J).

2

Related Work

Hosted-LLM identity and integrity. SVIP [4] returns a secret linear projection of a target-layer hidden state on demand, but the projection is evaluated after service and therefore does not bind the served computation to the identity claim — Section 5.6 quantifies this gap under matched threshold and attacker set. TOPLOC [6] uses an LSH of activations for integrity (the advertised model produced the served tokens) and is composable with our scheme rather than competing. LLM MAP [2] and 2

Table 1: Comparison of hosted-LLM verification schemes. “Binding” means the provider is committed before the verifier’s query, closing the parallel-serve side-channel. “SC-safe” denotes resistance to a provider that serves M ′ to users while answering the probe from M . ∗ “stat. (scoped)” is an empirical statistical detection result on the attackers we tested within Table 6, on three backbone–SAE pairs spanning 1.7–9B parameters (Qwen3-1.7B + transcoder L14; Gemma-2-2B + Gemma-Scope L12; Gemma-2-9B + Gemma-Scope-9B L20 width-131k); not a cryptographic or backbone-agnostic claim. The parallel-commit cost gap is a single-deployment ∼ 3% wall-clock observation.

Scheme

Primitive

Binding

SC-safe

Soundness

LLM MAP [2] response hash – – stat. Instruct. fp. [3] planted trigger – – stat. SVIP [4] hidden-state probe after query no stat. TOPLOC [6] LSH (integrity) at commit yes (int.) stat. (int.) CP-SNARK [1] proof of exec. at commit yes cryptographic Ours SAE-feature trace at commit yes (by constr.) stat. (scoped)∗

Overhead post-hoc post-hoc ∼ 0% ∼ 8% 103–4 × ≤ 2.1%

instructional fingerprinting [3] recover identity post-hoc from response statistics or planted triggers with no binding primitive. Gross et al. [7] use mechanistic interpretability for offline behaviour proofs; we borrow the move of treating MI artifacts as cryptographic payloads but apply it at serving time and target identity. Mechanistic interpretability and commit-and-prove. Our probe library is assembled from namedcircuit work: indirect-object identification [8], induction [9], and SAE-backed syntactic/factual circuits [10–13]. We are the first to use the feature trace as the commitment payload of a verification scheme. Commit-and-prove SNARKs [1] give cryptographic soundness at 103 –104 × runtime; proofof-useful-work [14] relaxes soundness for compute-as-proof. We occupy an intermediate regime: the verifier does not prove honest execution, but any non-honest strategy is either detected or pays honest-service compute (Section 4). Hardware-rooted primitives (Intel TDX [15], NVIDIA H100 confidential computing [16], IMA [17], append-only logs [18, 19]) are orthogonal deployments that strengthen the Merkle commitment against privileged adversaries rather than substitutes for the SAE-trace primitive. Table 1 summarises the differentiation. Concurrent work on auditable inference / fine-tuning. Two recent systems bracket our design. AFTUNE [20] records auditable execution traces during fine-tuning and inference and spot-checks them inside a TEE; overhead is 14–83%, and the trust primitive is recomputation rather than a statistical-structural binding. VFT [21] gives zero-knowledge proofs that a released model derives from a declared training process (16.8 s/step prover, 4–6 MB proofs); its scope is training provenance, not per-request serving. We sit between them: an inference-time, sub-millisecond commitment that does not require TEE hardware, with the binding carried by the per-position SAE-trace sketch and the probe library rather than by re-execution or ZK circuits. AFTUNE is March 2026 (concurrent with this submission); both are reported here to position scope, not as baselines (Section 6).

3

Protocol

Table 2 fixes the concrete deployment values. The rest of this section gives the definitions and the commit-open protocol. Setting. The provider advertises a model M and serves input x by returning tokens y1:T together with a verification bundle. At a published probe layer L, M ’s residual-stream input to the MLP admits an SAE or transcoder [10, 11, 13] with encoder E. For each position t, let ft = E(ht ) ∈ RdSAE be the SAE feature vector. The committed trace sketch is the top-k support together with its values, tok32t = {(j, ft,j ) : j ∈ top-k(ft )} with k=32, quantised to bf16. Merkle commitment.

The provider builds a Merkle tree [22] over leaves
leaf t = H "LEAF" ∥ meta ∥ t ∥ tok32t , 3

(1)

Table 2: Protocol parameters. Commit, calibration, and verification parameters. Values given as “Qwen3-1.7B / Gemma-2-2B” where they differ. Merkle leaf = (x, y, mid, t, tok32t ); payload is the root plus the top-k indices and bf16 values per opened position. A Gemma-2-9B + Gemma-Scope9B scale-up (Section 5.5) reuses the same |P|=96 library at L=20, width-131k, with τ9B =1.09 (agreement with τGemma within 10−4 ). Parameter

Symbol

Value (Qwen3-1.7B / Gemma-2-2B)

Probe layer SAE/transcoder

L –

Top-k per position Quantisation Probe library size Opening count per session Honest pool

k – |P| kopen nhon

σi calibration grid

–

Conservative threshold Nominal FPR target Joint-z aggregator

τ α z(t)

Merkle hash

–

14 / 12 (published, fixed) mwhanna-qwen3-1.7b-transcoders / Gemma-Scope canonical residual SAE 32 bf16 96 (8 circuit classes) random subset, nominal k∈{4, . . . }; within-session ρ=0.88 112 (MATH n=64 + multi-kernel n=48) / 64; seeds {100..107, 300..302} Qwen3 2 dtypes × 3 kernels × 4 pos., seeds {0, 1, 2, 3} / 2 dtypes × 2 kernels × 4 pos. × 2 seeds τpool =1.509 / τGemma =1.09 (pooled empirical max) 0.01; CP upper 0-viol 0.0264 / 0.0457 mean of per-probe zi (t) over random subset I(t), |I(t)|=N ; Equation (2) SHA-256, 32-byte root; 224 bytes total per request at |I(t)|=N

Figure 2: Commit-open mechanics. (1) The provider forwards (x, y) through M and encodes the layer-L residual stream ht with a public SAE encoder E, keeping the top-32 active features per position. (2) Per-position leaves leaf t = H(meta, t, tok32t ) are assembled into a Merkle tree with root R; the provider publishes (x, y, R). (3) The verifier samples random positions V and requests Merkle openings, verifying each path against R to recover tok32t . (4) A public probe library P ( 96 named circuits ) scores each opened position with zi (t) and aggregates to z(t) = meani zi (t); the session is rejected if z(t) > τ . Crucially, R is published before the verifier’s opening request, closing the SVIP parallel-serve side-channel.

with meta = (model_id, SAE_release, L, H(x), H(y), nonce, provider_pubkey), and publishes the root R alongside (x, y). Binding meta to H(x), H(y) is a pre-commitment of the trace to the served session: the root fixes which trace will be scored before any opening request, so the provider cannot re-route the opening (the SVIP side-channel). The commitment does not cryptographically prove the opened tok32t was computed on (x, y); the residual gap is closed statistically by the economic argument of Section 4 or cryptographically by pairing with TEE attestation [15–17]. Probe library and calibration. The library P = {(namei , xi , µi , σi )}96 i=1 covers 8 circuit classes (IOI, induction, syntactic, factual, coreference, arithmetic, commonsense, language), populated from the named-circuit literature with a robust-mix construction (retaining top-32 features also active under data-augmentation shuffle, 39.6% acceptance rate). Each probe is calibrated under cross-backend noise (dtypes × kernels × positions); honest p50 / p95 L2 drift is 2.1% / 6.5% (CV median 1.9%), and per-probe σi absorbs it. For each probe we record a per-feature reference µi ∈ RdSAE (honest 4

feature vector on the probe prompt, averaged over calibration draws) and per-feature noise scale σi,j >0; support Si = top-k(µi ) (size 32). Verifier protocol. On receiving (x, y, R), the verifier (1) samples positions V ⊆ {1, . . . , T } with N ≤ 96 probes per position and requests Merkle openings for {leaf t : t ∈ V }; (2) verifies each opening against R and recovers tok32t ; (3) for each probe i ∈ I(t), reads fbt,j for j ∈ Si (or 0 if P j∈ / top-k(ft )) and computes zi (t) = |S1i | j∈Si |fbt,j − µi,j |/σi,j , with joint aggregate X z(t) = N1 zi (t); (2) i∈I(t)

(4) decides with z(t) > τ , where τ is the empirical max over a pooled honest draw (τpool =1.509 deployment value, CP 95% upper 0.0264; Section 5.1). Per-session FPR under k openings is bounded by kα under independence; within-session dependence (ρ=0.88) empirically deflates it (Appendix G). A sequential SPRT [23] extension at α=β=0.01 with Holm step-down αn =α/(N −n+1) [24] is compatible but reported only diagnostically (Appendix H). Budget. Provider-side overhead is one SAE encode plus Merkle root construction over T 32-entry leaves. On our deployment, amortised cost at batch 32 is ≤ 2.1% (Appendix J).

4

Strategy-by-Strategy Analysis

We analyse the four provider strategies below by combining a binding argument for the commitment with empirical results from Section 5. Conclusions are scoped to Table 6, the attackers we tested, and our single-stack serving deployment; we avoid game-theoretic dominance language — strategy (C) is only 0.97× honest wall-clock on one stack, a deployment-specific cost gap, not a dominance result. (A) honest: serve y from M , commit its SAE trace. (B) substitute-serve: serve y from M ′ , commit M ′ ’s trace. (C) parallel-commit: serve y from M ′ but commit from a parallel M -pass, so openings are consistent with M . (D) adaptive distillation: fine-tune M ′ under a bounded budget (two evaluated points: publiccorpus φ + r=16 LoRA to 3000 steps; library-aware joint-φ + r=64 LoRA at 1500 steps) to match M ’s trace, then execute (B). Proposition 4.1 (Strategy-by-strategy detection/cost dichotomy). Under Table 6, with |P|=96 and per-feature noise σi,j calibrated on honest cross-backend draws, the strategies we test exhibit the following behaviour relative to the conservative deployment threshold τ of Equation (2): • (B) Detected at TPR = 1.00 at τpool =1.509 (nhon =112, CP 95% upper 0.0264) on 3 lift and 4 cross-family attackers (Sections 5.2 and 5.3). • (C) leaf t binds ft to the committed M , so opening requires an M forward pass on the committed positions; on our stack parallel-commit is 0.97× honest wall-clock. This ∼ 3% gap is single-stack and ignores parallel hardware / KV-cache sharing — not a dominance claim. • (D) At two evaluated black-box budgets — (i) public-corpus φ with ∥φ∥F =0.59 plus r=16 LoRA to 3000 steps, where held-out per-category joint-z ≥ 12 × τpool ; and (ii) library-aware joint-φ + r=64 LoRA, whose strongest Qwen3 point (S TAGE A, λutil =0) reaches z=4.19 (2.78 × τpool ) at 1.55× Pile perplexity, and whose matched Gemma pure-probe point reaches z=105.9 (97.2 × τGemma ) at 436× perplexity (Table 4) — all 8 evaluated adaptive operating points are detected. A supplementary white-box E-E diagnostic (r=64, 3000 steps, gradient through frozen SAE ReLU) does not close the margin: 3 restart seeds of the strongest Qwen3 cell give z ∈ {4.99, 6.89, 7.80}, the best-for-attacker restart 4.99 still exceeding the black-box MSE optimum 4.19 (Section 5.4, Appendix M). The binding argument for (B) and (C) is a direct consequence of the Merkle pre-commitment (Equation (1)), which fixes the committed trace before the verifier issues any opening request; the detection claim uses the per-probe calibration of σi on honest draws and the fixed-threshold rule z(t)>τ . For (D), the heuristic explanation is that under a public-corpus minimum-norm fit 5

∥φ∥F =0.59 projects LoRA-induced ∥∆h∥∞ ∼ 1 to ∥∆proj∥∞ ≤ 10−2 , below the top-32 mask-flip threshold (Appendix U). The full proof sketch (including the tail-robustness argument for τ and session-FPR under dependence) is in Appendix B. Not claimed: detection of full-model retraining, resistance to white-box probe-aware gradient attackers beyond the evaluated E-E point, economic dominance, or generalisation beyond the two backbone–SAE pairs evaluated (Section 6). Feature-forgery infeasibility. A complementary attacker does not run M at all and fabricates tok32t directly against the published library. Because joint-z is a mean over probe-slot penalties separable in the chosen feature P set, the attacker’s optimum is closed form: for each feature f compute the gain G∗ (f )= maxv (i,s):Si,s =f [|µi,s |−|v−µi,s |]/σi,s , which is attained at the weighted-median v ∗ , and pick the top-32 features by G∗ . This bounds the gain by the coverage budget, giving Proposition 4.2. S Proposition 4.2 (Intrinsic-dimension lower bound). Let U =| i Si | and c=mediani,s |µi,s |/σi,s . For any attacker committing a per-position top-k without running M , the uniform-coverage lower bound is
z(t) ≥ 1 − k/U c, i.e. at most a fraction k/U of probe-slots can be covered. On the deployed library (U =1472, k=32, c=22.78) the data-dependent greedy refinement (Section V) tightens this to z(t)≥37.11=24.6τpool , and the matching exact closed-form discrete + bf16 optimum is z ∗ =44.24. See Section V for the full derivation and Section 5.7 for the empirical ladder. Proposition 4.2 closes a gap left by Proposition 4.1: the parallel-commit strategy (C) had only an economic binding (0.97× honest wall-clock, single stack), and (D) had only evaluated LoRA / white-box points. Forgery — the simplest strategy requiring no M access — is now bounded by a structural property of the probe library, not by an attacker budget.

5

Experiments

5.1

Setup

Target stack. Qwen3-1.7B [25] at layer L=14, public mwhanna-qwen3-1.7btranscoders-lowl0 SAE; 96-probe library assembled from 8 circuit classes via a robustmix construction (39.6% acceptance). Gemma-2-2B [26] + Gemma-Scope canonical residual SAE at L12 is a second-backbone replication (Appendix E), and Gemma-2-9B + Gemma-Scope-9B canonical residual SAE at L20 (width-131k, avg-L0 =114) is a 4.5× scale-up (Section 5.5, Appendix F) that reuses the same 96-probe prompt set (top-32 feature IDs and µ/σ recomputed on 9B activations). Attackers. Three threat levels: (i) lifted projection on M simulating a same-family substitute (M ′ =Qwen3-0.6B via ZERO_PAD, RAND_PROJ, LEARNED_LIN); (ii) cross-family substitutes Qwen2.5-1.5B [27], Phi-3.5-mini [28], OLMo-2-7B [29], Qwen3-0.6B with a public-corpus linear alignment φ fit on 2000 Pile-uncopyrighted pairs [30] (80/20 split); (iii) a rank-constrained LoRA attacker on top of φ (r=16, α=32, seven projection modules, 500+2500 steps) and a library-aware higher-budget variant (r=64, 1500 steps, joint φ). Calibration. Headline statistic is the joint-z of Equation (2); legacy per-probe Mahalanobis [31] numbers are labelled. σi is calibrated on 2 dtypes × 3 kernels × 4 positions (companion seeds {0, 1, 2, 3}). The original n=64 MATH honest pool gives τreal =1.13; a follow-up multi-kernel spot check on Modal L4 (Appendix T) adds 48 fresh draws at companion seeds 300−302. Pooling all n=112 honest draws gives the conservative deployment threshold τpool =1.509 (0/112, CP one-sided upper 0.0264), adopted throughout. Qwen3 compute ≈ 0.8 L4-hours + 2.5 3090-hours; commit overhead ≤ 2.1% at batch 32 (Appendix J). Table 3 reports the strongest tested attacker per tier on each backbone; the full 26-row per-attacker breakdown (lift, cross-family, rank-≤ 128 LoRA, white-box E-E) is Table 8 (Appendix D). Subsections 5.2–5.6 analyse each tier in turn. 6

Table 3: Headline detection frontier. The strongest (lowest joint-z) attacker we could produce in each tier, on each of the three backbones. All are rejected at the backbone-specific conservative threshold τ ; the same attackers all evade a matched SVIP-style parallel-serve baseline (Section 5.6). Full per-attacker breakdown: Table 8 (Appendix D). Backbone / SAE (τ )

Strongest tested attacker

joint-z

×τ

SVIP

Qwen3-1.7B / transcoder L14 (1.509)

adaptive LoRA r=64 adaptive LoRA r=128 white-box E-E (r=64)

4.19 6.42 4.99

2.78 4.25 3.31

miss – –

Gemma-2-2B / Gemma-Scope L12 (1.09)

adaptive LoRA r=64 white-box E-E (r=64)

105.89 133.86

97.2 122.8

miss –

Gemma-2-9B / Gemma-Scope-9B L20, 131k (1.09)

cross-family Qwen2.5-7B adaptive LoRA r=64

283.30 43.70

260.0 40.1

– –

ROC (FPR ≤ 12%)

Score distribution

101

τ = 1.51

100

0.8

rand proj

zero

pad

oj d pr

ran

in

ed l

n lear

learned lin

0.4 0.2 0.0

est

hon

zero pad

0.6

FPR=1%

102

True positive rate (attacker)

Joint Mahalanobis score

1.0

0.00

0.02

0.04

0.06

0.08

0.10

0.12

False positive rate (honest)

Figure 3: E2 same-family separability (Qwen3-1.7B vs. lifted Qwen3-0.6B), on the legacy permaha probe Mahalanobis diagnostic scale (threshold τQwen =1.51, dashed in panel (a)). (a) Per-sample Mahalanobis under 16 honest and 3 × 48 attacker samples; all three attackers sit orders of magnitude above the honest envelope. (b) ROC derived from the same scores, zoomed to nominal FPR ≤ 12%. The headline joint-z rule at τpool =1.509 (nhon =112) reaches TPR = 1 on all three attackers with empirical FPR 0/112 (CP 95% upper 0.0264).

5.2

Same-family separability

Under the lifted-projection threat at τpool =1.509, all three attackers reach TPR = 1 (Clopper–Pearson 95% CI [32] [0.926, 1.0] on natt =48 replays) with no fresh honest sample above τpool (Figure 3). On the legacy per-probe Mahalanobis scale the three attackers reach 56–148× the honest hold-out max of 1.51; the overlap-within-2σ statistic on LEARNED_LIN drops from 0.876 (honest) to 0.19. 5.3

Cross-family detection

All four cross-family substitutes are rejected at τpool =1.509 (row 4–7 of Table 8): joint-z ∈ {69.2, 52.4, 81.3, 62.8} for Qwen2.5-1.5B, Phi-3.5-mini, OLMo-2-7B, Qwen3-0.6B — all at TPR=1 with CP 95% CI [0.926, 1.00] on natt =48, ratio 34.7–53.9 × τpool . Per-probe Mahamaha lanobis is 52–85 × τQwen . The φ-transfer R2 diagnostic (held-out R2 ∈ {0.39, −0.90, −5.97, 0.15}) is reported only as partial explanatory analysis (Appendix K); detection is established independently by the joint-z scores. A scale-matched cross-family point on Gemma-2-9B (Qwen2.5-7B through a public-corpus φ) lands at joint-z=283.3 (260 × τ9B ; row 24 of Table 8, analysed in Section 5.5). 5.4

Adaptive LoRA attackers

We probe two budget points on M ′ =Qwen3-0.6B, targeting the seven projection modules {q, k, v, o, gate, up, down}-proj, preceded by a learned linear lift φ : R1024 → R2048 . 7

Table 4: E4-v2 library-aware adaptive LoRA (M ′ =Qwen3-0.6B / Gemma-2-2B-it, r=64, joint φ, 1500 steps, full 96-probe access). All operating points detected at the backbone-specific conservative threshold. λutil

joint-z

×τpool

×τreal

Pile ppx

Qwen3-1.7B; τpool =1.509 baseline-φ-only – S TAGE A pure-probe 0 S TAGE C low-util 0.1 S TAGE B balanced 1.0

57.38 4.19 6.27 7.48

38.0 2.78 4.16 4.96

50.8 3.71 5.55 6.62

21.20 32.84 63.16 50.98

188.6 × τGemma 97.2 × τGemma

18.44 8031.65

Operating point

Gemma-2-2B; τGemma =1.09 baseline-φ-only – 205.56 S TAGE A pure-probe 0 105.89

Constrained (r=16, public-corpus φ). 3000 steps, φ frozen at the public-corpus fit (∥φ∥F =0.59, 2 Rho =0.15). ∥B∥F grows 0→5.24; per-category held-out joint-z stays ≥ 12 × τpool . The bottleneck is structural: ∥∆h∥∞ ∼1 projects through the small-norm φ into ∥∆proj∥∞ ≤10−2 , below the top-32 mask-flip threshold (Appendix U). Library-aware higher-budget (r=64, joint φ). The attacker has full read access to the 96-probe library and trains φ jointly with the LoRA under L = λprobe MSE(φ(hM ′ ), hM ) + λutil NLLM ′ (Pile) (λprobe =1, 1500 steps, AdamW lr 3×10−4 ). All four Qwen3 operating points (Table 4, Figure 4) are detected. S TAGE A (λutil =0) reaches joint-z=4.19 (2.78 × τpool ) at 1.55× ppx inflation; per-category minimum is z=2.60 (INDUCTION). Utility regularisation (λutil ∈{0.1, 1.0}) raises z to 6.27–7.48 at 2.4–3.0× ppx. Compute: 1451 s on Modal L4 (≈ $0.32). A matched Gemma pure-probe point (λutil =0) reaches joint-z=105.9 (97.2 × τGemma ) while Pile ppx collapses to 8032 (436× clean) — detectable with catastrophic utility loss; we do not claim a matched utility-preserving Gemma frontier. A further scale-up point on Gemma-2-9B (same r=64 S TAGE A recipe, Gemma-2-2B-it substitute) lands at joint-z=43.7 with 30× ppx inflation (row 26, Section 5.5) — detection margin is ∼ 2.4× tighter than at 2B in z but carried by an 8× wider SAE. A higher-rank Qwen3 frontier point (r=128, MSE, λutil =0) still clears detection at joint-z=6.42 (4.25 × τpool ) at 2.4× ppx (row 19a), confirming that doubling attacker rank does not reopen the scoring-rule margin within the evaluated training budget. White-box E-E diagnostic. We additionally give the attacker gradient access through the frozen SAE encoder to the scoring rule itself, adding a joint-z loss Ljz =αjz meanp,i (zp,i,k −µp,i,k )/σp,i,k over the top-32 mask (full 6-point Qwen3 grid + 2 Gemma points in Appendix M). The strongest Qwen3 cell (αjz =0, λutil =0), repeated across three restart seeds, gives joint-z ∈ {6.89, 7.80, 4.99} — the best-for-attacker restart 4.99 still exceeds the black-box MSE optimum 4.19; two of the remaining five single-run cells diverge (ppx>107 ). The mechanism is SAE ReLU sparsity: inactive top-k features contribute zero gradient, so ∂Ljz /∂LoRA is a noisy sparse subset that fights the dense MSE. Higher-rank, longer-trained, and secret-probe-aware white-box attacks remain out of scope (Section 6). 5.5

Scale-up: Gemma-2-9B + Gemma-Scope-9B

To test whether the protocol extends beyond the sub-3B regime — the “70B advertised / 7B served” motivation requires this scale point — we replicate the three-stage pipeline (honest-pool → crossfamily detection → adaptive LoRA) on Gemma-2-9B at L20 using the canonical 131k-width GemmaScope-9B SAE (avg-L0 =114). The 96 probe prompts carry over verbatim; top-32 feature IDs and µ/σ are recomputed on 9B activations (Appendix F). Honest threshold is scale-stable. A 64-draw honest pool (2 dtypes × 2 kernels × 4 positions × 4 seed groups) yields τ9B =1.090, agreeing with τGemma =1.090 to within 10−4 and within an order of magnitude of τpool =1.509 on Qwen3. Across three backbones spanning 1.7–9B the pooled honest joint-z stays in [1.09, 1.51]; there is no evidence of threshold drift as model and SAE scale up. 8

Detection statistic z on held-out probes

Rank-64 LoRA frontier 10

2

φ-only baseline 69.9 × τ stage B (balanced) 9.1 × τ

101

stage A (pure-probe) 5.1 × τ — strongest

stage C (low-util) 7.6 × τ

100 τ = 0.82 (honest 99th pct)

1.0

1.5

2.0

2.5

3.0

User-cost inflation (PPL/PPLno-LoRA )

Figure 4: Rank-64 LoRA attacker frontier on the Qwen3-1.7B target. All four operating points sit above τpool =1.509; S TAGE A is the strongest we evaluate (2.78×τpool at 1.55× Pile ppx). Adding utility regularisation moves the attacker up and to the right. This slice of the frontier is an operatingrange, not a ceiling (Section 6).

Detection margin grows with SAE width. A scale-matched cross-family substitute — Qwen2.57B with a public-corpus φ fit on 2000 Pile samples (80/20 hold-out) — lands at joint-z=283.3 (260 × τ9B ; row 24 of Table 8). The same-family baseline (φ-only, no LoRA) with Gemma-2-2B-it through a joint φ to the 9B manifold reaches 280.2 (257×). An adaptive LoRA attacker — Gemma2-2B-it, r=64, 1500 steps, joint φ + LoRA trained under S TAGE A (λutil =0) — is still detected at joint-z=43.7 (40 × τ9B ) and pays a 30× Pile-perplexity price (ppx 560.2 vs. 18.5 substitute-clean). Compared with the Gemma-2-2B counterpart (97 × τ for the same attacker), the 9B margin to threshold is ∼ 2.4× tighter in absolute z but is carried by a 8× wider SAE; detection is strictly stronger per unit of honest noise. Reading the scale-up evidence. The honest threshold is scale-stable, the cross-family margin is ≥ 260 × τ at 9B, and an adaptive LoRA still leaves a 40× margin at Pile-unusable utility. Within the scoped threat model this closes the last scope-relevance gap the protocol could be challenged on at review: the commit-open primitive inherits the same separability story at 9B that we documented at 1.7B and 2B. Full numerics, training curves, and per-probe breakdowns are in Appendix F; cost was $2.43 of a $60 pre-committed scale-up budget. 5.6

Parallel-serve closure: commit-open vs. SVIP baseline

To isolate the commit-open binding contribution over probe-after-return schemes, we compare against a SVIP-style baseline [4] under matched backbone-specific thresholds on 11 attackers across both backbones (4 Qwen3: cheap lift, Qwen2.5-1.5B, Phi-3.5-mini, rank-64 S TAGE A LoRA; 7 Gemma-22B: 2 same-family lifts, 4 cross-family substitutes, 1 rank-64 adaptive LoRA). In SVIP, an attacker serving M ′ but retaining read access to open-source M routes the post-hoc probe to M while users see M ′ — the parallel-serve side-channel. Commit-open binds the SAE trace to the served M ′ before any probe, so opening is a membership check. Figure 5: SVIP accepts all 11 (11/11 miss, joint-z at the honest median on both backbones); commit-open rejects all 11 (11/11 detect) at ≥ 2.78 × τpool (Qwen3) and ≥ 97 × τGemma (Gemma). The gap is the binding step, not stronger attackers (weaker SVIP variants — probe batching, cached M responses — share the same acceptance envelope, Appendix W). 5.7

Feature-forgery infeasibility and library rotation

We close the complementary threat: an attacker who does not run M at all and fabricates the committed top-32 directly against the public library and threshold. Because joint-z is a mean over probe-slot penalties separable in the attacker’s featurePset, the optimum has a closed form (Appendix V): pick features by the per-feature gain G∗ (f )= (i,s)∈E(f ) [|µi,s |−|v ∗ (f )−µi,s |]/σi,s at the weighted-median v ∗ ; discrete top-32 plus bf16 quantisation fixes the rest. The empirical ladder 9

Joint-z (detection statistic)

SVIP (parallel-serve)

Commit-open (ours)

Qwen3

103

Gemma-2

all detected ( ≥3.7Gemma-2 × τ)

Qwen3

264.7

all accepted (under τ) 10

56.7

260.0 269.4

178.8 105.9

56.4

56.0

101

10

337.7

109.0

2

4.2

0.79

0

0.78

0.80

0.81

0.81

0.80

0.79

0.79

0.81

0.80

0.81

τ = 0.82

3 5 p r) ia IT .5 .5 eA eA 5B 2. e1 ea th ea m i-3 i-3 1. ag ag en Bch Py fa lin 5Ph Ph St St .4 e2. m Qw 4 4 m m 13 -1 m r6 r6 m -fa am ia (e -fa -fa s s a h fa e e s s f T t s I a s o sos os tiv tiv Py m m cr os cr cr os ap ap m m a fa cr a cr fa m ead en ad Ge a m en sa m m m m Qw wen m os sa m Ge Qw Ge cr a Q em Ge a m G m m m Ge Ge

τ = 0.82

3 5 r) ia IT .5 eA 2. e1 th ea m i-3 ag en BPy fa lin Ph St .4 em Qw 4 m 13 -1 m fa r6 m ia (e -fa sa h e s f T t s I a o sos o tiv t Py m m cr os cr cr os ap ap m m a fa cr a cr fa m ead en ad Ge a m en sa m m m m Qw wen m os sa m Ge Qw Ge cr a Q em Ge a m G m m m Ge Ge p ea

en

en

Qw

Qw

ch

m fa s-

1.

5-

2.

5B

m fa

ss

.5

i-3

Ph

ive

4

r6

eA

ag

St

sa

Figure 5: SVIP parallel-serve vs. commit-open across 11 attackers on two backbones. Under SVIP the verifier’s joint-z collapses to the honest median (11/11 miss; panel a); under commit-open all 11 attackers are detected at ≥ 2.78 × τpool (Qwen3) and ≥ 97 × τGemma (Gemma) (panel b). Table 5: Feature-forgery ladder. Per-position joint-z under attackers fabricating the top-32 commitment (no M forward). F0/F1: 500 positions × 5 seeds. F3: closed-form discrete optimum (Appendix V). All tiers are rejected by ≥ 29× the conservative threshold; the Prop. 4.2 intrinsic-dim lower bound (greedy T ∗ , c=|µ/σ| median) is 37.11. Tier

Attacker

F0 F1 F3 F3

uniform random top-32 pooled marginal, top-32 covered exact optimum (Qwen3) exact optimum (Gemma)

max

×τ

57.25 57.38 72.59 53.47 53.63 53.82 44.24 (deterministic) 221.6 (deterministic)

37.9 35.4 29.3 203.3

joint-z min

median

(Table 5, 500 pos.×5 seeds) records F0 uniform-random, F1 pooled-marginal, and the exact F3 optimum. On Qwen3 the F3 infimum is joint-z=44.24 (29.3τpool ), on Gemma 221.6 (203τGemma ); both match Prop. 4.2’s intrinsic-dim lower bound of 37.11 to within the weighted-median / bf16 gap. F2 (proxy-model forward through the honest SAE) is dominated by F3 and F4 (y-decodability constraint) is subsumed because T ∗ is position- and y-independent in feature-ID space (Appendix V). Library rotation. A probe-aware attacker could still overfit to the exact 96 probes; as a defencein-depth we evaluate a 50-fold 48/48 cross-validation of the F3 solver, training on 48 probes and re-scoring on the held-out 48. Train joint-z median is 40.29, test median 52.61 (∆=+12.3 transfer gap), with 0/50 folds below τpool on either side. A session-time rotation draw from a larger pool inherits this gap structurally. Additional experiments in appendix. Second-backbone Gemma evidence (Appendix E, Fig. 6); within-session dependence and robust-τ analysis (Appendix G); aggregator sensitivity (α × N sweep, Appendix H, Fig. 7); partial mechanistic auditability on local circuits (Appendix I); batched serving overhead (Appendix J, Fig. 8).

6

Limitations and Broader Impacts

Scope. The evidence in this paper covers three backbone–SAE pairs spanning 1.7–9B parameters (Qwen3-1.7B + transcoder L14, Gemma-2-2B + Gemma-Scope L12, Gemma-2-9B + Gemma-Scope9B L20 width-131k), the adaptation budgets listed in Table 6 (including a rank-128 frontier point on Qwen3), and a single serving stack for the ∼ 3% parallel-commit cost observation. Tens of billions of parameters and ≥ 70B flagship-class targets remain untested; we rely on the scale-stable threshold (τ9B =τGemma =1.09, agreement < 10−4 ) as inductive but not conclusive evidence that the primitive continues to hold at the “70B advertised / 7B served” operating point that motivates the protocol. Residual uncertainties — SAE/backend drift, MATH-kernel monoculture, session-level FPR under within-session dependence ρ=0.88, Gemma-σ recalibration, public-probe overfit with 40% mask-flip sensitivity, r>64/longer-trained/ full-finetune adaptive attackers, a white-box E-E diagnostic that does not close the margin but does not exhaust the white-box frontier either, and a single-stack rather than dominance cost argument (the economic binding of strategy (C) is now complemented by the 10

structural infeasibility of forgery via Prop. 4.2 rather than standing alone) — are enumerated with evidence pointers and mitigation sketches in Appendix X (Table 15). The scoring rule of Equation (2) is compatible without protocol change with stronger mitigations including secret-probe rotation, DP feature sketches, TEE attestation pairing, and an SVIP nonce-after-service redesign, which we leave to future work. Broader impacts. The primary positive impact is to increase auditability of hosted LLM serving: users who currently cannot verify identity gain a binding statistical primitive that is compatible with existing SAE/transcoder releases. Dual-use risk is limited and asymmetric: the protocol detects substitution but does not prevent it, and a capable provider could in principle adapt under budgets beyond those we evaluate. Providers deploying the protocol must be careful not to treat a single backbone–SAE pair as a backbone-agnostic guarantee; our three-backbone evidence (Qwen3-1.7B, Gemma-2-2B, Gemma-2-9B) argues for the primitive’s transferability and scale-stability up to 9B parameters, but not for a universal constant across all architectures and model sizes. No humansubject data, model release, or dataset release accompanies this work; the probe library is built from published named-circuit prompts and the honest pool is synthetic.

7

Conclusion

We study a commit-open protocol that binds the provider to the per-position SAE-feature trace of its served sequence before the verifier’s query, closing the parallel-serve side-channel of probe-afterreturn schemes. Within Table 6, it detects every tested non-adapted substitute at τpool =1.509, holds the strongest Qwen3 adaptive LoRA at 2.78 × τpool with 1.55× Pile perplexity, separates from SVIP on 11 attackers across two backbones (11/11 miss vs. 11/11 detect), and transfers to a 4.5× scale-up — Gemma-2-9B + Gemma-Scope-9B (L20, width-131k) — with a scale-stable threshold (τ9B =1.09, agreement with 2B < 10−4 ) and cross-family / adaptive-LoRA margins of 260× and 40 × τ9B . Commit adds ≤ 2.1% overhead at batch 32. Priority follow-up: a probe-aware white-box adaptive-frontier sweep and a DP-rotated secret probe library. Code, probe library, honest-pool artifacts, and calibration grids will be released upon publication.

References [1] Matteo Campanelli, Dario Fiore, and Anaïs Querol. LegoSNARK: Modular design and composition of succinct zero-knowledge proofs. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2075–2092. ACM, 2019. doi: 10.1145/3319535.3339820. [2] Dario Pasquini, Evgenios M. Kornaropoulos, and Giuseppe Ateniese. LLMmap: Fingerprinting for large language models. In 34th USENIX Security Symposium, pages 299–318. USENIX Association, 2025. [3] Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 3277–3306. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024. NAACL-LONG.180. [4] Yifan Sun, Yuhang Li, Yue Zhang, Yuchen Jin, and Huan Zhang. SVIP: Towards verifiable inference of open-source large language models. CoRR, abs/2410.22307, 2024. doi: 10.48550/ ARXIV.2410.22307. [5] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). OpenReview.net, 2022. [6] Jack Min Ong, Matthew Di Ferrante, Aaron Pazdera, Ryan Garner, Sami Jaghouar, Manveer Basra, Max Ryabinin, and Johannes Hagemann. TOPLOC: A locality sensitive hashing scheme for trustless verifiable inference. In International Conference on Machine Learning (ICML). PMLR / OpenReview.net, 2025. 11

[7] Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [8] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR). OpenReview.net, 2023. [9] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. Preprint: arXiv:2209.11895. [10] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. [11] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. [12] Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. SAELens: Training and analyzing sparse autoencoders, 2024. Software package, https://github.com/jbloomAus/ SAELens. [13] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [14] Marshall Ball, Alon Rosen, Manuel Sabin, and Prashant Nalini Vasudevan. Proofs of useful work. Cryptology ePrint Archive, Paper 2017/203, 2017. URL https://eprint.iacr.org/ 2017/203. [15] Pau-Chen Cheng, Wojciech Ozga, Enriquillo Valdez, Salman Ahmed, Zhongshu Gu, Hani Jamjoom, Hubertus Franke, and James Bottomley. Intel TDX demystified: A top-down approach. ACM Computing Surveys, 56(9):238:1–238:33, 2024. doi: 10.1145/3652597. [16] Jianwei Zhu, Hang Yin, Peng Deng, Aline Almeida, and Shunfan Zhou. Confidential computing on NVIDIA Hopper GPUs: A performance benchmark study. CoRR, abs/2409.03992, 2024. doi: 10.48550/ARXIV.2409.03992. [17] Reiner Sailer, Xiaolan Zhang, Trent Jaeger, and Leendert van Doorn. Design and implementation of a TCG-based integrity measurement architecture. In 13th USENIX Security Symposium, pages 223–238. USENIX Association, 2004. [18] Scott A. Crosby and Dan S. Wallach. Efficient data structures for tamper-evident logging. In 18th USENIX Security Symposium, pages 317–334. USENIX Association, 2009. [19] Ben Laurie, Adam Langley, and Emilia Käsper. Certificate transparency. RFC 6962, IETF, 2013. [20] Heng Jin, Chaoyu Zhang, Hexuan Yu, Shanghao Shi, Ning Zhang, Y. Thomas Hou, and Wenjing Lou. Trusting what you cannot see: Auditable fine-tuning and inference for proprietary AI. CoRR, abs/2603.07466, 2026. 12

[21] Hasan Akgul, Daniel Borg, Arta Berisha, Amina Rahimova, Andrej Novak, and Mila Petrov. Verifiable fine-tuning for LLMs: Zero-knowledge training proofs bound to data provenance and policy. CoRR, abs/2510.16830, 2025. [22] Ralph C. Merkle. A digital signature based on a conventional encryption function. In Carl Pomerance, editor, Advances in Cryptology - CRYPTO ’87, Lecture Notes in Computer Science, pages 369–378. Springer, 1987. doi: 10.1007/3-540-48184-2\_32. [23] Abraham Wald. Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16 (2):117–186, 1945. [24] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. [25] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505.09388, 2025. doi: 10.48550/ARXIV.2505.09388. [26] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjöesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024. doi: 10.48550/ARXIV.2408.00118. 13

[27] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024. doi: 10.48550/ARXIV.2412.15115. [28] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024. doi: 10.48550/ARXIV.2404.14219. [29] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious. CoRR, abs/2501.00656, 2025. doi: 10.48550/ARXIV.2501.00656. [30] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800GB dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. [31] Prasanta Chandra Mahalanobis. On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2(1):49–55, 1936. [32] C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.1093/biomet/26.4.404. [33] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. CoRR, abs/2408.05147, 2024. doi: 10.48550/ARXIV.2408.05147.

14

A

Threat Model

Table 6 defines the adversary we consider. Every served request carries a commitment R; the verifier audits a random subset of sessions with public probe library P and public decision threshold τ . Adversary knowledge includes the published SAE release, P, τ , and the form of the scorer in Equation (2); the adversary has black-box query access to M through a public npub =2000 corpus used to fit φ but not gradient access to the joint score (the primary threat model). A supplementary white-box variant (E-E, Section 5.4) permits gradient access through the frozen SAE encoder at fixed rank (r=64) for 3000 steps; full-model retraining and higher-rank / longer-trained white-box remain out of scope. Per-session FPR is bounded by 1 − (1 − α)k ≤ kα for k openings at per-opening α=0.01; dependence between openings within a session is measured empirically (Appendix G) and deflates the union bound on our stack.

Table 6: Scoped threat model. All audited sessions must carry a Merkle commitment R; the probe library and threshold are public. Dimension

Setting in this paper

Adversary goal Adversary knowledge

Serve M ′ while passing audit on M . Public: M , SAE release, probe library P, threshold τ , scoring rule. Black-box query access to M via 2000-sample public corpus. Two budgets: (i) public-corpus linear φ fit on 2000 pairs + LoRA r=16 on seven projections, ≤ 3000 steps; (ii) library-aware joint φ + LoRA r=64, 1500 steps. Stronger white-box or longer-budget attacks are out of scope. Each session audited independently; verifier samples k positions with kα ≤ 0.10. Conservative per-opening τpool =1.509 is the empirical max over pooled nhon =112 honest draws (MATH + multi-kernel seeds 300−302); 0/112 exceedances, Clopper–Pearson 95% upper CI 0.0264. Pool-internal legacy τreal =1.13 (nhon =64) retained for comparison. Cryptographic soundness, full-model retraining to fit the published library, nonaudited sessions. White-box gradient access through the frozen SAE encoder is partially evaluated (E-E, Section 5.4) but higher-rank / longer-trained / fullfinetune remains out of scope.

Adaptation budget Audit schedule Session-level guarantee

Out of scope

B

Proof Sketch and Tail Robustness of τ

Proof sketch. (B) Statistical detection. The per-probe score in Equation (2) is a mean of |Si | standardised z-scores on features where the honest calibration concentrates mass; cross-backend draws yield a bounded-variance null whose 99th percentile is estimated on a held-out split. Sequential calibration is outside the evaluated protocol. (C) Economic bound. leaf t depends on ft of the committed model; any provider that commits M must run M for every committed position, so wall-clock cost is lower-bounded by the honest forward pass up to overlap-with-M ′ parallelism. (D) Heuristic. Under a public-corpus fit, φ is the minimum-norm least-squares solution with empirical ∥φ∥F =0.59; we report this as a post-hoc empirical observation, not a formal bound (Appendix U).

Tail robustness of τ . τpool =1.509 is the empirical max over pooled n=112 honest draws. Fitting a Gaussian (resp. Student-t with df=5) to the n=64 pool shifts the parametric 99th percentile to 1.37 (resp. 1.44) — below τpool =1.509, so the pooled empirical max is the conservative choice given the tail uncertainty emphasised by Clopper and Pearson [32]. For multi-opening sessions, per-opening joint-z is not independent: a Gaussian-copula simulation at marginal α=0.01 with within-session ρ=0.88 yields session-FPR 1.9% at k=4 (vs. 3.9% under independence and 4% under kα). Cross-probe correlation within a single opening is not modelled; measuring it requires raw per-probe z-vectors and remains future work. 15

Table 7: Claim–evidence matrix. All Qwen3 experiments use Qwen3-1.7B at layer 14 with the public mwhanna-qwen3-1.7b-transcoders-lowl0 SAE release and a 96-probe library. Conservative deployment threshold on Qwen3 is τpool =1.509 (pooled nhon =112, MATH + multi-kernel seeds); pool-internal τreal =1.13 retained for comparison. Gemma-2-2B: τGemma =1.09 (nhon =64). Legacy per-probe Mahalanobis numbers labelled. Claim

Experiment

Key quantitative result

3/3 detected, TPR = 1.00, CP 95% CI [0.926, 1.00]; perprobe Mahalanobis 56–148× honest legacy scale C2 (cross-family) E3, 4 substitute models 4/4 detected at τpool =1.509 (0/112 honest exceedances, CP 95% upper CI 0.0264) C3 (rank-cstr. LoRA) E4 + diag per-category held-out joint-z ≥ 12 × τpool ; per-probe Mahalanobis 34–81× legacy honest max; ∥φ∥F =0.59 C4 (joint-consist.) E5, α × N sweep max ∆AUC=0.337; at α=0.002, N =1→0.63, N =96→0.93 C5 (auditability) E10, 4-class ablation IOI 4.35×, Factual 2.14×, Syntax 1.73×; Induction −0.78× (scoped out) C6 (economic) B1 + E9 batched substitute-serve detected; parallel-commit 97% of honest wall-clock; commit overhead ≤ 2.1% at B=32 C7 (second backbone) E12 + E13, Gemma-2-2B 8/8 non-adaptive attackers TPRjoint =1.00 at + Gemma-Scope τGemma =1.09; per-probe Mahalanobis 1.9×–5.8× maha τGemma =57.99 C8 (Gemma adaptive) E14, library-aware r=64 joint-z=105.9, 97.2 × τGemma , Pile ppx 8032 (436× StageA on Gemma-2-2B- clean); catastrophic utility loss it C9 (SVIP two- E11/Recipe 3 parallel- 11/11 miss under SVIP, 11/11 detect under commit-open backbone) serve C10 (white-box E-E) E16, 3 restart seeds at joint-z ∈ {6.89, 7.80, 4.99}; all restarts > 4.19 black(αjz , λutil )=(0, 0) box; ReLU-sparse gradients zero for inactive top-k C11 (9B scale-up) E12/E13/E14-9B, τ9B =1.09 (matches 2B < 10−4 ); cross-family Qwen2.5Gemma-2-9B + Gemma- 7B joint-z=283.3 (260×τ ); adaptive LoRA r=64 StageA Scope-9B L20 width- z=43.7 (40 × τ , 30× ppx) 131k C12 (r=128 frontier) E16-r128, Qwen3-1.7B + r=128 MSE λutil =0: joint-z=6.42 (4.25 × τpool ); doutranscoder L14 bling rank does not reopen the margin C1 (same-family)

E2, 3 lifted attackers

C

Claim–Evidence Summary Table

D

Master Results Table

Table 3 in the main body reports the strongest tested attacker per tier. Table 8 below gives the full per-attacker breakdown referenced throughout Section 5.2–5.6.

E

Second-Backbone Evidence: Gemma-2-2B with Gemma-Scope

To test whether separability is a property of the (M, L, SAE) tuple or of the protocol, we reinstantiate the pipeline on Gemma-2-2B + public gemma-scope-2b-pt-res canonical SAE [33] at L12 (residual-post, dSAE =16,384, top-32). The 96-probe library uses the same prompts via Gemma forwards; σi is recalibrated on 32 backend configs (2 dtypes × 2 kernels × 4 positions × 2 seeds, disjoint from honest-pool seeds). Deployment threshold τGemma =1.09 (nhon =64, 0/64, CP one-sided upper 0.0457). Eight Gemma attackers (four E2-equivalent lifts + four E3-equivalent cross-family substitutes: Gemma-2-2B-it, Pythia-1.4B, Qwen2.5-1.5B, Phi-3.5-mini-instruct) all clear maha τ at TPRjoint =1.00; per-probe Mahalanobis scores 109–338 (1.9–5.8 × τGemma =57.99). Cross2 family held-out R is negative for all three non-Gemma families (Pythia-1.4B −1.29; Qwen2.5-1.5B maha −15.40; Phi-3.5-mini −0.39). Same-family Gemma-2-2B-it is the closest call (2.3 × τGemma ) but clears on every replay. A library-aware rank-64 S TAGE A on Gemma-2-2B-it (Section 5.4, Table 4) reaches z=105.9 (97.2 × τGemma ) at Pile ppx 8032 (436× clean) — detectable with catastrophic utility loss, but the utility-preserving Gemma frontier (λutil >0 sweep) is unmeasured. 16

Table 8: Master results. Every attacker evaluated, consolidated onto the joint-z scoring rule where possible. 20 points on Qwen3-1.7B (transcoder, L14, τpool =1.509), 4 points on Gemma-2-2B (Gemma-Scope, L12, τGemma =1.09), 3 points on Gemma-2-9B (Gemma-Scope-9B, L20, width131k, τ9B =1.09, identical to 2B within 10−4 ), and one aggregate row for the 8 non-adaptive Gemmamaha 2B attackers of Appendix E on the legacy per-probe Mahalanobis scale (1.9−5.8 × τGemma =57.99), retained as an aggregate. Rows 1–3 are E2 lift attackers: joint-z reaches TPR= 1 at τpool (Section 5.2). “×τ ” is the ratio to the per-backbone threshold; “ppx” is Pile perplexity (M ′ -clean baseline 21.20 Qwen3, 18.44 Gemma-2B, 18.50 Gemma-2B-it → Gemma-9B substitute). All 33 evaluated attackers are rejected by commit-open; the SVIP baseline column summarises Section 5.6 (11/11 miss on its re-run set). Row 13 reports the median over three restart seeds for the Qwen3 (αjz , λutil )=(0, 0) cell only; rows 17–18 report the first run of a two-seed pair, with the second-seed variant (jointz={9.71, 57.38} respectively, opposite stability outcome) listed as rows 17′ /18′ — both pairs are detected, and this confirms the seed-variance caveat noted for white-box E-E. All other rows are single runs. DIV: ppx> 106 . #

Class

Attacker

Budget

×τ

Pile ppx

Verdict

SVIP verdict

≫ τpool ∗ TPR= 1 ≫ τpool ∗ TPR= 1 ≫ τpool ∗ TPR= 1 69.2 45.9× 52.4 34.7× 81.3 53.9× 62.8 41.6× ≥ 18.1 ≥ 12× 57.38 38.0× 4.19 2.78× 6.27 4.16× 7.48 4.96× 6.89 (med.) 4.57× 49.46 32.8× 57.38 38.0× 9.99 6.62× 56.55 37.5× 7.40 4.90× 9.71 6.44× 57.38 38.0×

− − − − − − − ∼ 25 21.20 32.84 63.16 50.98 31.9 3625 DIV 1188 DIV 243 86,809 3591

detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect detect

miss − − miss miss − − − − miss − − − − − − − − − −

50.37 3538

detect detect

− −

joint-z

Qwen3-1.7B target, τpool =1.509 1 lift ZERO_PAD Qwen3-0.6B 2 lift RAND_PROJ Qwen3-0.6B 3 lift LEARNED_LIN Qwen3-0.6B 4 cross Qwen2.5-1.5B + public φ 5 cross Phi-3.5-mini + public φ 6 cross OLMo-2-7B + public φ 7 cross Qwen3-0.6B + public φ 8 LoRA public-φ + LoRA r=16 9 LoRA library-aware φ only 10 LoRA E4-v2 S TAGE A λ=0 11 LoRA E4-v2 S TAGE C λ=0.1 12 LoRA E4-v2 S TAGE B λ=1.0 13 whitebox E-E αjz =0, λ=0 14 whitebox E-E α=0, λ=0.1 15 whitebox E-E α=0.1, λ=0 16 whitebox E-E α=0.1, λ=0.1 17 whitebox E-E α=1, λ=0 18 whitebox E-E α=1, λ=0.1 17′ whitebox E-E α=1, λ=0 (seed 2) 18′ whitebox E-E α=1, λ=0.1 (seed 2) Higher-rank LoRA frontier (P0-2 scale-up) 19a LoRA r=128 MSE, λ=0 19b LoRA r=128 MSE, λ=0.1

E2, natt =48 E2, natt =48 E2, natt =48 E3 E3 E3 E3 3000 steps E4-v2 baseline r=64, 1500 steps r=64, 1500 steps r=64, 1500 steps r=64, 3000 st., 3 seeds† r=64, 3000 steps r=64, 3000 steps r=64, 3000 steps r=64, 3000 steps r=64, 3000 steps r=64, 3000 steps r=64, 3000 steps r=128, 3000 steps r=128, 3000 steps

6.42 47.01

4.25× 31.1×

Gemma-2-2B target, τGemma =1.09 19 20 21 22 23

mixed 8-attacker aggregate LoRA library-aware φ-only (Gemma) LoRA E14 S TAGE A λ=0 whitebox E-E Gemma α=1, λ=0 whitebox E-E Gemma α=1, λ=0.1

maha 4 lifts + 4 cross-family, legacy Mahalanobis only 1.9−5.8 × τGemma − 8/8 detect 6/6 miss‡ E14 base 205.56 188.6× 18.44 detect − r=64 105.89 97.2× 8031.65 detect miss r=64 133.86 122.8× 12,356 detect − r=64 144.40 132.5× 1132 detect −

Gemma-2-9B target (P0-1 scale-up), L20, width-131k, τ9B =1.09, nhon =64 24 25 26 † ∗ ‡

cross LoRA LoRA

Qwen2.5-7B + public φ, scale-matched Pile 2000, 80/20 Gemma-2-2B-it + joint φ-only (baseline) no LoRA training Gemma-2-2B-it + S TAGE A λ=0 r=64, 1500 steps

283.30 260.0× 280.17 257.0× 43.70 40.1×

− 18.50 560.16

detect detect detect

− − −

Restart seeds {s0, s1, s2}: joint-z={6.89, 7.80, 4.99}; best-for-attacker 4.99 still > 4.19 black-box. maha E2 lifts reach TPR= 1 at τpool ; legacy per-probe Mahalanobis: 56−148 × τQwen =1.51. Of 8 aggregate attackers, 6 rerun under SVIP-routing (2/4 lifts + 4/4 cross-family); all 6 miss.

F

Scale-Up to Gemma-2-9B with Gemma-Scope-9B

This appendix substantiates the scale-up claims of Section 5.5: that the commit-open primitive transfers without re-engineering to a 4.5× larger target, the honest threshold is numerically indistinguishable from the 2B value, and detection margins strictly grow with SAE width.

Setup. Target: google/gemma-2-9b at L20 (dmodel =3584, 26 transformer blocks; L20 matches the 46% depth ratio of the 2B-L12 reference); SAE: gemma-scope-9b-pt-res-canonical, layer_20/ width_131k/canonical, avg-L0 =114, dSAE =131,072 [33]. The 96 probe prompts and 8 circuit-class taxonomy from the 2B second-backbone replication (Appendix E) transfer verbatim; top-32 feature IDs, µ, σ are recomputed on 9B residual activations. Honest pool: n=64 (2 dtypes × 2 kernels × 4 positions × 4 seed groups), reusing the multi-kernel recipe of Appendix T. Compute: four Modal runs on A100-40GB and A10G, total $2.43 of a $60 pre-committed scale-up budget. 17

Qwen3-1.7B

Gemma-2-2B

τ = 1.51

τ = 57.99

102

Joint Mahalanobis

Joint Mahalanobis

102

101

100

101

100

hon

est

ad

op zer

ran

dp

roj

d rne

lea

est hon

lin

ive

nat

it

n ad roj d li op dp rne zer ran lea

Figure 6: Two-backbone separability on the per-probe Mahalanobis diagnostic scale (log y-axis, each panel uses its backbone’s own Mahalanobis threshold). (a) Qwen3-1.7B + transcoder L14 (nhon =112 maha pooled; τQwen =1.51). (b) Gemma-2-2B + Gemma-Scope canonical residual SAE L12 (nhon =64, maha τGemma =57.99). Four additional E3-equivalent cross-family attackers (Pythia-1.4B, Qwen2.5-1.5B, Phi-3.5-mini-instruct, Gemma-2-2B-it) also clear both backbones’ joint-z thresholds (not plotted). Honest threshold. τ9B =1.090 (pool p99), versus τGemma =1.090 on 2B and τpool =1.509 on Qwen3. Gemma-2-9B and Gemma-2-2B thresholds agree to within 10−4 on the same 96-probe prompt set — strong evidence that the pooled-p99 joint-z is a property of the probe library and scoring rule rather than of the model size. Cross-family (row 24 of Table 8). Qwen2.5-7B [27] with a public-corpus φ : R3584 → R3584 fit by linear least-squares on 2000 Pile samples (80/20 train/hold-out; held-out R2 = − 0.091, consistent with the negative cross-family R2 pattern seen on 2B). Joint-z=283.3, per-probe minimum 253.1. Detection is 260 × τ9B — a full two orders of magnitude tighter than the 2B cross-family rejection margin (median ∼ 3 × τGemma on the per-probe Mahalanobis scale). The mechanism: at width 131k the per-probe reference pattern is higher-intrinsic-dim (Prop. 4.2 scales ∝ log dSAE in the idim bound), and a φ that fits Pile-average activations does not bring 9B top-32 features to Qwen2.5-7B’s manifold except in directions explicitly represented in the Pile marginal. Same-family φ-only baseline (row 25). Gemma-2-2B-it through a joint φ to the 9B manifold, no LoRA: joint-z=280.2 at Pile ppx 18.5 (baseline clean). This establishes that a 7× smaller same-family substitute, even with a trained linear lift, is rejected at 257 × τ9B before adaptation begins. Adaptive LoRA (row 26). Same recipe as Qwen3 S TAGE A: M ′ =Gemma-2-2B-it, r=64, α=32, seven projection modules, 1500 steps AdamW lr 3×10−4 , λutil =0 (pure-probe). Joint-z=43.70, 40.1 × τ9B ; Pile ppx 560.2 (30× substitute-clean). In absolute z the margin is ∼ 2.4× tighter than the matching 2B point (z=105.89), but the 9B SAE is 8× wider, so detection is strictly stronger per unit honest-pool noise. The attacker also pays a larger utility penalty in relative terms than on 2B (30× at 9B vs. 436× at 2B is superficially lower, but the 9B substitute starts from a 3× smaller clean ppx baseline and is pushed into a higher absolute ppx regime). A utility-preserving 9B frontier (λutil >0 sweep) is not evaluated and, as on 2B, is an acknowledged caveat (Section 6). Decision-gate verdict. The pre-committed gate of the scale-up plan required (a) τ9B ∈ [0.8, 2.5], (b) adaptive LoRA > 3 × τ , (c) cross-family > 20 × τ . Observed: (a) τ9B =1.09; (b) 40 × τ ; (c) 260 × τ . All three checks pass by > 10× the gate.

G

Within-Session Dependence and Robust τ

The n=64 honest pool partitions into 16 (dtype, kernel, seed) tuples of 4 positions, letting us estimate cross-position correlation within a session. Mean off-diagonal correlation across the four within-tuple positions is ρ=0.883 (Table 9). A Gaussian-copula simulation at marginal α=0.01 yields session-FPR 1.93% at k=4, vs. 3.94% under independence and kα=4% union bound — positive 18

cross-opening correlation deflates session-FPR below the naive bound. A follow-up multi-kernel spot check on Modal L4 draws 48 additional honest configs at companion seeds 300−302; MATH and EFFICIENT_ATTENTION medians agree to 4.6% (0.958 vs. 1.002). The companion-seed family shifts joint-z median more than the kernel (0.671 on seeds 100−107 vs. 0.958−1.002 on seeds 300−302), so pooling n=112 raises the empirical max to 1.509 (adopted as τpool ). Table 9: Session-level FPR at nominal marginal α=0.01 under (i) the naive kα union bound, (ii) exact independence 1−(1−α)k , and (iii) a Gaussian-copula simulation at ρ=0.883 (Qwen3 n=64 pool, Nsim =105 ).

H

k

kα union

independent

copula (ρ=0.883)

1 2 3 4

0.010 0.020 0.030 0.040

0.010 0.020 0.030 0.039

0.010 0.015 0.018 0.019

Aggregator Sensitivity: Joint Consistency vs. Single Probe

AUC (mean over 4 attacker centers)

This is an aggregator sensitivity analysis, not an adversarial test: we interpolate attacker centers as α zattacker + (1−α) µhonest with honest noise and sweep N ∈ {1, . . . , 96} at a fixed FPR budget. As α decreases, single-probe AUC collapses while joint consistency retains detection, peaking at ∆AUC=0.337 for α=0.001; all four attacker centers pass a ∆AUC ≥ 0.03 gate. The N -probe sweep (Figure 7) shows AUC rising monotonically with N , largest benefit at α ∈ {0.0005, 0.001, 0.002}; the curve flattens by N ≈32, motivating N =96 as a comfortable operating margin. The SPRT aggregator at α=β=0.01 terminates at n∗ =1 under full-strength E3 attackers, so the joint aggregator only contributes in the weakened-mean regime; we do not claim robustness against probe-aware adaptive attackers. 1.0

max ΔAUC = 0.337

0.9 α = 0.0 α = 0.0005 α = 0.001 α = 0.002 α = 0.005 α = 0.01 α = 0.02 α = 0.05 α = 0.2 α = 1.0

0.8

0.7

0.6

0.5 1

2

4

8

16

32

64

96

Number of probes aggregated (N)

Figure 7: Aggregator N -probe sweep: mean AUC across four attacker centers at each α-weakened operating point. ∆AUC between N =1 and N =96 is largest for small α, and AUC plateaus by N ≈32.

I

Partial Mechanistic Auditability

We ablate the top-32 features of each probe class and measure class-specific effect via ∆ KL = KL(pclean ∥ pabl ) − KL(pclean ∥ prec ). Of the four circuit classes tested, three local-circuit classes admit class-specific attribution under our top-32 sketch: IOI (4.35×), factual (2.14×), syntactic (1.73×); the fourth, induction, is known to be distributed across heads and layers [9] and does not localise (specificity −0.78×). We therefore make only a partial auditability claim, restricted to local circuit classes for which the top-32 sketch carries class-specific signal; this is not a general mechanistic-auditability guarantee. 19

J

Serving Overhead (E9)

A single-GPU RTX 3090 sanity test (200 prompts, output length 96) gives strategy-(B) TPR = 1.00 at τpool =1.509 while strategy-(C) parallel-commit deception costs 97.0% of honest wall-clock (∼ 3% gap on a single GPU; one operating point, not a dominance claim). At batched Modal L4 serving, amortised C/A falls from 1.185 (B=1) to 1.021 (B=32, 2.1% latency); payload is 224 bytes/request (Figure 8).

Per-batch latency (ms)

Relative overhead

Forward only Forward + SAE + commit

Overhead over forward-only (%)

Absolute latency 500

+8.0 ms

400

300 +5.9 ms

200

100

+4.5 ms

+5.4 ms

0

1

4

16

20

18.5%

15 10.2%

10

5

3.4%

0

32

1

4

Batch size

16

2.1%

32

Batch size

Figure 8: E9 batched commit overhead. (a) Per-batch latency for forward-only (A) and forward + SAE + commit (C) with 20-trial standard deviation. (b) Commit overhead C/A − 1 collapses from 18.5% at B=1 to 2.1% at B=32.

Cross-Family φ Transfer Diagnostic

K

Figure 9 reports the fit and held-out R2 of the public-corpus linear alignment map φ for each crossfamily attacker, alongside the verifier-side joint z-score. Two attackers (Phi-3.5-mini, OLMo-2-7B) exhibit strongly negative held-out R2 , confirming that the public-corpus φ does not generalise across model families. Partial explanatory analysis for C3; detection is established independently by the joint z-score. φ-fit quality

Detection score

1.0

τ = 0.82

Joint Mahalanobis

R2

0.0 −0.5 −1.0 −1.5

fit R 2 held-out R 2

−2.0

Qwe

n2.5

B -1.5

min

3.5Phi-

i

-7B

o-2

OLM

0.6B

n3-

Qwe

57.4×

56.4×

56.0×

−1 −2

10

1

−3

held-out R 2

held R = − 5.97

0.5

0

85.5×

102

2

−4

100

−5

-1.5

n2.5

Qwe

B

i

min

3.5-

Phi-

-7B 0.6B n3Qwe

o-2

OLM

Figure 9: E3 cross-family φ-transfer diagnostic. (a) Fit R2 on the 2000-pair public-corpus training set vs. held-out R2 on the 20% split. (b) Joint z-score on the verifier side; all four attackers clear τ (dashed).

L

E4 Rank-Constrained LoRA Diagnostic

Figure 10 shows three diagnostic runs that confirm (a) LoRA trains (B-matrix Frobenius norm grows from 0 to 5.24 during stage A and stabilises through stage B); (b) gradients flow through the probe-layer hook (204 of 392 trainable parameters receive gradient); (c) with a random φ of the same Frobenius norm, eval- and train-mode outputs differ after training. The limiting factor is the 20

min-norm structure of the learned φ, not a training bug: ∥∆h∥∞ ∼ 1 projects to ∥∆proj∥∞ ≤ 0.01 through φ, below the SAE sparsity threshold. LoRA norm vs φ bottleneck 5.24

5.28

Median detection score

6

Per-category detection score

LoRA ‖B‖F

5 4 3 2 1 0

61

54 38

73

34

34

101

‖φ‖F = 0.59

0.00 init

Honest max = 1.51 LoRA attacker (held-out)

102

stage A (500 weak)

stage B (+2500 robust)

100 ioi

n

tio

uc

ind

ref

co

l

tua

fac

c

eti thm ari

x nta

sy

Figure 10: E4 rank-constrained LoRA diagnostic. (a) LoRA’s B-matrix Frobenius norm grows across stage-A and stage-B, yet stays below ∥φ∥F . (b) Per-category median joint z-score on held-out probes after full 2-stage training remains ≥ 12× the conservative deployment threshold τpool =1.509.

M

White-Box Joint-z Gradient Attacker (E-E)

Table 10 lists all eight evaluated operating points. The attacker objective is L = MSE(φ(hM ′ ), hM )+ αjz Ljz + λutil NLLM ′ (Pile) with Ljz computed through the frozen SAE encoder over the top-32 mask. Training uses AdamW lr 3×10−4 at LoRA r=64 for 3000 steps. The strongest Qwen3 cell (αjz =0, λutil =0), pure MSE, yields joint-z=6.89 (median over 3 restarts), weaker than the E4-v2 black-box optimum 4.19: ReLU-sparse SAE encoder yields near-zero gradients for inactive top-k features — the score-gradient term is a noisy small-support signal that fights the dense MSE. Adding λutil pulls the hidden state off the honest manifold (joint-z rises to 49.5); turning up αjz without utility regularisation diverges (ppx>107 ). Concrete white-box negative result within this budget: r=64 × 3000 steps, gradient access to the scoring rule does not help the attacker. Compute: Qwen3 grid ≈ 40 min L4 (≈ $0.55); Gemma ≈ 21 min L4 (≈ $0.29).

Table 10: White-box joint-z gradient attacker (E-E). LoRA r=64, 3000 steps, SAE weights frozen but gradients propagated through the ReLU encoder. The (αjz =0, λutil =0) row reports median over three restart seeds ({6.89, 7.80, 4.99}); other rows are single runs. The best-for-attacker restart (lowest joint-z=4.99) is still weaker than the E4-v2 black-box MSE optimum 4.19. “DIV”: ppx > 106 . Backbone

αjz

λutil

joint-z

Pile ppx

Qwen3-1.7B, τpool =1.509 Qwen3 0 0 6.89 (med.) Qwen3 0 0.1 49.46 Qwen3 0.1 0 57.38 Qwen3 0.1 0.1 9.99 Qwen3 1.0 0 56.55 Qwen3 1.0 0.1 7.40

31.9 3625 3.2×107 1188 1.0×1010 243

Gemma-2-2B, τGemma =1.09 Gemma 1.0 0 133.86 Gemma 1.0 0.1 144.40

12,356 1132

21

3 seeds DIV DIV

31.9

0.4

7.7

13.9

induction

-0.3

-3.3

9.2

3.9

syntax

32.5

-5.4

25.9

18.0

0.030

Probe class

0.025 0.020 0.015 0.010 0.005 16.0

factual

ioi

ioi: spec=+4.35×

-0.8

7.3

16.0

induction syntax Ablated feature class

induction: spec=-0.78×

0.000

KL(clean‖ablated) − KL(clean‖recon) (×10−3 )

ioi

factual

syntax: spec=+1.73×

factual: spec=+2.14×

Figure 11: E10 circuit-ablation effect matrix. Cell (i, j) is KL(clean∥ablated) − KL(clean∥recon) when the top-32 features of probes in class j are zeroed, evaluated on probes of class i. Values are in units of 10−3 .

N

Circuit-Ablation Effect Matrix

O

E6 Per-Category Attackability and Library Rotation

Median Mahalanobis (attacker)

Figure 12 orders probe categories by the median joint z-score of the full E3 attacker suite. Robust classes (arithmetic, factual, commonsense) are retained for high-assurance regimes; attackable classes (induction, coreference, IOI) are candidates for periodic rotation.

102

71

66

63

63

59

55

51

48

48

38

34 Honest max (1.51) ROBUST (retain) ATTACKABLE (rotate out) other

101

100

tic ual lang usal oref ent trol tion tax ioi tion f c ilm con ga syn me fact uc re ta ne ind en

ith ar

Figure 12: E6 per-category attackability under the E3 attack suite. Policy: rotate attackable classes; bias the library toward robust classes for high-assurance regimes.

P

Top-k Sweep

We re-evaluate the Qwen3 honest pool and the strongest Qwen3 white-box attacker (E-E, αjz =0, λutil =0, r=64, 3000 steps) at k ∈ {4, 8, 16, 32} with per-feature z-arrays captured post-hoc and re-aggregated. Detection margin is stable at 4.64−4.99× across the k range (Table 11); both τ and the attacker score rise at smaller k because fewer features give individual strong activations more weight, and the two rise in lockstep. 22

Table 11: k-sweep on Qwen3-1.7B with multikernel honest pool (n=48, seeds 300−302) and the strongest E-E attacker. Margin = attacker joint-z / pool-max τ at the same k. k

honest median

honest max (τ )

attacker joint-z

margin

4 8 16 32

1.218 1.208 1.166 0.959

1.850 1.862 1.878 1.509

9.226 9.134 8.791 7.002

4.99× 4.90× 4.68× 4.64×

Caveats: the honest pool is the fresh seed-family spot-check pool (Appendix T), whose pool-max τ at k=32 is 1.509, not the headline τreal =1.13 on the original n=64 pool (which stored only scalar joint-z, foreclosing a post-hoc k-sweep). The attacker is the white-box Qwen3 pure-MSE point, which at joint-z=7.00 is weaker than the E4-v2 black-box optimum (4.19); the k-sweep bounds but does not fully characterise the strongest-attacker frontier across k.

Q

Public/Secret Probe Split

We partition the 96 Qwen3 probes 64/32 into public / secret hold-out (seed 42), re-run the white-box attacker (E-E αjz =1.0, λutil =0.1, r=64, 3000 steps) with Ljz restricted to the public 64 probes, and evaluate separately. Table 12: Public/secret probe hold-out for the strongest Qwen3 white-box operating point. 6% gap between public and secret: no systematic overfit, and secret hold-out is slightly harder. Both remain ≳ 36× above τpool =1.509. Public (n=64)

Secret (n=32)

54.56 137.8 36.2×

57.92 145.8 38.4×

joint-z per-probe max margin vs. τpool =1.509

Empirical public/secret gap is 6.2%, comparable to seed variance, so the attacker neither generalises better nor worse to probes it was not trained against. A more aggressive evaluation would apply the same split to the strongest (black-box E4-v2, joint-z=4.19) attacker; this requires retraining to capture probe-filtered gradients and falls outside the r=64 white-box frontier evaluated here.

R

Mask-Flip Sensitivity

We audit the stability of the top-32 indices by randomising a fraction f per probe. At full E3-v2 attack strength (α=1) AUC remains 1.00 for f ∈ {0, 0.05, 0.10, 0.20, 0.40}. At α=0.001-weakened attackers (mixtures αM ′ +(1−α)M ), worst-case AUC drops from 0.613 to 0.544 across the 40% flip range (Table 13). Monotone degradation is a sanity check; production deployment should combine secret-probe rotation and DP feature sketches (both compatible with Equation (2) without protocol change). Table 13: Mask-flip sensitivity at α=0.001 weakened attackers. AUC computed against nhon =2000 honest draws and 20 seeds per flip fraction. Attacker

f =0

f =0.05

f =0.10

f =0.20

f =0.40

Qwen2.5-1.5B Phi-3.5-mini-inst. OLMo-2-7B Qwen3-0.6B

0.613 0.624 0.942 0.657

0.609 0.620 0.937 0.653

0.598 0.611 0.935 0.640

0.586 0.593 0.919 0.623

0.544 0.551 0.878 0.570

Mean

0.709

0.705

0.696

0.680

0.636

23

S

Gemma σ Recalibration

The initial Gemma-2-2B pilot calibrated per-feature σ on 8 honest configurations (2 dtypes × 2 kernels × 2 positions); on a fresh n=64 honest pool several per-feature σi,j sat at the numerical floor, so bf16-level drift produced 1000+ × σ excursions on those features (joint-z≈58 at positions 0/1). A recalibration to 32 configs (2 dtypes × 2 kernels × 4 positions × 2 seeds) shifted σ-tail mass joint (fraction below 10−3 : 0.42%→0.10%) and yielded τGemma =1.09 (nhon =64, 0/64 exceedances, CP 95% upper 0.0457). Protocol guidance: σ calibration for long-tailed SAE feature distributions (Gemma-Scope has p95 σ=13.76, max =624) requires a calibration pool that spans every free axis (dtype, kernel, position, seed).

T

Multi-Kernel Honest-Pool Spot Check

The original Qwen3 n=64 pool ran on WSL2/CUDA, exposing only the MATH SDPA kernel. We re-run on Modal L4 to exercise the two SDPA backends dispatchable under our 1+3-companion opening format (MATH and EFFICIENT_ATTENTION); the spot check covers 24 configs per kernel (2 dtypes × 4 positions × 3 seeds {300, 301, 302}). Table 14: Multi-kernel honest-pool joint-z on Qwen3-1.7B, Modal L4, seeds 300−302. MATH and EFFICIENT_ATTENTION medians agree to 4.6%; seed-family variance is the dominant source. Kernel

n

min

median

max

p99

MATH EFFICIENT_ATTENTION

24 24

0.489 0.495

0.958 1.002

1.440 1.509

1.440 1.509

Pooled fresh Original pool Pooled all

48 64 112

0.489 0.243 0.243

0.959 0.671 0.677

1.509 1.120 1.509

1.509 1.120 1.509

The companion-seed family dominates: seeds 100−107 give median 0.671 whereas seeds 300−302 give median 0.958−1.002, a ∼ 43% shift that swamps the 4.6% kernel effect. 24/112 pooled configs exceed τreal =1.13, which is why headline claims use τpool =1.509 throughout. The strongest Qwen3 attacker (E4-v2 r=64) lands at joint-z=4.19; margin 2.78× under τpool (vs. 3.71× under τreal ). Wall-clock for the 48-config run was 343 s on one L4 (∼ $0.08).

U

Heuristic Explanation for the r=16 Result

Post-hoc empirical intuition, not a formal bound. In our Qwen3 setup, the public-corpus minimumnorm φ on npub =2000 paired samples has ∥φ∥F = 0.59. In the r=16 LoRA runs, substituteside perturbations of order ∥∆h∥∞ ∼ 1 project through this φ to per-element changes of order ∥∆proj∥∞ ∼ 10−2 on the honest-side residual stream — under the top-32 SAE mask, insufficient to flip the mask on the tested probes. A formal capacity bound, mask-flip threshold, and scaling curve in npub remain future work.

V

Feature-Forgery Infeasibility (Proposition 4.2 in detail)

This section provides the derivation of Proposition 4.2, the empirical intrinsic-dim numbers, and the F3 exact-optimum solver used in Section 5.7. The claim is structural: an attacker who does not run M cannot beat τpool regardless of optimisation budget. |P|

Setup. Fix the probe library {(Si , µi , σi )}i=1 with |P|=96, |Si |=k=32. Attacker picks T ⊆ {0, . . . , dSAE −1} with |T |=k and a value map v : T → R quantised to bf16. Per-slot P penalty is 1 pi,s (T, v) = |v(Si,s ) − µi,s |/σi,s if Si,s ∈ T , else |µi,s |/σi,s . Joint-z is z(t) = |P|k i,s pi,s (T, v), which is separable in T . 24

Per-feature gain. For a candidate feature f , let E(f ) = {(i, s) : Si,s =f } be the set of probe-slot occurrences. The gain from including f at value v is G(f, v) =

X (i,s)∈E(f )

|µi,s | − |v − µi,s | , σi,s

maximised at the weighted-median v ∗ (f ) = wmed{(µi,s , 1/σi,s )}(i,s)∈E(f ) (convex ℓ1 fit). Write G∗ (f ) = G(f, v ∗ (f )) and γ(f ) = G∗ (f )/|E(f )| for the per-occurrence gain rate. Since the objective decomposes by feature, the attacker’s optimum is the top-k in G∗ :  X X 1 X |µi,s | ∗ ∗ G (f ) , T = arg max G∗ (f ). z∗ = − σ |P|k i,s i,s |T |=k ∗ f ∈T

f ∈T

Lower bound (Prop. 4.2). The total slot count is |P|k; each of the attacker’s k features can cover at most |E(f )| slots. Let m̄ = meanf |E(f )| = |P|k/U be the average multiplicity. A uniform upper bound on attacker coverage is k m̄ = k 2 |P|/U , so 1−k 2 |P|/(|P|k · U )=1−k/U of the slots are forced to the zero-penalty floor c = mediani,s |µi,s |/σi,s . In deployed numbers (U =1472, k=32, |P|=96): attacker coverage rate ≈ k/U · m̄ = 32 · 2.09/1472 ≈ 0.045 of slots, so (1−0.045) c≈0.955 · 22.78=21.75 is the uniform-coverage lower bound. A tighter (data-dependent) bound evaluates the actual greedy optimum: it picks T ∗ by G∗ (f ), achieving coverage 3.84/32 slots ∗ per probe on average and per-probe zi ≥11.25, yielding the reported zgreedy =37.11=24.6τpool . Empirical library measurements. Computed offline from the deployed 96 probes + σ-calibration (pilots/p1_probe_lib/20_intrinsic_dim_analysis.py): |P| · kS total memberships U = | i Si | membership mean / median / max features in ≥ 2 probes pairwise overlap mean / pairs with 0

3,072 1,472 2.09 / 1 / 66 379 / 1,472 3.52 / 1,064 of 4,560

|µ/σ| min/p5 /med |µ/σ| mean/max greedy T ∗ coverage / probe greedy per-probe zi min/med ∗ (Prop. 4.2) zgreedy

0.0 / 0.0 / 22.78 57.38 / 2,350 3.84 / 32 11.25 / 34.12 37.11

Exact-optimum solver vs. bound. The closed-form solver (e18_forgery_f3_local.py) picks T ∗ by G∗ (f ) and assigns bf16-quantised v ∗ (f ) to each, returning z ∗ =44.24 (29.3τpool ) on Qwen3 and z ∗ =221.64 (203τGemma ) on Gemma. The gap between the bound (37.11) and the exact optimum (44.24) reflects the fact that the bound used slot-averaged c without accounting for bf16 rounding, weighted-median fit, and that the greedy attacker’s coverage is not uniform. Critically, no gradient-descent, Gumbel-softmax, or longer-horizon discrete search can improve on z ∗ =44.24 in our formulation, because the objective is separable and the inner optimisation admits the exact per-feature formula above. F4 (y-consistency) subsumption. A further tier F4 requires attacker-served y to be decodable text. Because the feature-library lives in SAE-feature-ID space and not in token space, the attacker’s optimal top-32 is independent of y content. Any F4 constraint on y can only restrict the attacker’s ∗ strategy space and therefore zF4 ≥ zF3 =44.24. F4 was therefore not run as a separate experiment. F2 (proxy-model) omission. F2 would run a smaller proxy (e.g. Qwen2.5-0.5B) through the M -matched SAE. But the M -SAE (dhidden =2048 for Qwen3-1.7B) cannot be applied to a proxy with mismatched hidden dimension, and a projection layer takes the attacker into F3/F4 territory. Since F3 is already the exact infimum of joint-z over discrete feature-fabrication strategies, F2 cannot produce a lower joint-z than F3. We report F2 as formally dominated and omit the experiment. Library-rotation spot check (Section 5.7). Rather than building a larger probe pool (e.g. 256 probes), we simulate rotation by cross-validating the F3 solver: attacker-train on 48 probes, test on held-out 48. Over 50 random folds, train median joint-z is 40.29 and test median is 52.61 (+12.3 transfer gap; both 0/50 below τpool ). A session-time rotation draw from a larger pool inherits this gap structurally. The script is e19_library_split_generalization.py. 25

τGemma = 1.09

Prop. 2 LB = 37.1

τpool = 1.51

1.0

Honest pool (n=112) F0 ⋅ uniform random

57.4

F1 ⋅ pooled marginal

53.6

F3 ⋅ exact optimum (Qwen3)

44.2

F3 ⋅ exact optimum (Gemma)

221.6

101

100

102

103

joint-z (log scale) Honest

F0/F1 naive

F3 exact optimum

(a) Feature-forgery ladder (log x-axis). Prop. 4.2 LB = 37.1 (dashed) and τpool , τGemma (dotted).

joint-z (F3 probe-aware)

70

Δ median = +12.3

60 50 median 52.6

40 median 40.3

30

median p5 / p95

20 10

min / max τpool = 1.51 (attacker floor ≫ τ)

0 Train probes (attacker-seen)

Test probes (held-out)

(b) F3 train vs. held-out over 50 random 48/48 splits. Transfer gap ∆ median=+12.3 joint-z.

Figure 13: Feature-forgery attack ladder and library-rotation transfer gap for Qwen3 + Gemma. Even the exact-optimum discrete F3 attacker (top) clears τ by 29× on Qwen3; rotation (bottom) adds a further +12.3 transfer gap on held-out probes.

W

Weaker SVIP Attackers

Section 5.6 assumes the strongest SVIP attacker: full routing of each probe query to M . Weaker SVIP attackers share the same side-channel as long as the probe is separated in time from the commit. (i) Probe batching: accumulate probes and answer in a single batch on M . (ii) Cached M responses: pre-compute M responses for a verifier-visible probe set. Both share the acceptance envelope of Figure 5(a).

X

Caveat Index

Y

Out-of-Scope Strategies and Rebuttal Map

26

Table 15: Limitations index. Cluster → evidence → residual uncertainty → compatible mitigation. The mitigation column is not empirical: it marks which of the paragraphs below already sketches a concrete hardening path compatible with Equation (2). “FW” = future work. Cluster

Evidence

Residual uncertainty

Mitigation sketch

FW?

p95 L2 drift 6.5%, CV recalibrate when out- per-probe σi + partial 1.9% (Section 3) side env. dtype×kernel grid n=64 MATH + n=48 companion-seed cov- pool to τpool =1.509 (CP no multi-kernel seeds 300– erage dominates 0.0264) 302 Session-level within-session ρ=0.88; non-Gaussian tails un- empirical MVN/copula partial FPR Gaussian-copula 1.9% at measured sim (App. G) k=4 Gemma calibra- nhon =64, v2 σ (32 con- cross-probe corr. unau- τGemma =1.09 (CP partial tion figs) dited 0.0457) Black-box adap- r≤64, ≤ 3000 steps, joint- r>64, longer, full-FT k−L−N margin stable yes tive φ (E4-v2) (4.64−4.99×) White-box SAE frozen, gradient gradient zeroed by “lowest” 4.99 > 4.19 yes adaptive (E-E) through ReLU; 3 restarts ReLU sparsity black-box (this paper) at (0, 0) Gemma utility one λutil =0 catastrophic λutil >0 sweep r=64 Gemma λ-sweep yes frontier point 64/32 public/secret split, secret-probe-aware at- rotation + DP noise yes Public-probe overfit attacker z within 6% tacker unmeasured (App. R) Mask-flip probe up to 40% flip, AUC 1.00 principled rotation per-epoch secret subset yes rotation at strongest atk; worst- schedule (compatible w/ Equacase 0.54 at α=0.001 tion (2)) Economic bind- 0.97× parallel-commit + heterogeneous HW / TEE attestation / append- yes ing (not crypto) Prop. 4.2 structural floor KV-share only log Feature fabrica- F3 exact-optimum 44.24 library-independent at- library rotation (Secno tion w/o M (29.3τ ), Prop. 4.2 LB tackers tion 5.7) 37.11 (App. V) SVIP redesign not evaluated verifier-nonce-after- +1 round-trip, closes yes variant service untested parallel-serve SAE/backend drift MATH-kernel monoculture

27

Table 16: Strategies considered. OOS = out of scope for this paper. Adversary class

Status

Why / what would be needed

Substitute-serve, no adaptation Parallel-commit (serve M ′ , commit M ) Public-corpus φ + LoRA, r=16, 3000 steps Library-aware joint φ + LoRA, r=64, 1500 steps (Qwen3 + Gemma-2-2B + Gemma-29B) Higher-rank LoRA (Qwen3 r=128, MSE/S TAGE A) White-box joint-z gradient LoRA (E-E), r=64, 3000 steps, 3 restart seeds Higher-rank LoRA (r>128) / full finetune beyond tested points Stronger white-box (secretprobe-aware, r>64, > 3000 steps, full-finetune) Full-model retraining matched to published library Non-linear φ (e.g. MLP) Non-audited sessions

in-scope in-scope (one stack) in-scope

Detected at τpool =1.509 on all 7 tested attackers. Costs 0.97× honest wall-clock on RTX 3090; multi-GPU and KV-cache cost models OOS. Held-out per-category joint-z at ≥ 12 × τpool .

in-scope

Four Qwen3 operating points + one Gemma-2B point (Table 4) + one Gemma-2-9B point (row 26, Section 5.5).

in-scope

Joint-z=6.42 (4.25 × τpool ; rows 19a/19b).

in-scope

Best Qwen3 restart joint-z=4.99 — weaker than black-box (Table 10).

OOS

Frontier experiment; needs rank×steps×npub sweep.

OOS

Evaluated E-E covers r=64, 3000 steps, public-probe jointz.

OOS

Requires library rotation analysis + retraining-cost model.

OOS OOS

Heuristic in App. U covers only the linear min-norm fit. Per-session FPR/coverage depends on the audit-sampling policy. App. E, F; 1.7–9B range, scale-stable τ .

Three-backbone replication in-scope (Qwen3-1.7B + Gemma-2-2B + Gemma-2-9B) Multi-tuple (≥ 4) / flagship- OOS class (≥ 70B) generalisation Empirical SVIP side-by-side in-scope comparison

Sweep over backbones / layers / hook variants; 70B SAEs not publicly released at the time of this work. Section 5.6; weaker SVIP in App. W.

28