arXiv CSopen access

MOSAIC: Masked Outsourcing of Secure AI Computations

MOSAIC: Masked Outsourcing of Secure AI Computations James Hsin-yu Chiang

Sheila Zingg

ETH Zurich Switzerland

ETH Zurich Switzerland

Kari Kostiainen

Srdjan Capkun

ETH Zurich Switzerland

ETH Zurich Switzerland

arXiv:2607.29221v1 [cs.CR] 31 Jul 2026

Abstract

centers, where a standard (non-confidential) inference request cannot be practically served by a single GPU node and is distributed across a heterogeneous accelerator pool, with each node contributing to different model layers (attention vs. MLP) and computation phase (prefill vs. decode) [26, 27, 31]. Here, MOSAIC permits a small, user-dedicated TCB (e.g. confidential compute) to be scaled by outsourcing the bulk of the AI computation to untrusted accelerators. Alternatively, extending the TCB would imply secure attestation of the entire accelerator pool and networking stack.

We address the challenge of securely and efficiently outsourcing AI computations from a trusted but computationally weak client to an untrusted but powerful server, in the setting where the client holds both the input and the model, and the server must learn neither. We present MOSAIC, whose core is a novel matrix-multiplication masking protocol that scales to far larger matrices than prior work, enabling the safe outsourcing of modern workloads such as large transformer inference. By introducing small amounts of noise to the multiplication result and thereby relaxing correctness, MOSAIC achieves optimal asymptotic client overhead and concrete runtimes orders of magnitude faster than prior work. Its security reduces to the decisional LWE and LPN assumptions. Because this noise accumulates across the many layers of a transformer, a key technical challenge is bounding error growth; MOSAIC addresses this with an error-scaling mechanism based on random Hadamard rotations. On large 70B transformer models, MOSAIC’s perplexity is comparable to popular quantization approaches and even matches full-precision BF16 inference on HumanEval. Finally, we present an end-to-end implementation1 showing how ideas like MOSAIC can promise a path towards large-scale confidential AI in modern data centers. Non-confidential inference is already distributed across phase (prefill/decode), layer, and time to maximize utilization of heterogeneous hardware, using RDMAlike networking to move activations, cached KV values, and weights across nodes. MOSAIC enables scaling of confidential compute by keeping the trusted computing base (TCB) small and outsourcing the bulk of the AI computation to untrusted accelerators.

1

Layer-by-layer outsourcing. We consider a path towards practical confidential AI computation that considers the total computational cost of a forward-pass across a transformer-style model, and focuses on outsourcing the dominating, linear computations at each model layer to an untrusted accelerator G. In this approach, the non-linear computation is performed by the trusted client C, inducing a slalomlike [28] execution between a trusted client C and untrusted server (accelerator) G, as the forward-pass computation of a transformerstyle model over hundreds of alternating linear and non-linear layers, with expensive linear layers outsourced to computationally powerful G (as shown in Figure 1). Linear computation consists of matrix multiplications of model weights 𝑊 (𝑚 × 𝑛) and activations 𝑋 (𝑛 ×𝑙). Given that repeated linear matrix arithmetic dominates computation during each forwardpass, there has been recent effort to develop cryptographic protocols to securely outsource matrix-matrix multiplications [5, 6]. However, such solutions have significant limitations that prevent their deployment in practice. (1) State-of-the-art in secure outsourcing of matrix multiplication of 𝑊 (𝑚 × 𝑛) and 𝑋 (𝑛 × 𝑙) induces 𝑂 (𝑚𝑛𝜖 𝑙) overhead for the client [6], where 𝜖 must be parameterized for given 𝑛 to satisfy LPN security. This prevents its application to larger modern models, where inner dimensions are very large (e.g. an MLP projection of dim 29 568 in Qwen2.5-72B). An optimal protocol achieves 𝑂 ((𝑚 + 𝑛)𝑙) trusted client complexity, and practical concrete runtimes. (2) Cryptographic proposals generally operate in the discrete, integer ring domain. This requires fixed-point translation to emulate the underlying computation over floating point. Limited dynamic range in standard fixed-point arithmetic has a catastrophic effect on large, modern LLM architectures (Figure 8). No prior work scales AI outsourcing up to modern (70B) models and consequently does not address this problem. (3) Prior work does not investigate practical applications amenable to AI outsourcing that relies on layer-by-layer communication.

Introduction

We investigate the problem of outsourcing AI computations, which are dominated by linear matrix multiplications over billions of model weights. The large dimensions of model weight matrices have given rise to highly parallelized GPU accelerators to speed up computations over billions of model weight values. Our contributions in this work enable practical outsourcing of large AI computations to untrusted accelerators, which are not permitted to learn anything about the AI computation other than approximate topology of the model architecture. Prior state-of-theart solutions do not scale to larger model dimensions [5, 6], or offer concretely impractical runtimes [7, 12, 14, 15, 18, 20, 21, 34] or no formal security [17, 25, 29, 32]. MOSAIC promises a practical pathway towards confidential AI computations in modern AI data 1 Implementation available at https://github.com/jachiang/mosaic

Our contributions address these open challenges as follows. 1

Chiang et al.

Secure Forward Pass (Section 5) Untrusted GPU G

Trusted client C Input tokens

Secure Linear Layer (Section 5)

RMSNorm

ΠSec-Linear

𝑊𝑄 ,𝑊𝐾 ,𝑊𝑉 MatMul

Secure Approx. MatMul (Section 4) Client masks weights:

𝑊

+

+

𝐸𝑤

Client masks activations:

𝑋

+

+

𝐸𝑥

𝑋 (BF16) Attention (softmax, scores)

ΠSec-Linear

Rotate 𝐻𝑋 , cast to 16-bit fixed-pt

𝑊𝑂 MatMul

×𝐿 layers

ΠSec-Approx-MatMul

Residual + RMSNorm

ΠSec-Linear

𝑊gate ,𝑊up MatMul

ΠSec-Linear

𝑊down MatMul

GPU: 𝑌 ′ = 𝑊 ′ 𝑋 ′

Cast to BF16, add bias 𝑏

SiLU gating

𝑌 ≈𝑊𝑋 +𝑏 Client unmasks:

𝑌 ≈𝑊𝑋

Residual

Next layer

Figure 1: Secure, confidential forward pass for a single transformer layer. The trusted client C (left) performs all non-linear operations (attention scores, activations, normalization) locally. All weight matrix multiplications are outsourced to the untrusted GPU G (right) via ΠSec-Linear , which exchanges only masked activations and results at runtime. Secure Private Client layer Practical Security Computational outsourcing inference complexity runtimes Input Model error Perplexity (70B) FHE [12, 15, 34] ✓ ✓ 𝑂 ((𝑚 + 𝑛)𝑙) # ✓ ✓ fixed-point MPC/FSS [7, 14, 18, 20, 21] × ✓ 𝑂 (𝑚𝑛𝑙) # ✓ ✓ fixed-point Slalom [28] ✓ × 𝑂 ((𝑚 + 𝑛)𝑙) # G ✓ × fixed-point 1.8273 4 Obfuscation [17, 25, 29, 32] ✓ × 𝑂 ((𝑚 + 𝑛)𝑙) heuristic / broken fixed-point Trapdoor Matrices [6] ✓ × 𝑂 (𝑚𝑛𝜖 𝑙) G # ✓ ✓ fixed-point MOSAIC (our work) ✓ × 𝑂 ((𝑚 + 𝑛)𝑙) ✓ ✓ fixed-point + 𝑒 1 1.8342 Table 1: Comparison against related work on confidential outsourcing of linear-layer computation. Client complexity is per multiplication of model weights (𝑚 × 𝑛) and activation (𝑛 × 𝑙). Perplexity values are reported for LLaMA-3-70B on WikiText-2. 1𝑒 denotes a low-norm Gaussian induced by our protocol. 2 Our protocol perplexity is considerably lower (better) than standard NF4 4-bit (2.247) quantization baselines and very close to the BF16 reference (1.821). 3 Idealized perplexity: prior-work perplexity entries assume our proposed random-rotation preconditioning that those works do not specify but which could in principle adopt for improved accuracy. 4 Obfuscation-based security is heuristic; we break security of ArrowCloak [29] (USENIX’25).

1) Inner protocol: Outsourcing of approximate MatMul. We propose a cryptographic protocol which permits the trusted client to efficiently outsource large matrix multiplications to an untrusted accelerator G. Our simplified protocol (described in Section 4.1 and illustrated in Figure 1) masks both model weight and input matrices by padding with low-rank uniform matrices and a full-rank, smallnorm gaussian matrix. The security of our full protocol (Section 4.2) can be reduced to LWE and LPN decisional hardness assumptions. Crucially, the matrix multiplication of the encrypted model weights and encrypted inputs can be recovered by the client with a small approximation error, which only induces a minor compromise in model accuracy in practice (Section 5.4). The underlying LWE security and noisy recovery is critical in the scalability of this approach. The low-rank of the uniform padding matrices remains constant across all matrix dimensions polynomial in the security parameter 𝜆. For an inner rank 𝑟 , fixed solely by the security parameter and gaussian noise norm, the optimal client overhead is thus 𝑂 (𝑟 (𝑚 + 𝑛)𝑙) ≈ 𝑂 ((𝑚 + 𝑛)𝑙). We report runtimes showing orders of magnitude runtime improvement compared to

the most efficient prior work [6], and defer the discussion to the related work in Section 2. Matrix (n × n) 36864 × 36864 16384 × 16384 8192 × 8192

4096 × 4096

Protocol MOSAIC MOSAIC [6] MOSAIC [6] [5]† MOSAIC [6] [5]‡

Server (ms) 8.57 2.04 1270 0.90 270.0 – 0.48 57.6 –

Client (ms) 0.29 0.16 47.5 0.12 14.5 – 0.12 6.05 –

Total (ms) 8.86 2.20 1318 1.02 284.5 ∼500 0.60 63.65 ∼130

Client / Total 3.3% 7.3% 3.6% 11.8% 5.1% – 20.0% 9.5% –

Figure 2: Wall-clock runtime of a single secure matrix-vector multiplication for MOSAIC versus prior state-of-the-art [5, 6]. Entries for [5] are placed in the nearest row by dimension: † 8192 × 10000 and ‡ 2048 × 10000. 2) Secure linear layer and noise mitigation. From our secure approximate MatMul protocol, we construct a full linear layer protocol 2

MOSAIC: Masked Outsourcing of Secure AI Computations

which is applied as a drop-in-replacement for the General Matrix Multiply (GEMM) or 𝛼𝑊 𝑋 + 𝛽𝐶. We highlight two challenges. Our secure linear layer (ΠSec-linear ), shown in Figure 1, introduces both fixed-point quantization (16-bit) and protocol noise (𝑊 𝐸𝑥 + 𝐸 𝑤 𝑋 + 𝐸 𝑤 𝐸𝑥 ). We note that the fixed-point quantization error is inherent to all cryptographic protocols [5, 6, 29] that securely outsource linear computation. However, fixed-point impact on model accuracy has not been evaluated in prior, related cryptographic techniques. Let the private inputs 𝑋 and model weights (𝑊 ) be cast to 16-bit integers (with appropriate fixed-point scales) and 𝑊 𝑋 aggregated over a 32-bit integer ring. To mitigate the quantization error, we propose the application of an efficient, random Hadamard rotation [4] to activation and weights matrices. With a series of error accumulation studies on 70B LLM models, we show that even with our protocol noise, the accuracy of the internal activations across all internal layers is close to full precision (BF16) and can exceed common NF4 and INT8 quantization. Further, we report accuracy studies (Perplexity, HumanEval) with our protocol to demonstrate that our (noise) parameter range required for 140bit security is practical even for models with known quantization challenges (e.g., LLaMA3-70B [22]).

on 70B models. We think much of this gap can be explained by the emulation overhead, in which each 32-bit integer MatMul is realized with 10x underlying 8-bit integer MatMul launches. We hope that works such as MOSAIC can inspire AI hardware manufacturers to consider native 32-bit integer support in future accelerator designs.

2

Related work

We outline three categories of related work. (1) Private inference refers to the task of computing inference over client’s private inputs and server’s private model. (2) Secure outsourcing of matrix multiplications assumes a client with both inputs and model weights that wishes to offload the computation to an untrusted server. (3) Model obfuscation refers to the task of keeping proprietary model weights private from an untrusted server or accelerator. Table 1 summarizes our discussion and comparison to related work. Private inference. Private inference is typically realized with cryptographic techniques such as secure multi-party computation (MPC) or fully homomorphic encryption (FHE). MPC techniques focus on secret-sharing (SS) [7, 18, 21] and functional secret-sharing (FSS) [14, 20]. The latter still requires 40s per autoregressive decoding step for a small 13B transformer model, thus not enabling practical runtimes for most applications. Our proposal demonstrates autoregressive decoding runtimes below 1s for a 70B model (Figure 11). Approaches with fully homomorphic encryption [12, 15, 34] employ a powerful, general cryptographic primitive that incurs a very high computational overhead that is multiple orders of magnitude slower than dedicated matrix multiplication protocols. Gazelle proposes a hybrid between MPC and FHE [16]. We note that FHE schemes like CKKS [8] exist which enable approximate arithmetic. These are distinctly different than our approximate secure MatMul protocol, which does not incur the heavy overhead of FHE.

3) Case studies for layer-by-layer outsourcing. We evaluate two application scenarios for our solution. In our primary use-case in Section 6, we report on prefill and decoding runtimes in a datacenter setting, where a trusted computing base is extended with untrusted GPUs, connected with fast interconnects. Here, we highlight two important ongoing trends in AI. Firstly, the leading open and closed sourced frontier models now exceed 1 trillion parameters; larger model dimensions permit ever more efficient outsourcing of AI computations in MOSAIC (Figures 6 and 13), which induces a client overhead that scales optimally with larger matrices, unlike prior work. Secondly, whilst MOSAIC requires layer-by-layer communication of masked activations, a standard, non-confidential inference in practice already induces layer-by-layer communication in today’s AI data centers, as a single inference task for frontier models is no longer feasible on a single cluster, necessitating fast remote direct memory access (RDMA) networking across the data center [26, 31]. Solutions such as MOSAIC can leverage existing ultra-low latency interconnect technology for scalable, confidential AI computation. Indeed our experiments provide evidence that modern GPU-to-GPU interconnects are not the bottle-neck for MOSAIC and that outsourcing AI computations in the data-center is efficient in practice (Figure 11). In a secondary use-case illustrated in Appendix H, we implement a remote decisional inference application, where the expensive AI computation is outsourced to the cloud over the public internet. In this application, only a single forward pass may be sufficient to produce a complicated decision, limiting the number of round-trips. We show that such a decision could be produced in under 5 seconds for large 70B models.

Secure outsourcing of matrix multiplications. Slalom [28] outsources the computationally expensive matrix multiplication operations layer-by-layer to a remote GPU, an outsourcing pattern adopted by this work as well. However, it does not provide privacy for the model weights and requires expensive pre-processing material consumed with each inference run. Our work protects both the model and client inputs. A recent work [5] introduces a matrix encryption technique from pairs of secret dual codes, whose orthogonality cancels the dominant masking cross-term during recovery. The construction focuses on matrix-vector multiplication and operates exclusively over prime fields. This approach can be efficient for models with small dimensions but (comparable) runtimes are superseded by [6]. The authors of [6] introduce a matrix outsourcing technique (trapdoor matrices) that relies on recursive masking with both low-rank, dense and sparse full-rank masking elements - the security of their scheme reduces to decisional LPN. The trusted, outsourcing client incurs a 𝑂 (𝑚𝑛𝜖 𝑙) cost and scales poorly for concrete 𝜖 and larger matrix dimensions required in modern LLM models. Moreover, the paper introduces a conjecture that LPN security holds for small 𝜖 in order for concrete efficiency with large matrix dimension 𝑛, and no concrete protocol parameters are provided to support this in practice. Still, [6] is the closest to our work in concrete performance

Current hardware limitations. Cryptographic protocols such as MOSAIC operate over integer rings, necessitating 32-bit integer arithmetic, which is not natively supported in the fastest cores of modern AI accelerators. We implement an emulation thereof over 8-bit Tensor cores, which results in a ∼3× (decode) to 5–11× (prefill) slow-down versus a baseline, non-confidential inference 3

Chiang et al.

and technique; their recursive masking technique permits an elegant trade-off between memory and online computation, which we show can be adapted as an optimization to our approximate matrix multiplication scheme (section 4.2) whilst retaining optimal client overhead. Our key cryptographic contribution is to introduce a secure approximate matrix multiplication scheme with optimal computational complexity for the client (𝑂 ((𝑚 + 𝑛)𝑙)) and the untrusted accelerator (𝑂 (𝑚𝑛𝑙)), providing accuracy comparable to full precision in many practical AI computation tasks. In Figure 2, we highlight client and server run-times in MOSAIC that are orders of magnitude faster than those reported by [6], which in turn improves on all prior work for matrix dimension 𝑛 ≥ 2048. Note that [6] only reports CPU runtimes and provides no public protocol parameters for reproduction. Client/Total walltime ratios are skewed in MOSAIC by slower Nvidia CUDA core implementation of the client, whilst the server is implemented on the faster Nvidia Tensor cores (See client vs. server implementations in Section 6). We show the theoretical client/total computation complexity ratio in fig. 13 for a large range of matrix dimensions.

Attention projects 𝑋 ∈ R𝑠 ×𝑑model into ℎ query/key/value√heads via 𝑊𝑄 ,𝑊𝐾 ,𝑊𝑉 , and computes per head softmax(𝑄𝑖 𝐾𝑖⊤ / 𝑑𝑘 )𝑉𝑖 , then re-projects through 𝑊𝑂 . The position-wise MLP is a SiLUgated two-layer feed-forward 𝑊down (SiLU(𝑊gate𝑋 ) ⊙ 𝑊up𝑋 ) with 𝑑 ff > 𝑑 model . The seven dense projections (𝑊𝑄 ,𝑊𝐾 ,𝑊𝑉 ,𝑊𝑂 ,𝑊gate,𝑊up,𝑊down ) dominate compute and memory; softmax, RMSNorm, and SiLU are cheap elementwise ops. Our protocol outsources only those seven dense matmuls, leaving non-linear and per-head work on the trusted client.

Model obfuscation. A line of work [17, 25, 29, 32] intends to shield private LLM models from users, by obfuscating weights and hosting de-obfuscation material inside the user’s trusted execution environment (TEE). These works use efficient scaling and permutation techniques to obfuscate the weights, but such approaches lack security guarantees; [29] introduces a deobfuscation attack which exploits similarity between pre-trained open-source models and fine-tuned, private weights. Further, the authors introduce an improved model weights masking scheme called ArrowCloak, for which they claim a reduction to LWE hardness. However, we show that the security of their reduction is broken in Appendix F. Whilst such techniques could in principle be related to matrix multiplication outsourcing, they rely on heuristic security arguments.

4

3

GPU compute and quantisation. Modern NVIDIA GPUs expose two compute paths: CUDA cores (general purpose cores) and tensor cores (matrix-multiply-accumulate units in FP16/BF16/INT8/FP8 with FP32 or INT32 accumulation). Tensor-core throughput exceeds CUDA-core matmul by an order of magnitude; LLM inference is consequently bottlenecked by tensor-core matmul and memory bandwidth. Quantisation casts weights and activations from BF16 to lower precision to shrink the model and accelerate compute; we use NF4, INT8 (and BF16) model accuracy baselines in Section 5.3.

A key task in our confidential AI computation is the outsourcing of linear matrix multiplications in the floating-point domain. We first focus on the outsourcing of matrix multiplications in the the integer ring domain of chosen bit-width, in which our cryptographic protocol operates. Our solution approximately outsources a matrix multiplication 𝑊 𝑋 over the integer ring domain of chosen bit-width - where 𝑊 ∈ Z𝑚×𝑛 is a quantized model weight matrix 2𝜅 and 𝑋 ∈ Z𝑛×𝑙 an activation matrix - to an untrusted accelerator 𝜅 2 G while hiding both 𝑊 and 𝑋 . We present an asymptotically optimal, approximate protocol with 𝑂 ((𝑚 + 𝑛)𝑙) client online cost, and prove post-quantum security under LWE and LPN assumptions. The optimality claim follows directly from the input 𝑛𝑙 and output 𝑚𝑙 dimensions of the outsourcing protocol. In later Section 5, we bridge the integer ring and floating point domains with a secure linear protocol which mitigates the effects of fixed-point quantization and induced protocol error in the floating point domain, in which the interleaved non-linear layers are performed on the trusted client.

Background

Learning with Errors (LWE) and Learning Parity with Noise (LPN). The security of our solution reduces to two standard post-quantum distinguishing assumptions. Decisional LWE [23]: distinguish samples (𝑎, ⟨𝑎, 𝑠⟩ + 𝑒) from uniform, where 𝑎 ∈ 𝑅𝑛 is uniform, 𝑠 ∈ 𝑅𝑛 secret, and 𝑒 is small Gaussian noise; hardness is conjectured for polynomially many samples. Decisional LPN [3]: the same shape but 𝑒 is sparse uniform with noise rate 𝜇 above a secret-rank and security level determined threshold. In this work, we mask weights and activations with a low-rank dense term plus small Gaussian noise (LWE, Section 4.1), then nest the masking by recursively replacing the dense term with an LPN instance of strictly smaller rank (Section 4.2).

4.1

Simple protocol with noisy unmasking

We illustrate the core idea with a simplified construction to outsource the computation of 𝑊 𝑋 , where 𝑊 (𝑚 × 𝑛) and 𝑋 (𝑛 × 𝑙). To mask the quantized weight matrix, the trusted client samples


𝑚×𝑛 𝐿 ← U Z𝑚×𝑟 , 𝑀 ← U Z𝑟2𝜅×𝑛 , 𝐸 𝑤 ← N 0, 𝜎 2 and sets: 2𝜅 𝑊 ′ = 𝑊 + 𝐿𝑀 + 𝐸 𝑤

(2)

Let us denote 𝐿 the left and 𝑀 the middle low-rank matrices in the mask, and 𝐸 𝑤 the low-norm, full-rank Gaussian error term, parameterized by standard-deviation 𝜎. We note that 𝑀 can be a public matrix whilst retaining LWE security. Here, rank 𝑟 denotes the rank of both uniform, low-rank matrices 𝐿 and 𝑀. Critically, 𝑟 is solely a function of security parameter 𝜆 and must be strictly lower than matrix dimensions 𝑚, 𝑛 for client efficiency. Given 𝜆 ≈ 140, inner rank 𝑟 can be set to 1536 (Example 4.1). For a modern model dimension of 16 384 ≫ 1536, the efficiency condition can easily be

Transformer architecture. A decoder-only transformer maps tokens to logits via an embedding, 𝐿 transformer blocks (or model layers), and a final language-model head. Each model layer follows the residual + pre-norm pattern ℎ ℓ = ℎ ℓ −1 + MLP(ℎ ℓ −1 + Attn(RMSNorm(ℎ ℓ −1 ))) ,

Secure approximate matrix multiplication

(1)

combining attention and a√position-wise MLP, each preceded by RMSNorm(ℎ) = ℎ/(∥ℎ∥ 2 / 𝑑) ⊙ 𝑔, which projects ℎ onto a sphere of radius ∥𝑔∥ 2 , collapsing any radial disagreement. 4

MOSAIC: Masked Outsourcing of Secure AI Computations

𝑊

+

𝐿

+

𝑀

𝐸𝑤

=

𝑊′

𝑟 ×𝑛 𝑚 ×𝑛

𝑚 ×𝑟

𝑚 ×𝑛

𝑚 ×𝑛

Figure 3: Structure of the masking scheme, consisting of private weights matrix 𝑊 , left low-rank matrix 𝐿, middle low-rank (r) matrix 𝑀 and full-rank gaussian 𝐸 𝑤 . The activation matrix 𝑋 is masked analogously.

𝑊 ′𝑋 ′

=

𝑊𝑋

+

𝐿

𝑀 (𝑋 +𝐸𝑥 )

+ 𝑊 ′𝑁

𝑟 ×𝑙 𝑚 ×𝑙

𝑚 ×𝑙

𝑚 ×𝑟

𝑅

+

𝐸𝑤 𝑋 + 𝑊 𝐸𝑥 + 𝐸 𝑤 𝐸𝑥

𝑟 ×𝑙 𝑚 ×𝑟

𝑚 ×𝑙

Figure 4: Unmasking the outsourced matrix product. The GPU returns 𝑊 ′𝑋 ′ , which expands into the desired product 𝑊 𝑋 (blue), low-rank correction terms (green), and small-norm noise (red). The client efficiently removes the green terms in 𝑂 ((𝑚 + 𝑛)𝑙) using precomputed material, recovering 𝑊 𝑋 up to a small approximation error. Optimal scalability for larger dimensions. Crucially, LWE security is robust against polynomially many adversarial samples. This means the LWE parameters (rank 𝑟 and noise of standard deviation 𝜎) are fixed by the security level and do not need to grow with the matrix dimensions 𝑚, 𝑛, 𝑙. Concretely, for 140-bit security, we can fix 𝑟 = 1536 (a GPU-friendly dimension) and gaussian noise with stddev 𝜎 = 0.5 ([2]) regardless of whether the matrix has rank 4096 or 29568. This constitutes a very small norm Gaussian relative to the full 32-bit integer ring we instantiate our protocol in. Whilst the rank 𝑟 is fixed by the choice of security parameter 𝜆, the pure LWE unmasking for client C still costs 𝑂 (𝑟 (𝑚 + 𝑛)𝑙) and is linear in 𝑟 . Our full protocol in Section 4.2 improves concrete efficiency further by replacing the uniform, low-rank components with structured, sparse masks that are cheaper to undo.

met (Figure 6). For smaller model dimensions, we can further reduce the inner masking rank in our full protocol, detailed in Section 4.2. Note that the masked matrix 𝑊 ′ is now indistinguishable from a uniformly random matrix from the view of the untrusted accelerator G. The security of the encryption reduces to Learning With Errors (LWE) (lemma 1). Whilst the cost of masking model weights is 𝑂 (𝑟𝑚𝑛), it only has to be performed once for all matrix multiplications with fixed 𝑊 . The input matrix is masked
similarly. For all inference runs, we sample a single 𝑁 ← U Z𝑛×𝑟 2𝜅 . Then, for each individual inference

𝑛×𝑙 pass, we sample 𝑅 ← U Z𝑟2𝜅×𝑙 and 𝐸𝑥 ← N 0, 𝜎 2 and set: 𝑋 ′ = 𝑋 + 𝑁 𝑅 + 𝐸𝑥 Again, let 𝑁 be the middle and 𝑅 denote the right low-rank masking matrices, where 𝑁 can be a public matrix. The client C sends 𝑊 ′, 𝑋 ′ to the untrusted accelerator G, which computes 𝑌 ′ = 𝑊 ′𝑋 ′ and returns the encrypted result 𝑌 ′ back to the client. In terms of the individual matrix components, this gives 𝑌 ′ = (𝑊 + 𝐿𝑀 + 𝐸 𝑤 )(𝑋 + 𝑁 𝑅 + 𝐸𝑥 )

Indistinguishability. It is easy to see that the masked weights (and analogously activation) matrices are computationally indistinguishable from uniform. We provide the proof in Appendix A.3.

(3)

Lemma 1. The LWE mask is computationally indistinguishable from uniform: 𝑐 𝐿𝑀 + 𝐸 𝑤 ≈ 𝑈 .

Expanding: 𝑌 ′ = (𝑊 +𝐿𝑀 +𝐸 𝑤 )𝑋 + (𝑊 +𝐿𝑀 +𝐸 𝑤 )𝑁 𝑅 + (𝑊 +𝐿𝑀 +𝐸 𝑤 )𝐸𝑥 (4) = (𝑊 + 𝐿𝑀 + 𝐸 𝑤 )𝑋 + 𝑊 ′ 𝑁 𝑅 + (𝑊 + 𝐿𝑀 + 𝐸 𝑤 )𝐸𝑥

(5)

= 𝑊 𝑋 + 𝐿𝑀 (𝑋 + 𝐸𝑥 ) + 𝑊 ′ 𝑁 𝑅 + 𝐸 𝑤 𝑋 + 𝑊 𝐸𝑥 + 𝐸 𝑤 𝐸𝑥

(6)

As a consequence, for any 𝑊 the masked weight 𝑊 ′ = 𝑊 +𝐿𝑀 + 𝐸 𝑤 is computationally indistinguishable from uniform. Proof sketch 𝑐 in Appendix A.3. Analogously for 𝑁 𝑅 + 𝐸𝑥 ≈ 𝑈 .

The client removes two correction terms: Firstly, 𝐿𝑀 (𝑋 + 𝐸𝑥 ), which costs 𝑂 (𝑟 (𝑚 + 𝑛)𝑙); 𝑀 (𝑋 + 𝐸𝑥 ) induces client computational complexity 𝑂 (𝑟𝑛𝑙), and subsequent left-multiplication 𝑂 (𝑟𝑚𝑙). The second correction term (𝑊 ′ 𝑁 )𝑅 is also a rank 𝑟 computation if 𝑊 ′ 𝑁 ∈ F𝑚×𝑟 is precomputed once and reused, costing 𝑂 (𝑚𝑟𝑙) 𝑝 per subsequent matmul. Since both 𝑊 ′ and public matrix 𝑁 can be given to untrusted G, 𝑊 ′ 𝑁 can also be outsourced. This process is illustrated in Figures 3 and 4, and yields 𝑌 ≈ 𝑌 ′ − 𝐿𝑀 (𝑋 + 𝐸𝑥 ) − (𝑊 ′ 𝑁 )𝑅 = 𝑊 𝑋 + 𝐸 𝑤 𝑋 +𝑊 𝐸𝑥 + 𝐸 𝑤 𝐸𝑥 (7)

Bounded protocol error. We first analytically bound the protocol error term in Theorem 1 with proof and supporting lemmas in Appendix A.1. In subsequent Section 5, we illustrate how to effectively mitigate approximation errors in the context of full AI computations via an efficient random rotation of both 𝑊 and 𝑋 . Unmasking of 𝑊 ′𝑋 ′ leaves the following error terms in the result: 𝑊 𝐸𝑋 + 𝐸𝑊 𝑋 + 𝐸𝑊 𝐸𝑋 , where 𝐸𝑊 ∈ Z𝑚×𝑛 and 𝐸𝑋 ∈ Z𝑛×𝑙 2𝜅 2𝜅 are discrete Gaussian noise matrices with parameter 𝜎 (distribution

N Z2𝜅 ; 0, 𝜎 2 over Z2𝜅 , with Pr[𝑋 = 𝑘] ∝ exp −𝑘 2 /(2𝜎 2 ) ).

The three trailing error terms remain as an error in the result. For fixed security level 𝜆, the client complexity is 𝑂 ((𝑚 + 𝑛)𝑙).

𝑛×𝑙 𝑚×𝑛 Theorem 1. Let 𝑊 ∈ Z𝑚×𝑛 2𝜅 , 𝑋 ∈ Z2𝜅 , and let 𝐸𝑊 ∈
Z2𝜅 , 2 . Then 𝜅 𝐸𝑋 ∈ Z𝑛×𝑙 have independent entries drawn from N Z ; 0, 𝜎 2 2𝜅

5

Chiang et al.

Protocol Π Sec-Approx-MatMul : Secure Approximate MatMul.

instance (as described in Section 4) has a fixed rank determined by security parameter 𝜆, and is independent of matrix dimensions. The protocol [6] in original form has a mask with rank that must scale with matrix dimensions. We obtain trusted client efficiency of 𝑂 ((𝑚 + 𝑛)𝑙), rather than 𝑂 (𝑚𝑛𝜖 𝑙) in [6]. For intuition, we replace the left private 𝐿 matrix from our naive LWE-only protocol (Equation (2)) with an LPN instance 𝐿2 𝑀2 + 𝑆 2 , that is computationally indistinguishable from uniform 𝐿.

Parties: Trusted client C and untrusted accelerator G . Init W: Compute one-time masking of weights matrix. 𝑟 ×𝑟

𝑚×𝑟𝑑

×𝑟

, 𝑀𝑖 ∈ [𝑑 ] ← Z2𝜅𝑖 𝑖 −1 , 𝑁𝑖 ∈ [𝑑 ] ← Z2𝜅𝑖 −1 𝑖 ,   𝑟 ×𝑟 Z2𝜅𝑖 𝑖 −1 ; 𝜇𝑖 Í −1 (2) C computes: 𝑈 𝑤 = 𝐿 (𝑀𝑑 𝑀𝑑 −1 ...𝑀1 ) + 𝑑𝑖=1 𝑆𝑖+1 (𝑀𝑖 𝑀𝑖 −1 ...𝑀1 ) + 𝐸 𝑤 . (3) C stores: 𝑁 1 𝑁 2 ...𝑁𝑖 and 𝐽𝑖 = (𝑊 + 𝐸 𝑤 )𝑁 1 𝑁 2 ...𝑁𝑖 for 𝑖 ∈ [𝑑 ] . (1) C samples: 𝐿 ← Z2𝜅

𝑟

𝐸 𝑤 ← N (Z𝑚×𝑛 , 𝜎 2 ) , 𝑆𝑖 ∈ [2:𝑑 ] ← S 2𝜅

(4) C stores: 𝑎 ← Z𝑛2𝜅 , 𝑌 chk ← 𝑎𝑇 𝑊 ′ .

𝑐

𝑊 ′ = 𝑊 + 𝐿1 𝑀1 + 𝐸 𝑤 ≈ 𝑊 + (𝐿2 𝑀2 + 𝑆 2 )𝑀1 + 𝐸 𝑤

(5) C stores 𝑊 ′ = 𝑊 + 𝑈 𝑤 and 𝑀𝑖 𝑀𝑖 −1 ...𝑀1 for 𝑖 ∈ [𝑑 ] on untrusted G . Online phase: On fresh input 𝑋 , securely outsource 𝑊 𝑋 to G and recover result. (1) Init X: C computes and stores masking material consumed  by X.  𝑟 ×𝑙

where 𝐿2, 𝑀2 are dense and uniform and 𝑆 2 a sparse matrix (sparse LPN error). The rank of 𝐿2 𝑀2 is strictly smaller than the original 𝐿 (𝑟 2 < 𝑟 1 ), allowing us to decrease the internal rank of the non-error masking term. We continue to apply the substitution to 𝐿2

×𝑟

, 𝐸𝑥 ← N (Z𝑛×𝑙 , 𝜎 2 ) and 𝑇𝑖 ∈ [2:𝑑 ] ← S Z2𝜅𝑖 −1 𝑖 ; 𝜇𝑖 . 2𝜅 Í𝑑 −1 (b) 𝑈𝑥 = 𝑁 1 𝑁 2 ...𝑁𝑑 𝑅 + 𝑖=1 𝑁 1 𝑁 2 ...𝑁𝑖 𝑇𝑖+1 + 𝐸𝑥 . Í −1 (c) 𝐾 = 𝐽𝑑 𝑅 + 𝑑𝑖=1 𝐽𝑖 𝑇𝑖+1 (Unmasking pre-computation). (2) C forwards to G : 𝑋 ′ = 𝑋 + 𝑈𝑥 . (3) G returns 𝑌 ′ = 𝑊 ′ 𝑋 ′ and 𝑄𝑖 = (𝑀𝑖 𝑀𝑖 −1 ...𝑀1 )𝑋 ′ for 𝑖 ∈ [𝑑 ] . 𝑟

(a) 𝑅 ← Z2𝑑𝜅

𝑐

𝑊 ′ ≈ 𝑊 + ((𝐿3 𝑀3 + 𝑆 3 )𝑀2 + 𝑆 2 )𝑀1 + 𝐸 𝑤

(4) C asserts 𝑌 chk 𝑋 = 𝑎𝑇 𝑌 ′ and aborts otherwise. (5) C outputs 𝑌 ≈ 𝑌 ′ − 𝐾 − 𝐿𝑄𝑑 −

More generally over multiple substitution steps

Í𝑑 −1

𝑖=1 𝑆𝑖+1 𝑄 𝑖 (Unmask Y).

𝑐

𝑊 ′ ≈ 𝑊 + (((𝐿𝑑 𝑀𝑑 + 𝑆𝑑 )𝑀𝑑 −1 + 𝑆𝑑 −1 )...𝑆 2 )𝑀1 + 𝐸 𝑤

Figure 5: Secure approximate matrix multiplication outsourcing protocol.

= 𝑊 + 𝐿(𝑀𝑑 𝑀𝑑 −1 ...𝑀1 ) +



Pr |(𝐸𝑊 𝑋 + 𝑊 𝐸𝑋 + 𝐸𝑊 𝐸𝑋 ) 𝑗𝑘 | > 𝑡 ≤ 2exp −

2

2𝜎 2 (∥𝑥𝑘 ∥ 22 + 𝑤 𝑗 2 ) up to 𝑒 −Ω (𝑛) , where 𝑤 𝑗 is the 𝑗-th row of 𝑊 and 𝑥𝑘 is the 𝑘-th column of 𝑋 . Proof in Appendix A.2. Leakage & integrity. The untrusted accelerator G clearly learns the matrix dimensions and invocation of each secure MatMul outsourcing. Without padding matrices, the model architecture can be inferred. With padding, an upper-bound on the number of model layers and matrix dimensions is leaked, at the cost of additional computation. A naive padding scheme could add (1) dummy outsourced linear layers or (2) padded output 𝑚 or input 𝑛 dimensions. Option (1) incurs additional communication and client masking per dummy layer, whilst option (2) increases the per-query client complexity by 𝑂 (𝑚 ′𝑙) (output padding 𝑚 ′ ) or 𝑂 (𝑛 ′𝑙) (input padding 𝑛 ′ ), where 𝑙 is the batch (token) dimension; padding the batch by 𝑙 ′ analogously adds 𝑂 ((𝑚 + 𝑛)𝑙 ′ ). Integrity can be straightforwardly achieved by applying wellknown Freivalds’ algorithm for asserting correctness of 𝑊 𝑋 = 𝑌 computed by the untrusted GPU. We reproduce this correctness check in Appendix B for the reader’s convenience.

4.2

𝑆𝑖+1 (𝑀𝑖 𝑀𝑖 −1 ...𝑀1 ) + 𝐸 𝑤

(9)

Note the rank of the dense term 𝐿(𝑀𝑑 𝑀𝑑 −1 ...𝑀1 ) has been reduced to 𝑟𝑑 < 𝑟𝑑 −1 < ... < 𝑟 1 for 𝐿 as a dense 𝑚×𝑟𝑑 matrix. This ultimately permits a trade-off between storage and client runtime, by reducing the inner rank of matrix computations. We illustrate this trade-off concretely in Example 4.1 and Table 2. We emphasize that inner dimensions 𝑟𝑖 are solely a function of security parameter 𝜆, and independent of the matrix dimensions. Further, the terms 𝑆𝑖+1 (𝑀𝑖 𝑀𝑖 −1 ...𝑀1 ) for 𝑖 ∈ [1 : 𝑑 − 1] are strictly over sparse matrices, with row sparsity also independent of the matrix sizes. During the online phase, the input matrix of dimension 𝑛 × 𝑙 is masked analogously, naturally imposing 𝑂 (𝑛𝑙) online runtime.

!

𝑡2

𝑑 −1 ∑︁ 𝑖=1

entry √︃ ( 𝑗, 𝑘) of the total error 𝐸𝑊 𝑋 + 𝑊 𝐸𝑋 + 𝐸𝑊 𝐸𝑋 is approximately 2 𝜎 ∥𝑥𝑘 ∥ 22 + 𝑤 𝑗 2 -sub-Gaussian: for all 𝑡 > 0, 

(8)

𝑋 ′ = 𝑋 + 𝑁 1 𝑁 2 ...𝑁𝑑 𝑅 +

𝑑 −1 ∑︁

𝑁 1 𝑁 2 ...𝑁𝑖𝑇𝑖+1 + 𝐸𝑥

(10)

𝑖=1

The (public) matrix product chains can be precomputed during initialization and outsourced. The masking operation has 𝑂 (𝑛𝑙) ≈ 𝑂 (𝑛𝑙 + 𝑛𝑟𝑑 𝑙 + 𝑟 1 𝜇 2𝑟 2𝑙 + ... + 𝑟𝑑 −1 𝜇𝑑 𝑟𝑑 𝑙) complexity, where sparsity 𝜇𝑖 and 𝑟𝑖 are fixed by 𝜆 and independent of 𝑚, 𝑛, 𝑙. We note that the masks for quantized activation 𝑋 can be computed independently from 𝑋 . Unmasking. Upon receiving 𝑊 ′𝑋 ′ from the untrusted accelerator G, the client removes all cross terms with low-rank dense and sparse matrices in 𝑂 (𝑚𝑙), whilst retrieving a result with the same error as the simplified LWE-only protocol in Equation (7). Let 𝑊 ′𝑋 ′ = (𝑊 + 𝑈 𝑤 )(𝑋 + 𝑈𝑥 ) = 𝑊 𝑋 + 𝑊 𝑈𝑥 + 𝑈 𝑤 𝑋 + 𝑈 𝑤 𝑈𝑥 , where 𝑈 𝑤 and 𝑈𝑥 are the matrix masks from Equation (9) and Equation (10) respectively. To recover 𝑊 𝑋 , the trusted client must remove the cross-terms, or a close approximation thereof. Firstly, consider term 𝑊 𝑈𝑥 ,

Full protocol using noisy recursive masking

Next, we describe a full protocol, specified in Figure 5, that takes advantage of a recursive masking technique from [6] by recursively replacing dense, uniform matrices of low-rank with LPN instances of ever decreasing rank. In contrast to [6], the outer LWE

𝑊 𝑈𝑥 = (𝑊 𝑁 1 𝑁 2 ...𝑁𝑑 )𝑅 +

𝑑 −1 ∑︁ 𝑖=1

6

(𝑊 𝑁 1 𝑁 2 ...𝑁𝑖 )𝑇𝑖+1 + 𝑊 𝐸𝑥

(11)

MOSAIC: Masked Outsourcing of Secure AI Computations

Measured efficiency of outsourcing (𝑙 = 1)

This term can be efficiently computed up to 𝑊 𝐸𝑥 by the client during the online phase, given pre-computation of the chain products 𝑊 𝑁 1 𝑁 2 ...𝑁𝑖 , amortized over all multiplications over the same weight matrix. The client complexity to compute this term is 𝑂 (𝑚𝑙) = 𝑂 (𝑚𝑟𝑑 𝑙) + 𝑂 (𝑚𝜇2𝑟 2𝑙) + ... + 𝑂 (𝑚𝜇𝑑 𝑟𝑑 𝑙) as 𝑟 2, 𝑟 3 ..., 𝑟𝑑 , 𝜇 2, 𝜇3, ..., 𝜇𝑑 are fixed by the security parameter and not by matrix dimension 𝑚, 𝑛, 𝑙. The cross-term 𝑈 𝑤 𝑋 ′ = 𝑈 𝑤 (𝑋 + 𝑈𝑥 ) is computed up to error 𝐸 𝑤 𝑋 + 𝐸 𝑤 𝐸𝑥 . To illustrate this, consider the following expansion.

Trusted C runtime: outsourced / local 𝑊 𝑋

𝑈 𝑤 𝑋 ′ = 𝐿(𝑀𝑑 𝑀𝑑 −1 ...𝑀1𝑋 ′ ) +

𝑑 −1 ∑︁

attn proj MLP down MLP up LLaMA-3.2 1B Gemma-2-27B LLaMA-4 Maverick 400B Kimi-K2.5 DeepSeek-V3 LLaMA-3.1 70B GPT-3 175B LLaMA-3.1 405B

0.4

𝑆𝑖+1 (𝑀𝑖 𝑀𝑖 −1 ...𝑀1𝑋 ′ ) + 𝐸 𝑤 𝑋 ′

0.3

0.2

𝑖=1

0.1

(12) Here, the multiplicative terms 𝑀𝑖 𝑀𝑖 −1 ...𝑀1𝑋 ′ can be securely outsourced to the untrusted accelerator G. Thus, the first and second terms above can be computed by the client in 𝑂 (𝑚𝑙) = 𝑂 (𝑚𝑟𝑑 𝑙) + 𝑂 (𝑚𝜇 2𝑟 2𝑙) +...+𝑂 (𝑚𝜇𝑑 𝑟𝑑 𝑙) time. The error term expands to 𝐸 𝑤 𝑋 ′ = 𝐸 𝑤 𝑋 + 𝐸 𝑤 𝑈𝑥 , of which 𝐸 𝑤 𝑋 is left as protocol error. We expand the trailing term 𝐸 𝑤 𝑈𝑥 further;

𝐸 𝑤 𝑈𝑥 = (𝐸 𝑤 𝑁 1 𝑁 2 ...𝑁𝑑 )𝑅 +

𝑑 −1 ∑︁

(𝐸 𝑤 𝑁 1 𝑁 2 ...𝑁𝑖 )𝑇𝑖+1 + 𝐸 𝑤 𝐸𝑥

2

4

6

8

10

12

14

16

Rank of Model Weight Matrix (×103 )

Figure 6: Outsourcing efficiency: Measured client runtime vs. local 𝑊 𝑋 runtime (𝑋 as vector, 𝑙 = 1) for various model weight dimensions. The mask initialization is batched for 512 invocations and amortized over each X. One-time initialization of W is excluded.

(13)

𝑖=1

Here, terms 𝐸 𝑤 𝑁 1 ...𝑁𝑖 can be preprocessed by the client and amortized over all matrix multiplications over the quantized model weights, whilst 𝐸 𝑤 𝐸𝑥 is left as protocol error. Thus the trusted client complexity of computing this is 𝑂 (𝑚𝑙) = 𝑂 (𝑚𝑟𝑑 𝑙)+𝑂 (𝑚𝜇2𝑟 2𝑙)+...+ 𝑂 (𝑚𝜇𝑑 𝑟𝑑 𝑙): again, rank 𝑟𝑖 and sparsity 𝜇𝑖 are fixed and independent of matrix dimensions. Overall, this represents an optimization to reduce the rank of dense matrix multiplications at runtime compared to the simplified protocol in Section 4.1. The final resulting protocol error is 𝑊 𝐸𝑥 + 𝐸 𝑤 𝑋 + 𝐸 𝑤 𝐸𝑥 , identical to the naive protocol in Section 4.

𝑑

C compute/(𝑚+𝑛) 𝑙

LWE only + LPN-1 + LPN-2

𝑟 1 = 1536 𝑟 2 +𝑡 2 = 1104 𝑟 3 +𝑡 2 +𝑡 3 = 812

C state (𝑚+𝑛) 𝑟 1 + 𝑚 𝑟 1 = 1536 (2𝑚+𝑛) (𝑚+𝑛)(𝑟 1 +𝑟 2 ) + 𝑚 𝑟 2 = 2560 (𝑚+𝑛) + 1024 𝑚 (𝑚+𝑛)(𝑟 1 +𝑟 2 +𝑟 3 ) + 𝑚 𝑟 3 = 3212 (𝑚+𝑛) + 652 𝑚

Table 2: Client compute/storage tradeoff of mask nesting.

Under decisional LWE and LPN, the nested LWE+LPN mask is computationally indistinguishable from uniform (Lemma 4); hence for any 𝑊 the masked matrix 𝑊 ′ is indistinguishable from uniform.

Example 4.1. We provide an example parameterization for security level 𝜆 = 140. Security of the LWE and LPN instances were parameterized with the LWE [2] and LPN [33] estimators by Albrecht et al. and Yu et al. respectively. We assume 𝑑 = 3 for the number of nested LWE/LPN instances.

Theorem 2 (Security of ΠSec-Approx-MatMul ). Under decisional LWE and LPN: (1) (Privacy.) The adversary G’s view (𝑊 ′, 𝑋 ′ ) is computationally indistinguishable from uniform over Z𝑚×𝑛 × Z𝑛×𝑙 2𝜅 2𝜅 . (2) (Integrity.) See Appendix B.

• LWE: 𝑟 1 = 1536, 𝜎 = 0.5 • LPN-1: 𝑟 2 = 1024, 𝑁 2 = 1536, 𝑡 2 = 80 • LPN-2: 𝑟 3 = 652, 𝑁 3 = 1024, 𝑡 3 = 80

𝑐

Proof. (Privacy.) Lemma 4 in Appendix A.4 gives 𝑊 ′ ≈ 𝑈𝑚×𝑛 under decisional LWE and LPN. The activation mask in Equation (10) follows the same nested LWE+LPN form (with roles of 𝐿, 𝑀𝑖 , 𝑆𝑖 , 𝐸 𝑤 taken by 𝑅, 𝑁𝑖 ,𝑇𝑖 , 𝐸𝑥 ) and is sampled independently, so 𝑐 an analogous hybrid argument yields 𝑋 ′ ≈ 𝑈𝑛×𝑙 . □

Let 𝑟𝑖 be the secret dimension and 𝑁𝑖 the number of permitted adversarial samples (polynomially many for LWE). For LWE, 𝜎 denotes the standard deviation of the dense, Gaussian noise term. For LPN instances, 𝑡 denotes the hamming weight of the noise vector.

4.3

Compute-storage tradeoff of the LPN nesting. Each additional LPN level shrinks the dense rank that the client multiplies through (𝑟 1 → 𝑟 2 → 𝑟 3 ) at the cost of one extra chain product 𝑁 1 · · · 𝑁𝑖 to cache and one extra sparse mask 𝑆𝑖+1 of Hamming weight 𝑡𝑖+1 . Table 2 makes this concrete for the parameters above, omitting (i) Í the small sparse-mask client state 𝑑𝑖=2 𝑟𝑖 𝑡𝑖 and (ii) the GPU-side chain products (𝑟 1 + · · · +𝑟𝑑 ) 𝑛.

Outsourcing efficiency in ΠSec-Approx-MatMul

Efficiency of a matrix multiplication outsourcing protocol relies on a smaller ratio of (1) trusted client protocol overhead to (2) the cost of the client computing 𝑊 𝑋 locally. This depends on both the concrete size of the matrix dimensions (larger is better), as well as the ratio of matrix dimensions (balanced dims are better, given a fixed element count). For a theoretical assessment of this client 7

Chiang et al.

efficiency ratio, we illustrate this in Figure 13 in the appendix for protocol parameters in Example 4.1 and raw computational work (not run-time). To experimentally evaluate the efficacy of client outsourcing for different model dimensions in practice, we show the ratio of (1) trusted client protocol runtime to (2) the runtime of computing 𝑊 𝑋 locally in Figure 6 with our protocol implementation and hardware setup described in Section 6. Overall, model weight matrices with higher rank are clearly more amenable to outsourcing with our protocol than those with lower rank, validating the theoretical scalability of our protocol. We note an implementation nuance: for a given rank, the client C efficiency in the implemented protocol can be observed to be positively correlated to the matrix output dimension 𝑚; MLP down projection matrices offer the best outsourcing efficiency in our experiments. We partially attribute this artifact to our implementation and NVIDIA runtime environment, where the client masking/unmasking computation over low-rank and sparse matrix elements is implemented on the slower, yet more general CUDA cores, and the full-rank 𝑊 𝑋 is optimized to run on the faster Tensor path. The trusted client implementation is likely more sensitive to increased input matrix dimensions, for which it must perform additional work on the slower CUDA path. As future model dimensions increase, we anticipate outsourcing to become significantly more worthwhile, as shown in Figure 13.

this arises from a particular sensitivity of fixed-point arithmetic to large floating-point ranges. Thus, we adapt an approach from model quantization literature [4, 19, 30], that applies a random Hadamard rotation to activation and model weights, which only induces a 𝑂 (𝑛 log(𝑛)) runtime cost. We empirically demonstrate that this approach results in nondestructive error propagation across large (70B) models, and offer accuracy that is comparable with standard 4-bit and 8-bit quantization schemes. Our resulting secure-linear protocol Π Sec-Linear is shown in Figure 7.

5.1

The secure forward pass replaces each linear layer with an invocation of Π Sec-Linear and evaluates every non-linear layer locally on the trusted client. A full secure forward pass on the model can be described as follows. (1) C holds the BF16 activation 𝑋 1 . (2) For ℓ = 1, . . . , 𝐿: C and G jointly invoke 𝑌ℓ ← ΠSec-Linear-Layer (𝑊ℓ , 𝑋 ℓ ) returning a BF16 approximation of 𝑊ℓ 𝑋 ℓ + 𝑏 ℓ to C; C then computes non-linear 𝑋 ℓ+1 ← 𝜙 ℓ (𝑌ℓ ) locally. (3) C outputs activation 𝑋𝐿+1 in BF16. Security and client efficiency follows by sequential composition of the per-layer protocol - we detail asymptotic efficiency in Appendix E for a single secure forward-pass. The per-invocation error was previously bounded in Theorem 1, but is naturally dependent on distributions of actual model weights and activations.

Local attention computation. Efficiency of our protocol rests on a one-time 𝑂 (𝑚𝑛) weight-masking step (Init W) that is amortized across every subsequent forward pass reusing the same 𝑚×𝑛 matrix. During prefill this amortization is unavailable for attention: 𝑄𝑖 , 𝐾𝑖 , and 𝑉𝑖 are all per-query activations, so neither operand of 𝑄𝐾𝑖⊤ nor of score · 𝑉𝑖 is a frozen matrix that can be masked once and reused. During autoregressive decoding the situation is better. The KV cache is built once per sequence and reused for every generated token, so one could in principle run Init W on 𝐾𝑖 and 𝑉𝑖 at cachebuild time and amortize it over all subsequent decode steps. Even with that amortization, however, the head dimension in models today is too small for outsourcing to pay off in practice. The client-side overhead of Π Sec-Approx-MatMul scales as 𝑂 ((𝑚 + 𝑛) 𝑙) against a GPU cost of 𝑂 (𝑚𝑛𝑙), so the useful regime is one where rank 𝑟 = min(𝑚, 𝑛) is large (Figure 13). The outsourced weight projections sit comfortably in this regime with min(𝑚, 𝑛) = 𝑑 model ≥ 4096, whereas per-head attention matmuls only have inner dimension 𝑑𝑘 = 𝑑 model /ℎ ≈ 128 for the model sizes we consider (e.g. LLaMA-3 70B and Qwen2.5-72B both with 𝑑 model = 8192, ℎ = 64).

5

Secure forward pass from ΠSec-Linear

Straw-man solution: error explodes with outliers. We show naive fixed-point scaling over 16-bits with 32-bit accumulations results in catastrophic error in Figure 8 in the early layers. For the chosen Qwen2.5-72B model, both cosine similarity with the full-precision, BF16 activation is shown, as well as the relative ℓ2 error to the BF16 reference, for 20 forward-pass runs on randomly sampled 512token windows from Wikitext-2. Clearly, the noise accumulation from fixed-point quantization alone causes accuracy to collapse in Figure 8. Whilst the cosine and ℓ2 show a mild tendency for recovery in the medium layers (RMSNorm, Non-linear activations can dampen error), this does not suffice for any useful forward pass computation. We explain this phenomena analytically. In ΠSec-linear-layer (Figure 7), each row of 𝑊 (resp. each column of 𝑋 ) is independently scaled by its peak magnitude before quantization to a 𝑏-bit integer range [−𝑞 max, 𝑞 max ]. Concretely, for a row 𝑤 of 𝑊 , the quantization scale is 𝑠 𝑤 = ∥𝑤 ∥ ∞ /𝑞 max , and the quantized representation is 𝑤ˆ = round(𝑤/𝑠 𝑤 ). Activation columns are scaled analogously with 𝑠𝑥 = ∥𝑥 ∥ ∞ /𝑞 max . The window 𝑞 max must also be chosen so that the inner-product accumulator does not wrap around the 𝜅-bit ring: 2 , so we require each entry (𝑊ˆ 𝑋ˆ ) 𝑗,𝑘 has magnitude at most 𝑛 𝑞 max 2 𝑛 𝑞 max ≤ 2𝜅 −1 . After the protocol computes the integer-domain result 𝑌ˆ = 𝑊ˆ 𝑋ˆ + noise, dequantization recovers the approximate floatingpoint output by rescaling each entry: 𝑌 𝑗,𝑘 = 𝑌ˆ𝑗,𝑘 · 𝑠 𝑤𝑗 · 𝑠𝑥𝑘 . The protocol noise, which has fixed variance in the integer domain, is therefore amplified by the product of scales 𝑠 𝑤𝑗 ·𝑠𝑥𝑘 . When either 𝑤

Secure forward pass for LLM models

Our cryptographic protocol ΠSec-Approx-MatMul works over the integer ring domain. In order to leverage this to realize a secure forward pass over an entire model, we must instantiate a secure MatMul outsourcing at each linear layer, and bridge the floating point domain to the inner, integer ring domain. A common approach in cryptographic protocols (Section 2) is to cast to the fixed point domain for the secure linear computation. These works do not consider large modern architectures, for which fixed-point quantization results in destructive error propagation across the many sequential model layers (for 70B models: ≈ 80 blocks, each with 4 linear layers), as illustrated in Figure 8. We show 8

MOSAIC: Masked Outsourcing of Secure AI Computations

Protocol Π Sec-Linear : Secure Linear Layer. Parties: Trusted client C and untrusted accelerator G . Inputs: BF16 weight matrix 𝑊 ∈ R𝑚×𝑛 , optional bias 𝑏 ∈ R𝑚 , and on each invocation a BF16 activation 𝑋 ∈ R𝑛×𝑙 . Output: BF16 activation 𝑌 ≈ 𝑊 𝑋 + 𝑏 . Init W: One-time pre-processing of floating-point weights (reused across inputs). (1) C samples and stores 𝑅 = WHT𝑛 · diag(𝑑 ) with 𝑑 ∈ { −1, +1} 𝑛 . f ← 𝑊 𝑅 ⊤ and quantizes each row of 𝑊 f to a signed 𝑏 -bit integer with per-row scale 𝑠 𝑤 = 𝑤 e𝑗 ∞ /𝑞 max , where the operand window 𝑞 max is (2) C rotates the weights 𝑊 𝑗 chosen so that the inner-product accumulator stays inside Z2𝜅 . (3) C embeds 𝑊ˆ into Z2𝜅 and invokes the Init W phase of Π Sec-Approx-MatMul to produce the masked weights 𝑊ˆ ′ stored on G and the unmasking material stored on C . Online phase: On fresh floating point input 𝑋 , outsource 𝑊 𝑋 + 𝑏 to G and return a floating point result. e ← 𝑅𝑋 and quantizes each column of 𝑋 e to a signed 𝑏 -bit integer with per-column scale 𝑠𝑥 = ∥e (1) C rotates the activation 𝑋 𝑥𝑘 ∥ ∞ /𝑞 max , where 𝑞 max is chosen to 𝑘 ensure inner-product accumulator stays within Z2𝜅 . (2) C and G run the Online phase of Π Sec-Approx-MatMul on (𝑊ˆ , 𝑋ˆ ) over the integer ring Z2𝜅 . (3) C recovers 𝑌ˆ ≈ 𝑊ˆ 𝑋ˆ ∈ Z𝑚×𝑙 up to the protocol noise of Equation (7), and dequantizes entry-wise back to FP from 𝑌 𝑗,𝑘 ← 𝑠 𝑤 𝑗 𝑠𝑥𝑘 𝑌ˆ 𝑗,𝑘 , and adds the bias to output 2𝜅 𝑌 ← 𝑌 + 𝑏.

6

1

Relative ℓ2 error

Cosine similarity

Figure 7: Secure linear layer with input/outputs in the floating-point domain.

0.5

0 −0.1

Qwen2.5-32B

20

40

60

Layer index (ℓ)

LLaMA-3-70B

Configuration

mean

med.

max

mean

med.

max

mean

med.

max

4

INT16-rot-𝜎 = 0.0 INT16-rot-𝜎 = 0.5 INT16-rot-𝜎 = 1.0 INT16-rot-𝜎 = 1.5 INT16-rot-𝜎 = 2.0

0.47 0.66 0.95 1.29 1.65

0.50 0.66 0.91 1.22 1.56

0.96 1.48 2.48 3.55 4.66

0.48 0.77 1.20 1.68 2.18

0.44 0.65 0.98 1.36 1.76

1.65 2.42 3.55 4.97 6.39

0.53 0.85 1.32 1.85 2.40

0.47 0.71 1.07 1.49 1.93

46.4 78.7 122.5 179.7 232.4

3 2 1

Table 3: Error from a single outsourced matrix multiplication under Π Sec-Linear , in relative ℓ2 error ∥𝑌e − 𝑌 ∥ 2 /∥𝑌 ∥ 2 (%) of one outsourced instance of 𝑌 = 𝑊 𝑋 + 𝑏 evaluated on clean BF16 inputs 𝑋 . We report the mean, median, and maximum over all model linear projections (attention/MLP/LM head).

0 0

Qwen2.5-72B

5

0

20

40

60

Layer index (ℓ)

Figure 8: Per-layer error accumulation for Qwen2.5-72B without Hadamard rotation, INT16 quantization at 𝜎 = 0 (no protocol noise). Cosine similarity/relative ℓ2 error vs BF16 reference collapses after 2 layers motivating random Hadamard rotation in our protocol.

above. Their product is therefore also orthogonal, 𝑅 ⊤ 𝑅 = 𝐷 ⊤ 𝐻 ⊤ 𝐻𝐷 = 𝐼, or 𝑥 contains outlier entries, ∥·∥ ∞ is large relative to ∥·∥ 2 , inflating the scales and thus the float-domain noise. In particular, activation vectors in transformer models are inherently concentrated: a small number of entries carry disproportionately large magnitude while the majority remain close to zero. This is not an artefact but a structural necessity - nonlinearities such as softmax concentrate probability mass on a few channels, ReLU and its variants zero out negative entries. Activation outliers of 100× the median magnitude are common in large language models [9].

which gives us two properties we rely on below. First, the matrix product is preserved exactly, 𝑊 𝑋 = (𝑊 𝑅 ⊤ )(𝑅𝑋 ) = 𝑊 ′𝑋 ′

(14)

where 𝑊ˆ = 𝑊 𝑅 ⊤ and 𝑋ˆ = 𝑅𝑋 , so applying the secure matrix multiplication protocol to 𝑊ˆ and 𝑋ˆ recovers the same output as the original product. Second, the rotation preserves ℓ2 norms (∥𝑅𝑤 ∥ 2 = ∥𝑤 ∥ 2 ), so only the ℓ∞ norm is changed by the rotation. The rotation can be applied in 𝑂 (𝑛 log 𝑛) via the fast Walsh–Hadamard transform [11]. We illustrate in Appendix D how this enables the floating-point domain error variance induced by our protocol (Equation (7)) to be 2 2 reduced by factor ( 𝑤 𝑗 2 log 𝑛)/( 𝑤 𝑗 ∞ 𝑛). In Section 5.2 and Section 5.3 we empirically investigate the error contribution by a single securely outsourced linear layer and the error accumulation over all secure linear layers in selected models respectively.

Random Hadamard rotation. Model quantization research [4, 19, 30] has proposed techniques to rotate the activation and spread the activation energy of individual channels more uniformly across all internal model dimensions. We adopt this technique for Πsec-linear-layer , and show this minimizes the resulting error in the floating point output domain, even if intermediary computation is emulated in fixed-point. Let 𝐻 ∈ R𝑛×𝑛 denote the normalized Walsh–Hadamard matrix (𝐻𝐻 ⊤ = 𝐼 ) and let 𝐷 = diag(𝑑) where each 𝑑𝑖 ∈ {+1, −1} is drawn uniformly at random. Then, we denote the randomized rotation 𝑅 = 𝐻 𝐷. Both factors are orthogonal: 𝐷 ⊤ 𝐷 = 𝐼 because each diagonal entry squares to +1, and 𝐻 ⊤ 𝐻 = 𝐼 by the normalization

5.2

Error from a single secure linear layer

We isolate the error introduced by a single outsourced matrix multiplication in Π Sec-Linear , independent of how error propagates 9

Chiang et al.

through the network. For each weight matrix 𝑊 we take the reference activation 𝑋 from a clean BF16 forward pass (no MOSAIC), compute the outsourced product 𝑌e under Π Sec-Linear , and compare it to the exact product 𝑌 = 𝑊 𝑋 via the relative ℓ2 error ∥𝑌e−𝑌 ∥ 2 /∥𝑌 ∥ 2 . Because the input is the clean reference in every case, this measures per-multiplication error in isolation, with no accumulation across layers. Table 3 reports the mean, median, and maximum of this error across all weight projections of Qwen2.5-32B and -70B as well as LLaMA-370B class models at increasing protocol noise 𝜎. A single outsourced multiplication and additive bias is highly accurate: the typical (median) error is 1–2% across all models, and for both Qwen models even the worst-case projection stays below 6.4% at even 𝜎 = 2.0. The large LLaMA-3 maxima are concentrated in the earlylayer value projections (the layer-0 v_proj alone accounts for the reported maximum); excluding the v_proj family, the LLaMA-3 worst case is below 4% for 𝜎 ≤ 1.0. At this noise level, these outliers do not propagate and end-to-end accuracy is ultimately preserved (Table 4); only at 𝜎 = 2.0 does LLaMA-3 degrade. Still, LLaMA-370B is known to be very sensitive to similar noise introduced by standard model weight quantization [22]. In Section 5.3, we illustrate that model accuracy is well-preserved, even when per-layer error contributions propagate and accumulate across the entire model.

obtainable by prior work (Section 2) that computes over integer rings or fields in cryptographic settings. Empirical error accumulation. We measure how per-layer error propagates and accumulates through the full forward pass of Qwen2.5-32B with 64 model layers (Figure 18), and several 80layer models, namely LLaMA-3-70B (Figure 20), Qwen2.5-72B (Figure 19), and DeepSeek-R1-Distill-LLaMA-70B (Figure 21), with different sensitivity to quantization error. We report well-formed, nondestructive error accumulation across the forward pass of models for our protocol (INT16+𝜎) for 𝜎 < 1.5, with final layer error comparable or improving upon INT4 and INT8 quantization (without secure outsourcing). For each model, we run inference passes on 20 WikiText-2 contexts (512 tokens each): a clean reference at BF16, and a perturbed pass in which weights and activations are INT16-quantized, rotated via a random Hadamard matrix (eq. (14)), and sub-Gaussian noise 𝐸 𝑤 , 𝐸𝑥 at standard deviation 𝜎 is sampled and induced by our protocol as described in Equation (7). After each transformer model layer ℓ we capture the residual stream 𝑥 ℓ , the 𝑑 model -dimensional hidden state defined by eq. (1) that is both the output of block ℓ and the input to block ℓ + 1, from both clean and perturbed passes. We report, averaged over all token positions, the per-layer cosine similarity cos(ℓ) =

5.3

Error accumulation across layers

⟨𝑥 ℓbf16, 𝑥 ℓint16−rot−𝜎 ⟩

(15)

𝑥 ℓbf16 2 𝑥 ℓint16−rot−𝜎 2

and relative ℓ2 error 𝑥 ℓint16−rot−𝜎 − 𝑥 ℓbf16 2 / 𝑥 ℓbf16 2 . Each figure shows error accumulation for protocol noise std-deviation 𝜎 ∈ {0.5, 0.7, 1.0, 1.5, 2.0} together with three post-training quantization baselines (NF4, INT8) run through the identical measurement pipeline without rotation or injected noise. At moderate 𝜎 ≤ 1.0 the protocol curves track INT8 and substantially outperform NF4 throughout the forward pass, confirming that the per-layer protocol error accumulates non-destructively across depth. Figure 9 in the main body highlights differences in measured cosine similarity between Qwen2.5-72B and Llama3-70B. We highlight error accumulation behaviors that differ between models below.

Formal end-to-end error accumulation analysis requires quantifying the Lipschitz constant of each transformer sub-layer, which determines how much a perturbation at one layer is amplified before reaching the next. Scaman and Virmaux [24] show that computing the exact Lipschitz constant of even a two-layer ReLU network is NP-hard and the situation for transformer layers is strictly harder. Thus, we rely on empirical study of error propagation in models, and compare accumulated error induced by many layers of ΠSec-Linear-Layer with a baseline of commonly used model quantization techniques intended for efficiency rather than security. We show comparable model accuracy for protocol noise necessary for 140-bit protocol security, as parameterized in Example 4.1.

Quantization-friendly models. The cosine similarity and relative ℓ2 error propagation for chosen models Qwen2.5-72B and Qwen2.532B are well-behaved, as illustrated in Figure 19 and Figure 18 respectively in the Appendix. All INT16-rot-𝜎 runs show error propagation that match or is less than INT4 for all 𝜎 ≤ 2.0, and improve on INT8 for 𝜎 ≤ 1.0. The earlier layers in Qwen2.5-72B show a higher sensitivity to our protocol error, but this effect is nondestructive, as it recovers in both cosine-similarity and ℓ2 errors in later model layers. We attribute this heuristically to both RMSNorm and channel-wise activation functions, that are known to contribute well-formed error accumulation. For the chosen protocol parameter 𝜎 = 0.5 (Example 4.1), this error accumulation is strictly better than INT4 or INT8 quantization schemes across all layers.

Quantization baselines (4-bit NF4, INT8, INT16-rot-𝜎). We compare our protocol against two standard post-training quantization schemes as reference points. 4-bit NF4 [10] uses a non-uniform 4-bit NormalFloat grid whose levels track the quantiles of a standard normal distribution, exploiting the approximately Gaussian distribution of transformer weights. LLM.int8() [9] represents weights and activations as per-channel-scaled 8-bit integers in [−127, 127], with matrix multiplications accumulated in higher precision, as well as weight and activation outliers. We refer to (1) INT4, (2) INT8 and (3) INT16-rot-𝜎 as running model inference with 4-bit NF4, LLM.int8() and our protocol parameterized with gaussian stdev parameter 𝜎 that induces a random hadamard rotation on activation and model weights. Note that for 𝜎 = 0, our protocol converges to the same precision as 16-bit fixedpoint with the additional hadamard rotation treatment, with the latter strictly improving the precision of the computation. Thus, we use the INT16+𝜎=0 baseline to refer to the optimistic precision

Quantization unfriendly models. Llama-3-70B (and derivative DeepSeek-R1-Distill-LLaMA-70B) are known to be sensitive to quantization [22]. Their cosine similarity and relative ℓ2 error accumulation is shown in Figure 20 and Figure 21 respectively. Indeed, in our experiments, INT8 and INT16-rot-𝜎 = 2.0 show destructive 10

MOSAIC: Masked Outsourcing of Secure AI Computations

Qwen2.5-72B

Llama-3-70B 1 Cosine similarity

4-bit NF4 INT8 INT16-rot 𝜎 = 0 INT16-rot 𝜎 = 0.5 INT16-rot 𝜎 = 0.7 INT16-rot 𝜎 = 1.0 INT16-rot 𝜎 = 1.5 INT16-rot 𝜎 = 2.0

Cosine similarity

1

0.98

0.8

0.6 0.96 0

20

40 Layer index (ℓ)

60

0

20

40 Layer index (ℓ)

60

Figure 9: Per-layer error accumulation: Consine similarity between activations of quantized and reference forward-pass (BF16) for 70B-class models (left: Qwen2.5-72B; right: Llama-3-70B). Whilst the two models share similar architectures, Qwen2.5-70B is more resilient against error accumulation. Still, for protocol noise std-deviation range 𝜎 ∈ [0.5, 1.0], our protocol can still be parametized for 140-bit security whilst resulting in well-formed error accumulation in both models. error propagation across layers for both Llama-3-70B and a derivative model DeepSeek-R1-Distill-LLaMA-70B. The relative ℓ2 error is generally large for both INT4 and INT16-rot-𝜎 for 𝜎 ≥ 1.0. For Llama-3-70B (Figure 20), we propose to parameterize our protocol with gaussian samples with parameter 𝜎 = 0.5 for 140-bit security (Example 4.1). For DeepSeek-R1-Distill-Llama-70B (Figure 21), the relative ℓ2 error at the final layer remains very high for all quantizations, making the model a border-line or even suboptimal candidate for quantized computation.

5.4

Qwen2.5-72B

LLaMA-3-70B

Qwen2.5-32B

Configuration

PPL

HE

PPL

HE

PPL

HE

BF16 (reference)

2.201

53.96

1.821

54.57

2.722

46.34

INT16-rot-𝜎 = 0.0 INT16-rot-𝜎 = 0.5 INT16-rot-𝜎 = 1.0 INT16-rot-𝜎 = 1.5 INT16-rot-𝜎 = 2.0

2.201 2.204 2.205 2.221 2.244

54.57 53.66 50.91 51.83 53.35

1.827 1.834 1.864 2.055 3.233

51.83 53.96 45.73 35.98 –

2.723 2.724 2.726 2.734 2.742

46.04 42.38 43.29 43.90 47.56

INT8 (LLM.int8()) NF4 (4-bit)

2.213 2.303

49.70 55.79

425.975 2.247

10.06 45.43

2.736 2.894

48.48 47.56

Table 4: Model accuracy under quantization baselines (INT8, NF4) and the protocol’s INT16-rot-𝜎 noise model at increasing noise levels. Two metrics per model: WikiText-2 perplexity (PPL, lower is better, sequence length 2048, stride 512) and HumanEval pass@1 (HE, higher is better, 𝑘 = 1).

Accuracy of outsourced inference

Table 4 reports model accuracy of an end-to-end forward pass under our protocol against two reference quantization baselines (LLM.int8() and 4-bit NF4), on two 70B-class and one 32B model spanning the quantization spectrum: Qwen2.5-72B and Qwen2.532B (quantization-friendly) and LLaMA-3-70B (quant.-unfriendly). Each model is evaluated on WikiText-2 perplexity (PPL, lower is better) and on HumanEval pass@1 (HE, higher is better, 𝑘 = 1). For the quantization-friendly Qwen2.5-72B, all five INT16-rot-𝜎 configurations stay within 2% of the BF16 reference on PPL across the full 𝜎 ∈ [0, 2] range, and HE on Qwen2.5-72B tracks the BF16 baseline to within ±4 points across 𝜎 ∈ [0, 2]. For Qwen2.5-32B, a potential drop in HE scores may be observed for 𝜎 = 0.5/1.0, that recovers at higher noise levels. For quantization-unfriendly LLaMA-3-70B, our protocol matches PPL scores of BF16 to within 3% for 𝜎 ≤ 1.0 and HE drops gracefully from 54.6 (BF16) to 45.7 at 𝜎 = 1.0; beyond 𝜎 = 1.0 both metrics degrade visibly - PPL grows to 2.055 at 𝜎 = 1.5 and 3.233 at 𝜎 = 2.0. Crucially, the noise parameter required for our 140-bit-secure parameterisation in Example 4.1 sits at 𝜎 = 0.5; for every model in the table there exists a secure parameterization of our protocol which performs at or above NF4/INT8 baselines. The comparison with INT8 on LLaMA-3-70B is informative as a challenging reference model: INT8 fails catastrophically on this model (PPL = 425.97, HE = 10.06) - a sensitivity our protocol does not exhibit because the random Hadamard rotation (Equation (14))

flattens the same outlier distribution that defeats LLM.int8() quantization. For 𝜎 = 0.5 (140-bit security), our protocol retains the accuracy of BF16 on both PPL and HE benchmarks.

6

Confidential AI outsourcing

Here, we show how a small trusted computing base (TCB) tasked with AI computations in a modern AI data-center can support larger AI computations that exceed its trusted resources by adding powerful, untrusted GPU accelerators. In this setting, we demonstrate that MOSAIC is not bottle-necked by communication in the modern AI data center setting where individual GPU’s are networked with fast interconnect technologies. In Appendix H we illustrate a remote application over public networks, which only requires an LLM classification, and does not require communication-intensive autoregressive decoding. Emulation approach. To explore these use-cases, we emulate a real-world implementation with current, limited hardware support. 11

Chiang et al.

Our Π Sec-Approx-Matmul protocol is parameterised over a 32-bit integer ring, which modern AI accelerators do not natively support. Concretely, on Nvidia GPUs the high-throughput Tensor cores that dominate LLM matmul performance accelerate FP16/BF16, FP8, and INT8 datatypes, but not 32-bit integer arithmetic. Native 32-bit integer is available only on the general-purpose CUDA pipeline, which delivers roughly an order of magnitude lower throughput than the Tensor cores and is therefore impractical for the matrix sizes involved in 70B-class models. To remain on the Tensor-core pipeline, we emulate each 32-bit integer multiply–accumulate as a sum of INT8 multiply–accumulates, splitting each 32-bit operand into four 8-bit limbs and reassembling the result in the trusted client. This emulation costs a constant overhead per ring multiplication, a slowdown relative to a hypothetical Tensor-core kernel with native INT32 support, but still markedly faster than falling back to the CUDA. A key detail is that the limbs are unsigned bytes ([0, 255]) but the Tensor-core kernel accepts only signed bytes ([−128, 127]). We bridge this by subtracting 128 from each limb before the matmul, which slides every value into signed INT8 range without losing any information. After the kernel returns, we undo the bias algebraically: expanding (𝑎 − 128) (𝑏 − 128) = 𝑎𝑏 − 128(𝑎 + 𝑏) + 1282 shows that Í Í the unsigned product 𝑝 𝑎𝑝 𝑏 𝑝 equals the signed product 𝑝 (𝑎𝑝 − 128) (𝑏 𝑝 − 128) plus a correction term that depends only on the row and column sums inducing 𝑂 (𝑚 + 𝑛) to apply. The final reduction to Z232 requires no extra work either, since native INT32 addition and bit-shift discard bits above position 31 automatically. The split between the two pipelines is also reflected in the protocol split between trusted client and untrusted GPU. The untrusted GPU performs the dense 𝑂 (𝑚𝑛𝑙) masked MatMul and runs strictly on the Tensor-core pipeline (via the INT8 emulation above). The trusted client, by contrast, performs only low-rank dense and sparse operations of inner rank 𝑟𝑑 or row Hamming weight 𝑡 (Section 4.2); these more general workloads cannot benefit from Tensor-core acceleration, and thus client kernels are implemented for generalpurpose CUDA pipeline at INT32 width.

Local GPU baseline (no protocol) Untrusted GPU (remote MatMul)

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

584 502

584 502

584 501

574 491

574 491

573 490

197

186

191

150

158

200

B

M

L

B

1k

M

L

B

4k

M

L

B

7k

M

L

B

1k

M

L

B

4k

LLaMA-3-70B

M

L

7k

Qwen2.5-72B

Figure 11: Per-token decoding latency for 70B-class models (𝐺 = 3 untrusted GPUs), across 1k/4k/7k token context lengths. Within each context: B = local-GPU baseline (no protocol), M = sharding of individual masked weight matrix, and L = sharding of masked model by layer across untrusted GPUs.

38.7 24.3

16.0 2.7

3.9

8.6

5.7 1.3

3.5

2.4

1.1

5.1

10

7.9

14.8

20

22.3

24.2

30

25.8

40 Latency (seconds)

Trusted GPU (mask/unmask + non-linear) Comm (trusted ↔ untrusted) 36.9

Local GPU baseline (no protocol) Untrusted GPU (remote MatMul)

0 B

M 1k

L

B

M

L

4k LLaMA-3-70B

B

M 7k

L

B

M 1k

L

B

M

L

4k

B

M

L

7k

Qwen2.5-72B

Figure 12: System-prompt prefill runtime for 70B-class models (𝐺 = 3 untrusted GPUs), across 1k/4k/7k system-prompt lengths, split into trusted-GPU walltime, untrusted-GPU remote MatMul, and trusted↔untrusted communication. Within each context: B = local-GPU baseline (no protocol), M = weight matrix shard, and L = model sharding by layer.

RDMA interconnects Trusted GPU 2

Trusted GPU 3

400

0

Untrusted GPU pool

Trusted GPU 1

600

155

Latency (ms / token)

800

Datacenter

𝑘 independent TCBs

Trusted GPU (mask/unmask + non-linear) Comm (trusted ↔ untrusted)

extend to GPU [13] hardware, they are limited in scope, and do not offer the same scaling and performance benefits as modern datacenter architectures optimized for AI inference. Confidential computing is also more expensive due to specialized chips, dedicated hardware resources, attestation key management, enhanced physical security, and other factors. We emphasize that frontier AI inference needs more than a single GPU cluster. A typical inference task is distributed across phase (prefill/decode), model layer (sharding) and time (interleaving between accelerators) to maximize utilization [27, 31]. Data-center accelerators are connected via remote direct memory access (RDMA) [26] technologies, featuring single-digit microsecond latency between

Figure 10: Datacenter outsourcing topology: 𝑘 independent trusted clients, each a small TCB, elastically share a common pool of untrusted GPUs, secure outsourcing AI computations over fast interconnects.

Secure outsourcing in an AI Datacenter. The common approach for confidential computation in datacenters is to deploy applications on a trusted computing base (TCB), realized by confidential computing technologies offered by all major cloud providers today (SGX/TDX/NCC). Whilst confidential hardware architectures now 12

MOSAIC: Masked Outsourcing of Secure AI Computations

GPUs. Such connectivity is enabled by fast intra-rack (e.g. NVLink, PCIe) and intra-data-center interconnects (e.g. Infiniband). Such distributed, AI-native architectures are not directly compatible with the currently available confidential computing technologies. Offering AI inference at scale would imply extending the TCB to the heterogeneous pool of accelerators and their full networking stack, requiring complex attestation key management, enhanced physical security measures, and so on. Each user of secure AI computation would need their models and requests securely isolated from others, while cloud providers would ideally be free to assign workloads across the distributed data center to efficiently use available resources. Secure outsourcing approaches like MOSAIC offer an alternative approach that combines (a) the benefits of modern (heterogeneous and distributed) AI inference architecture with (b) a small TCB for each user (Figure 10). Confidential (and potentially attested) compute represents the scarce resource, and the value of our protocol is that a trusted GPU can elastically capture additional throughput from untrusted accelerators without expanding the trusted computing base.

approach above); native 32-bit-integer support in AI accelerator cores (e.g. NVIDIA Tensor) would significantly reduce this gap. We highlight three observations in prefill (Figure 12) and decode (Figure 11) experiments on 70B class models. Firstly, communication between the trusted and untrusted pools is not the bottleneck under the available inter-GPU bandwidth: it accounts for only ∼4–7% of decode and ∼7–16% of prefill wall-clock, mirroring practicality of the per-layer interaction pattern as in real-world settings [26, 31]. Secondly, the wall-time differences between model-matrix sharding and model-layer sharding across untrusted GPU’s is more pronounced in prefill (Figure 12) than in autoregressive decoding (Figure 11). This is expected as autoregressive decoding is memorybounded; adding more parallel compute offers limited benefits. Third, autoregressive decoding (Figure 11) is essentially contextinvariant at ∼0.5–0.6 s per generated token across 1k–7k context windows, because per-token cost is dominated by the dense linear projections (independent of 𝑠 for 𝑙 = 1); the KV dimensions only enters through attention, which are implemented over the native floating-point domain and negligible at this dimension. Of that pertoken latency, the untrusted GPU accounts for ∼92–93% (∼85–90% remote MatMul, the rest communication), leaving under ∼8% on the trusted client. The one-time init_W weight-masking cost (≈ 55 s per 70B model) is amortized across all subsequent queries over the frozen model weights, quickly becoming a negligible fraction of cumulative wall-clock.

Implementation with fast GPU interconnects. We run our implementation on 70B models on Nvidia H200 with 141 GB VRAM connected with fast NVLink interconnects, featuring 900 GB/s GPUto-GPU transfer speeds. Whilst similar latency can be achieved with RDMA-style interconnects across the data center in practice [26, 27, 31], on-demand GPUs offered by AI cloud providers accessible limit us to such inter-node interconnects. For the 70B models we deploy, we use 1 trusted GPU, that securely outsources computation to 3 untrusted GPUs, where protocol communication is over the aforementioned NVLink GPUto-GPU interconnect. We measure end-to-end prefill and decode for 70B-class models (LLaMA-3, Qwen-2.5) on a deployment with one trusted GPU acting as the trusted client and 3 untrusted GPUs collectively serving the outsourced MatMul calls via the secure forward pass (Figures 11 and 12).

References [1] Nir Ailon and Bernard Chazelle. 2006. Approximate Nearest Neighbors and the Fast Johnson-Lindenstrauss Transform. In 38th Annual ACM Symposium on Theory of Computing (STOC). 557–563. https://dl.acm.org/doi/pdf/10.1145/ 1132516.1132597. [2] Martin R Albrecht, Rachel Player, and Sam Scott. 2015. On the concrete hardness of learning with errors. Cryptology ePrint Archive (2015). https://github.com/ malb/lattice-estimator. [3] Michael Alekhnovich. 2003. More on average case vs approximation complexity. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. IEEE, 298–307. [4] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems 37 (2024), 100213–100240. https://arxiv.org/pdf/ 2404.00456. [5] Fabrice Benhamouda, Caicai Chen, Shai Halevi, Yuval Ishai, Hugo Krawczyk, Tamer Mour, Tal Rabin, and Alon Rosen. 2025. Encrypted matrix-vector products from secret dual codes. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 394–408. . [6] Mark Braverman and Stephen Newman. 2025. Practical Secure Delegated Linear Algebra with Trapdoored Matrices. In Theory of Cryptography Conference (TCC). Springer, 97–118. https://arxiv.org/abs/2502.13060. [7] Hao Chen, Miran Kim, Ilya Razenshteyn, Dragos Rotaru, Yongsoo Song, and Sameer Wagh. 2020. Maliciously secure matrix multiplication with applications to private deep learning. In International Conference on the Theory and Application of Cryptology and Information Security. Springer, 31–59. https://eprint.iacr.org/ 2020/451.pdf. [8] Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. 2017. Homomorphic encryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information security. Springer, 409–437. https://eprint.iacr.org/2016/421.pdf. [9] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. 30318–30332. https: //arxiv.org/abs/2208.07339. [10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2305.14314. [11] Fino and Algazi. 1976. Unified matrix treatment of the fast Walsh-Hadamard transform. IEEE Trans. Comput. 100, 11 (1976), 1142–1146. https://ieeexplore.ieee. org/stamp/stamp.jsp?tp=&arnumber=1674569.

Protocol parameterization. We implement protocol parameters from Example 4.1 for 140-bit security, but omit the recursive LPNmask optimization for masking activations. This maintains security, but saves memory for the client, as the layer-specific matrices 𝐽𝑖 in step 4 of Init W (Figure 5) no longer occupies client memory; this slightly increases the cost of computing the activation mask 𝑈𝑥 = 𝐽𝑑 𝑅 in step 1b of Init X at (model-independent) rank of 1536 instead of 652. Exhaustive parameter optimization is left for future work as it is highly hardware dependent. Model sharding. We implement two parallelization techniques. (1) We shard each masked model matrix across the 3 GPUs, and broadcast each masked activation to all GPUs (model-matrix parallel); the result is gathered by the trusted GPU thereafter. (2) We shard the masked model by contiguous layer ranges across the untrusted GPUs (model-layer parallel). Evaluation results. MOSAIC is roughly 3× (decode) to 5–11× (prefill) slower than running inference for the same 70B models on a single local GPU of the same type, with the higher prefill factor corresponding to layer- rather than matrix-sharding. This overhead largely reflects our emulation of 32-bit integer arithmetic on 8-bit GPU cores (10 INT8 MatMuls per INT32 MatMul; see the emulation 13

Chiang et al.

[12] Yang Gao, Gang Quan, Soamar Homsi, Wujie Wen, and Liqiang Wang. 2024. Secure and efficient general matrix multiplication on cloud using homomorphic encryption: Y. Gao et al. The Journal of Supercomputing 80, 18 (2024), 26394– 26434. [13] Zhongshu Gu, Enriquillo Valdez, Salman Ahmed, Julian James Stephen, Michael Le, Hani Jamjoom, Shixuan Zhao, and Zhiqiang Lin. 2025. NVIDIA GPU confidential computing demystified. arXiv preprint arXiv:2507.02770 (2025). https://arxiv.org/pdf/2507.02770. [14] Kanav Gupta, Neha Jawalkar, Ananta Mukherjee, Nishanth Chandran, Divya Gupta, Ashish Panwar, and Rahul Sharma. 2024. SIGMA: Secure GPT Inference with Function Secret Sharing. In 24th Privacy Enhancing Technologies Symposium (PETS). https://eprint.iacr.org/2023/1269. [15] Xiaoqian Jiang, Miran Kim, Kristin Lauter, and Yongsoo Song. 2018. Secure outsourced matrix computation and application to neural networks. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 1209–1222. https://eprint.iacr.org/2018/1041.pdf. [16] Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. 2018. { GAZELLE } : A low latency framework for secure neural network inference. In 27th USENIX security symposium (USENIX security 18). 1651–1669. https://www. usenix.org/system/files/conference/usenixsecurity18/sec18-juvekar.pdf. [17] Qinfeng Li, Zhiqiang Shen, Zhenghan Qin, Yangfan Xie, Xuhong Zhang, Tianyu Du, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. Translinkguard: safeguarding transformer models against model stealing in edge deployment. In Proceedings of the 32nd ACM international conference on multimedia. 3479–3488. [18] Zhengyi Li, Kang Yang, Jin Tan, Wen-jie Lu, Haoqi Wu, Xiao Wang, Yu Yu, Derun Zhao, Yancheng Zheng, Minyi Guo, et al. 2024. Nimbus: Secure and efficient two-party inference for transformers. Advances in Neural Information Processing Systems 37 (2024), 21572–21600. [19] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406 (2024). https://arxiv.org/pdf/2405.16406. [20] Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and WenGuang Chen. 2023. Bumblebee: Secure two-party inference framework for large transformers. Cryptology ePrint Archive (2023). https: //eprint.iacr.org/2023/1678.pdf. [21] Zoltán Ádám Mann, Christian Weinert, Daphnee Chabal, and Joppe W Bos. 2023. Towards practical secure neural network inference: the journey so far and the road ahead. Comput. Surveys 56, 5 (2023), 1–37. https://eprint.iacr.org/2022/1483.pdf. [22] Minghai Qin. 2024. The uniqueness of llama3-70b series with per-channel quantization. arXiv preprint arXiv:2408.15301 (2024). https://arxiv.org/pdf/2408.15301. [23] Oded Regev. 2009. On lattices, learning with errors, random linear codes, and cryptography. Journal of the ACM (JACM) 56, 6 (2009), 1–40. [24] Kevin Scaman and Aladin Virmaux. 2018. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 31. https://arxiv.org/pdf/1805.10965. [25] Tianxiang Shen, Ji Qi, Jianyu Jiang, Xian Wang, Siyuan Wen, Xusheng Chen, Shixiong Zhao, Sen Wang, Li Chen, Xiapu Luo, et al. 2022. { SOTER } : Guarding black-box inference for general neural networks at the edge. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 723–738. https://www.usenix. org/system/files/atc22-shen.pdf. [26] Spheron. 2026. GPU Networking for AI Clusters: InfiniBand vs RoCE vs SpectrumX Decision Guide. Spheron Network Blog. https://www.spheron.network/blog/ gpu-networking-infiniband-roce-spectrum-x-guide. [27] Yijun Sun, Xudong Liao, Songrun Xie, Hao Chen, Han Tian, Wenxue Li, Yiming Zhang, and Kai Chen. 2026. Multi-stage Flow Scheduling for LLM Serving. arXiv preprint arXiv:2603.17456 (2026). https://arxiv.org/pdf/2603.17456. [28] Florian Tramèr and Dan Boneh. 2019. Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1806.03287. [29] Pengli Wang, Bingyou Dong, Yifeng Cai, Zheng Zhang, Junlin Liu, Huanran Xue, Ye Wu, Yao Zhang, and Ziqi Zhang. 2025. Game of Arrows: On the (In)Security of Weight Obfuscation for On-Device TEE-Shielded LLM Partition Algorithms. In 34th USENIX Security Symposium (USENIX Security 25). 279–298. https://www.usenix.org/system/files/usenixsecurity25-wang-pengli.pdf. [30] Songhao Wu, Ang Lv, Xiao Feng, Yufei Zhang, Xun Zhang, Guojun Yin, Wei Lin, and Rui Yan. 2025. Polarquant: Leveraging polar transformation for efficient key cache quantization and decoding acceleration. arXiv preprint arXiv:2502.00527 (2025). https://arxiv.org/pdf/2502.00527. [31] Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, et al. 2026. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference. arXiv preprint arXiv:2602.21548 (2026). https://arxiv.org/pdf/2602.21548. [32] Rongwu Xu and Zhixuan Fang. 2024. Tempo: Confidentiality preservation in cloud-based neural network training. In 2024 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–10.

[33] Yu Yu, Kang Yang, Xiao Wang, Anyu Wang, Tianrui Wang, Hanlin Liu, Xinpeng Hao, and Juanru Li. 2025. Estimator of LPN problems over any finite fields and power-of-two rings for PCG and MPC applications. https://lpnestimator.com. https://lpnestimator.com/. [34] Itamar Zimerman, Allon Adir, Ehud Aharoni, Matan Avitan, Moran Baruch, Nir Drucker, Jenny Lerner, Ramy Masalha, Reut Meiri, and Omri Soceanu. 2024. Power-Softmax: Towards Secure LLM Inference over Encrypted Data. arXiv preprint arXiv:2410.09457 (2024). https://arxiv.org/abs/2410.09457.

A Proofs A.1 Supporting lemmas The two supporting lemmas referenced by Theorem 1 are stated and proved below. Corollary 1 collects the comparison between their tail bounds that we invoke in the proof of Theorem 1. Lemma 2. Let 𝐸 ∈ Z𝑚×𝑛 have independent entries drawn from 2𝜅
N Z2𝜅 ; 0, 𝜎 2 , and let 𝑋 ∈ Z𝑛×𝑙 2𝜅 be fixed. Then entry ( 𝑗, 𝑘) of 𝐸𝑋 satisfies:   (1) E (𝐸𝑋 ) 𝑗𝑘 = 0. (2) (𝐸𝑋 ) 𝑗𝑘 is 𝜎 ∥𝑥𝑘 ∥ 2 -sub-Gaussian: for all 𝑡 > 0,  
Pr |(𝐸𝑋 ) 𝑗𝑘 | > 𝑡 ≤ 2exp −𝑡 2 /(2𝜎 2 ∥𝑥𝑘 ∥ 22 ) where 𝑥𝑘 is the 𝑘-th column of 𝑋 . Í Proof. Fix ( 𝑗, 𝑘) and write (𝐸𝑋 ) 𝑗𝑘 = 𝑛𝑖=1 𝑒𝑖 (𝑥𝑘
)𝑖 , where 𝑒𝑖 := (𝐸) 𝑗𝑖 are independent samples from N Z2𝜅 ; 0, 𝜎 2 and (𝑥𝑘 )𝑖 are fixed.
(1) By symmetry of N Z2𝜅 ; 0, 𝜎 2 , E [𝑒𝑖 ] = 0 for each 𝑖. By linÍ earity of expectation, E (𝐸𝑋 ) 𝑗𝑘 = 𝑖 (𝑥𝑘 )𝑖 E [𝑒𝑖 ] = 0. (2) A random variable 𝑍 is 𝛼-sub-Gaussian if E [exp (𝑡𝑍 )] ≤
exp 𝛼 2𝑡 2 /2 for all 𝑡 ∈ R. We
first show that each discrete gaussian sample 𝑒𝑖 ∼ N Z2𝜅 ; 0, 𝜎 2 is 𝜎-sub-Gaussian for any 𝜎 > 0. Let
Í 𝜌𝜎 (𝑆) = 𝑘 ∈𝑆 exp −𝑘 2 /(2𝜎 2 ) for any subset 𝑆 ⊆ Z2𝜅 . The MGF of 𝑒𝑖 is ∑︁ E [exp (𝑡𝑒𝑖 )] = exp (𝑡𝑘) · Pr[𝑒𝑖 = 𝑘] 𝑘

=

∑︁
1 exp 𝑡𝑘 − 𝑘 2 /(2𝜎 2 ) 𝜌𝜎 (Z2𝜅 )

(16)

𝑘 ∈Z2𝜅

Í Let us denote 𝜌𝜎 (Z2𝜅 −𝑐) = 𝑘 ∈Z2𝜅 exp −(𝑘 − 𝑐) 2 /(2𝜎 2 ) . Further, manipulating the term in the exponent of eq. (16) to obtain 𝑡𝑘 − 𝑘 2 /(2𝜎 2 ) = −(𝑘 − 𝑡𝜎 2 ) 2 /(2𝜎 2 ) + 𝜎 2𝑡 2 /2, gives
𝜌𝜎 (Z2𝜅 − 𝑡𝜎 2 ) E [exp (𝑡𝑒𝑖 )] = exp 𝜎 2𝑡 2 /2 · 𝜌𝜎 (Z2𝜅 ) By the Poisson summation formula, ∑︁ √
𝜌𝜎 (Z2𝜅 − 𝑐) = 𝜎 2𝜋 exp −2𝜋 2 𝜎 2𝑘 2 cos(2𝜋𝑘𝑐) 𝑘 ∈Z2𝜅

Since cos(2𝜋𝑘𝑐) ≤ 1 for all 𝑘, 𝑐 and each coefficient exp −2𝜋 2 𝜎 2𝑘 2 is non-negative, 𝜌𝜎 (Z2𝜅 − 𝑐) ≤ 𝜌𝜎 (Z2𝜅 ). Thus,

𝜌𝜎 (Z2𝜅 − 𝑡𝜎 2 ) 𝜌𝜎 (Z2𝜅 )
and E [exp (𝑡𝑒𝑖 )] ≤ exp 𝜎 2𝑡 2 /2 , confirming that 𝑒𝑖 is 𝜎-sub-Gaussian. 1≤

14

MOSAIC: Masked Outsourcing of Secure AI Computations

For independent 𝜎-sub-Gaussian variables, (𝐸𝑋 ) 𝑗𝑘 = Í is ( 𝑖 (𝑥𝑘 )𝑖2 𝜎 2 ) 1/2 = 𝜎 ∥𝑥𝑘 ∥ 2 -sub-Gaussian, since " !# Ö ∑︁ E exp 𝑡 (𝑥𝑘 )𝑖 𝑒𝑖 = E [exp (𝑡 (𝑥𝑘 )𝑖 𝑒𝑖 )] 𝑖

Í

𝑖 (𝑥𝑘 )𝑖 𝑒𝑖

entry √︃ ( 𝑗, 𝑘) of the total error 𝐸𝑊 𝑋 + 𝑊 𝐸𝑋 + 𝐸𝑊 𝐸𝑋 is approximately 2 𝜎 ∥𝑥𝑘 ∥ 22 + 𝑤 𝑗 2 -sub-Gaussian: for all 𝑡 > 0,

≤

exp 𝜎 2 (𝑥𝑘 )𝑖2𝑡 2 /2

up to 𝑒 −Ω (𝑛) , where 𝑤 𝑗 is the 𝑗-th row of 𝑊 and 𝑥𝑘 is the 𝑘-th column of 𝑋 .

𝑖

= exp 𝜎 2 ∥𝑥𝑘 ∥ 22 𝑡 2 /2 .

Proof. The linear terms (𝐸𝑊 𝑋 ) 𝑗𝑘 and (𝑊 𝐸𝑋 ) 𝑗𝑘 are independent, since they depend on the independent noise matrices 𝐸𝑊 and 𝐸𝑋 respectively. By lemma 2, (𝐸𝑊 𝑋 ) 𝑗𝑘 is 𝜎 ∥𝑥𝑘 ∥ 2 -sub-Gaussian and (𝑊 𝐸𝑋 ) 𝑗𝑘 is 𝜎 𝑤 𝑗 2 -sub-Gaussian. For independent sub-Gaussian variables,√︃the sub-Gaussian parameters add in quadrature, so their

The tail bound follows from the standard for any  Chernoff method:
 𝑡 > 0 and 𝜆 > 0, Pr[(𝐸𝑋 ) 𝑗𝑘 > 𝑡] ≤ E exp 𝜆(𝐸𝑋 ) 𝑗𝑘 /exp (𝜆𝑡) ≤
exp 𝜎 2 ∥𝑥𝑘 ∥ 22 𝜆 2 /2 − 𝜆𝑡 . Minimizing over 𝜆 yields     𝑡2 Pr |(𝐸𝑋 ) 𝑗𝑘 | > 𝑡 ≤ 2exp − 2𝜎 2 ∥𝑥𝑘 ∥ 22

2

sum is 𝜎 ∥𝑥𝑘 ∥ 22 + 𝑤 𝑗 2 -sub-Gaussian. By corollary 1, the cross

term 𝐸𝑊 𝐸𝑋 is negligible whenever 𝜎 2 ≪ 𝑥 2 , contributing only an additive 𝑒 −Ω (𝑛) to the tail probability. □

□ Lemma 3. Let 𝐸𝑊 ∈ Z𝑚×𝑛 and 𝐸𝑋 ∈ Z𝑛×𝑙 have mutually indepen2𝜅
2𝜅 dent entries drawn from N Z2𝜅 ; 0, 𝜎 2 . Then entry ( 𝑗, 𝑘) of 𝐸𝑊 𝐸𝑋 satisfies:   (1) E (𝐸𝑊 𝐸𝑋 ) 𝑗𝑘 = 0. (2) For all 𝑡 > 0,  
Pr |(𝐸𝑊 𝐸𝑋 ) 𝑗𝑘 | > 𝑡 ≤ 2exp −𝑡 2 /(4𝑛𝜎 4 ) + 𝑒 −Ω (𝑛)

A.3

Proof of Lemma 1

Lemma 1. The LWE mask is computationally indistinguishable from uniform: 𝑐

𝐿𝑀 + 𝐸 𝑤 ≈ 𝑈 . 𝑐

Í Proof. Fix indices 𝑗, 𝑘 and write (𝐸𝑊 𝐸𝑋 ) 𝑗𝑘 = 𝑛𝑖=1 𝑎𝑖 𝑏𝑖 , where 𝑎𝑖 = (𝐸𝑊 ) 𝑗𝑖 and 𝑏
𝑖 = (𝐸𝑋 )𝑖𝑘 are mutually independent samples from N Z2𝜅 ; 0, 𝜎 2 . Í (1) By independence and E [𝑎𝑖 ] = E [𝑏𝑖 ] = 0, E [ 𝑖 𝑎𝑖 𝑏𝑖 ] = Í 𝑖 E [𝑎𝑖 ] E [𝑏𝑖 ] = 0. Í (2) Condition on 𝑏 = (𝑏 1, . . . , 𝑏𝑛 ). Given 𝑏, the sum 𝑖 𝑎𝑖 𝑏𝑖 is a linear combination of independent 𝜎-sub-Gaussian variables with fixed coefficients. By lemma 2, the conditional tail satisfies " #   ∑︁ 𝑡2 Pr 𝑎𝑖 𝑏𝑖 > 𝑡 𝑏 ≤ 2exp − 2𝜎 2 ∥𝑏 ∥ 22 𝑖   Since each 𝑏𝑖 is 𝜎-sub-Gaussian, 𝑏𝑖2 is sub-exponential with E 𝑏𝑖2 = 𝜎𝑑2 ≤ 𝜎 2 . Standard sub-exponential concentration (Bernstein inequality) yields Pr[∥𝑏 ∥ 22 > 2𝑛𝜎 2 ] ≤ 𝑒 −Ω (𝑛) . On the high-probability
event {∥𝑏 ∥ 22 ≤ 2𝑛𝜎 2 }, the conditional bound becomes 2exp −𝑡 2 /(4𝑛𝜎 4 ) after substitution, and the total probability law gives the result. □

Proof sketch. We show 𝐿𝑀 + 𝐸 𝑤 ≈ 𝑈 column by column. Interpret the 𝑗’th column of 𝐿𝑀 + 𝐸 𝑤 as 𝐿𝑚 𝑗 + 𝑒 𝑤,𝑗 ∈ F𝑚 𝑝 and let 𝐻 0 = 𝐿𝑀 + 𝐸 𝑤 = (𝐿𝑚 1 + 𝑒 𝑤,1, . . . , 𝐿𝑚𝑛 + 𝑒 𝑤,𝑛 ). For 𝑛 ≥ 𝑖 > 0, define 𝐻𝑖 = (𝑢 1, . . . , 𝑢𝑖 , 𝐿𝑚𝑖+1 +𝑒 𝑤,𝑖+1, . . . , 𝐿𝑚𝑛 +𝑒 𝑤,𝑛 ), where 𝑢 1, . . . , 𝑢𝑖 are independent uniform samples over F𝑚 𝑝 . The two consecutive hybrids • 𝐻𝑖 , and • 𝐻𝑖+1 , which replaces the (𝑖 + 1)-th entry 𝐿𝑚𝑖+1 + 𝑒 𝑤,𝑖+1 with a fresh uniform sample 𝑢𝑖+1 , differ only in that single coordinate; distinguishing them is exactly a decisional-LWE distinguisher for the public matrix 𝐿 and noise 𝑒 𝑤,𝑖+1 . 𝑐 s s By the hybrid argument, 𝐻 0 ≈ 𝐻𝑛 ≈ (𝑢 1, . . . , 𝑢𝑛 ) ≈ 𝑈 , with total advantage bounded by 𝑛 times the decisional-LWE advantage. 𝑐 Hence 𝐿𝑀 + 𝐸 𝑤 ≈ 𝑈 . □

A.4

Nested LWE+LPN mask

Lemma 4. The nested LWE+LPN mask is computationally indistinguishable from uniform:

Corollary 1. The cross term 𝐸𝑊 𝐸𝑋 is dominated by the linear error terms 𝐸𝑊 𝑋 and 𝑊 𝐸𝑋 . Comparing the tail bounds from Lemma 2 and
Lemma 3: the linear terms decay as exp −𝑡 2 /(2𝜎 2 ∥𝑥 ∥ 22 ) , while the
cross term decays as exp −𝑡 2 /(4𝑛𝜎 4 ) . With ∥𝑥 ∥ 22 = 𝑛𝑥 2 (average squared entry 𝑥 2 ), the linear exponent scales as 𝑡 2 /(𝑛𝜎 2𝑥 2 ) versus 𝑡 2 /(𝑛𝜎 4 ) for the cross term. The cross term is therefore negligible whenever 𝜎 2 ≪ 𝑥 2 , a condition required by the protocol for useful signal-to-noise ratio.

A.2

2

2𝜎 2 (∥𝑥𝑘 ∥ 22 + 𝑤 𝑗 2 )

𝑖

Ö

!

𝑡2

  Pr |(𝐸𝑊 𝑋 + 𝑊 𝐸𝑋 + 𝐸𝑊 𝐸𝑋 ) 𝑗𝑘 | > 𝑡 ≤ 2exp −

𝑐

(((𝐿𝑑 𝑀𝑑 + 𝑆𝑑 )𝑀𝑑 −1 + 𝑆𝑑 −1 ) · · · 𝑆 2 )𝑀1 + 𝐸 𝑤 ≈ 𝑈 . Proof sketch. Let 𝐹𝑑 denote the nested mask of the statement 𝑐 and 𝐹 1 = 𝐿1 𝑀1 + 𝐸 𝑤 the simplified LWE mask. We show 𝐹𝑑 ≈ 𝑈 by a hybrid argument that unwinds the nested LPN instances one level at a time until we reach 𝐹 1 , then invokes Lemma 1. Define 𝑑 hybrids 𝐹𝑑 , 𝐹𝑑 −1, . . . , 𝐹 1 , where 𝐹𝑖 is obtained from 𝐹𝑖+1 by replacing the innermost LPN instance 𝐿𝑖+1 𝑀𝑖+1 + 𝑆𝑖+1 with a freshly sampled uniform matrix 𝐿𝑖 of the same dimensions. Here
×𝑟𝑖 −1
𝐿𝑖+1 ← U Z𝑟2𝑖𝜅×𝑟𝑖+1 , 𝑀𝑖+1 ← U Z𝑟2𝑖+1 and 𝑆𝑖+1 has row Ham𝜅 ming weight 𝑡𝑖+1 (with 𝑟 0 := 𝑛).

Proof of Theorem 1

𝑛×𝑙 𝑚×𝑛 Theorem 1. Let 𝑊 ∈ Z𝑚×𝑛 2𝜅 , 𝑋 ∈ Z2𝜅 , and let 𝐸𝑊 ∈
Z2𝜅 , 2 . Then 𝜅 𝐸𝑋 ∈ Z𝑛×𝑙 have independent entries drawn from N Z ; 0, 𝜎 2 2𝜅

15

Chiang et al.

Theoretical efficiency of outsourcing (𝑙 = 1)

Fix a transition 𝐹𝑖+1 → 𝐹𝑖 . Distinguishing these two distributions reduces to distinguishing, at the position where 𝐿𝑖 appears, • 𝐿𝑖+1 𝑀𝑖+1 + 𝑆𝑖+1 , a decisional LPN sample with secret dimension 𝑟𝑖+1 , sample count 𝑟𝑖 −1 , and noise Hamming weight 𝑡𝑖+1 , from
• uniform 𝐿𝑖 ← U Z𝑟2𝑖𝜅−1 ×𝑟𝑖 .

Trusted C Work: outsourced / local 𝑊 𝑋

The reduction receives the LPN challenge 𝐿 ∗ , embeds it at the 𝑖th position of the mask, samples all remaining 𝐿 𝑗 , 𝑀 𝑗 , 𝑆 𝑗 and 𝐸 𝑤 honestly, and forwards the constructed mask to the distinguisher. 𝑐 Its advantage transfers without loss, so 𝐹𝑖+1 ≈ 𝐹𝑖 . 𝑐 Chaining the 𝑑 − 1 LPN hybrid steps gives 𝐹𝑑 ≈ 𝐹 1 with distinguishing advantage at most (𝑑 − 1) times the best decisional-LPN 𝑐 𝑐 advantage. By Lemma 1, 𝐹 1 ≈ 𝑈 . Composing yields 𝐹𝑑 ≈ 𝑈 . □

B

𝑚 = 𝑛/4 𝑚 =𝑛 𝑚 = 4𝑛

100

10−1

10−2

Computational integrity 50

We refer to a well-known randomized check (Freivalds’ algorithm) to ensure correctness of 𝑊 𝑋 = 𝑌 computed by the GPU at runtime, at the cost of additional input-independent precomputation. For verification, the client samples a random vector (𝑎) and performs the following assertion;

200

Figure 13: Theoretical trusted-client work vs. local 𝑊 𝑋 work (𝑙 = 1; X as a vector) with protocol parameterization in Example 4.1.

Observe that 𝑎𝑇 𝑊 is input independent and can be preprocessed and reused for each inference run; at runtime, the assertion above can be evaluated in 𝑂 ((𝑛 + 𝑚)𝑙). To violate soundness, the cheating prover must return 𝑌 such that the following holds 𝑇

150

Rank of Model Weight Matrix 𝑛 (×103 )

𝑎𝑇 𝑊 𝑋 =? 𝑎𝑇 𝑌

𝑇

100

activation column. Dequantization rescales the integer result by both scales 𝑠 𝑤𝑗 · 𝑠𝑥𝑘 , giving a full (floating-point) variance of 2 Var(𝐸 𝑤 𝑋 ) 𝑗,𝑘 = 𝜎 2 · ∥𝑥ˆ𝑘 ∥ 22 · 𝑠 𝑤 · 𝑠𝑥2𝑘 𝑗

(19)

𝑇

𝑎 𝑊 𝑋 = 𝑎 𝑌 ⇐⇒ 𝑎 (𝑊 𝑋 − 𝑌 ) = 0

Note that 𝑥ˆ𝑘 = round(𝑥𝑘 /𝑠𝑥𝑘 ), so ∥𝑥ˆ𝑘 ∥ 22 ≈ ∥𝑥𝑘 ∥ 22 /𝑠𝑥2𝑘 . Substituting into eq. (19),

where 𝑊 𝑋 − 𝑌 ≠ 0. This occurs with probability 1/|F| (SchwartzZippel Lemma).

2 Var(𝐸 𝑤 𝑋 ) 𝑗,𝑘 ≈ 𝜎 2 · ∥𝑥𝑘 ∥ 22 · 𝑠 𝑤 . 𝑗

C

Theoretical outsourcing efficiency

Substituting 𝑠 𝑤𝑗 = 𝑤 𝑗 ∞ /𝑞 max , the dequantized variance simplifies to 2 𝑤𝑗 Var(𝐸 𝑤 𝑋 ) 𝑗,𝑘 = 𝜎 2 · ∥𝑥𝑘 ∥ 22 · 2 ∞ (20) 𝑞 max With Hadamard rotation, substituting eq. (17) into eq. (20), this becomes

For our proposed protocol, we show the ratio of (1) trusted client protocol overhead to (2) the cost of the client computing 𝑊 𝑋 locally in Figure 13. We consider multiplicative operations for (1) and (2).

D

Protocol error after rotation

Reduction of the infinity norm. For any fixed vector 𝑥 ∈ R𝑛 , each entry of 𝑅𝑥 is a normalized sum of 𝑛 random-sign terms. By sub-Gaussian concentration [1], with high probability, √︂ log 𝑛 ∥𝑅𝑥 ∥ ∞ ≲ ∥𝑥 ∥ 2 · (17) 𝑛

2

Var(𝐸 𝑤 𝑅𝑋 ) 𝑗,𝑘 ≲ 𝜎

2

· ∥𝑥𝑘 ∥ 22 ·

𝑤𝑗 2 2 𝑞 max

·

log 𝑛 𝑛

(21)

By symmetry, the noise from 𝑊 𝐸𝑥 is reduced analogously via the activation rotation. The ratio of eq. (21) to eq. (20) gives the variance reduction factor for the 𝐸 𝑤 𝑋 term:

In contrast, without rotation ∥𝑥 ∥ ∞ can be as large as ∥𝑥 ∥ 2 when a single entry dominates. Therefore the quantization scale after rotation satisfies √︂ log 𝑛 ∥𝑥 ∥ 2 ∥𝑅𝑥 ∥ ∞ ′ 𝑠𝑥 = ≲ · (18) 𝑞 max 𝑞 max 𝑛

Var(𝐸𝑀 𝑅𝑋 ) 𝑗,𝑘 ≈ Var(𝐸𝑀 𝑋 ) 𝑗,𝑘

2

𝑤 𝑗 2 · log 𝑛 2 𝑤𝑗 ∞ · 𝑛

= 𝜌 (𝑤 𝑗 ) 2 ·

log(𝑛) 𝑛

(22)

and analogously for 𝑊 𝐸𝑥 . This ratio equals log(𝑛)/𝑛 in the worst case of a single dominant outlier (∥𝑤 ∥ ∞ ≈ ∥𝑤 ∥ 2 ). For a transformer dimension 𝑛 = 4096, this yields an approximate 𝑛/log 𝑛 ≈ 340× reduction in noise variance. The cross term 𝐸 𝑤 𝐸𝑥 from eq. (7) is unaffected by the Hadamard rotation: 𝐸𝑀 and 𝐸𝑋 are sampled in the integer domain independently of both 𝑊 and 𝑋 , so their product distribution is invariant

Float-domain noise variance. Consider the dominant error term 𝐸 𝑤 𝑋 from eq. (7) in the floating-point domain. Entry ( 𝑗, 𝑘) of 𝐸 𝑤 𝑋ˆ Í in the integer domain is the inner product 𝑖 (𝐸 𝑤 ) 𝑗𝑖 𝑥ˆ𝑖𝑘 , where each (𝐸 𝑤 ) 𝑗𝑖 is an independent Gaussian with variance 𝜎 2 . By lemma 2, the variance of this entry is 𝜎 2 ∥𝑥ˆ𝑘 ∥ 22 , where 𝑥ˆ𝑘 is the quantized 16

MOSAIC: Masked Outsourcing of Secure AI Computations

Qwen2.5-72B latency: sys-prompt setup (1k/4k/7k tokens) vs. 35-token runtime query

Asymptotic cost Client C

Component / Regime

GPU G

Per-layer breakdown 7 weight projections (𝑄, 𝐾, 𝑉 , 𝑂 , gate, up, down) Per-MatMul outsourcing overhead ( × 7) Fast Walsh–Hadamard rotation (per call) Per-head attention (softmax, 𝑄𝐾 ⊤ , ·𝑉 ) Element-wise non-linearities (RMSNorm, etc.) Per-layer total (𝑠 ≥ log 𝑑 )

— 𝑂 (𝑑 𝑙 ) 𝑂 (𝑑 𝑙 log 𝑑 ) 𝑂 (𝑑 𝑠 𝑙 ) 𝑂 (𝑑 𝑙 )

𝑂 (𝑑 2𝑙 ) — — — —

O(d s l)

O(d2 l)

Latency (seconds)

103

Sys 1k tokens Sys 4k tokens Sys 7k tokens Rt (per query, 35 tokens)

102

101

Full forward pass over 𝐿 layers

𝑂 (𝐿 𝑑 𝑠 𝑙 ) 𝑂 (𝐿 𝑑 2𝑙 ) 𝑂 (𝐿 𝑑 𝑠 2 ) 𝑂 (𝐿 𝑑 2𝑠 ) 𝑂 (𝐿 𝑑 𝑠 ) 𝑂 (𝐿 𝑑 2 ) 𝑂 (𝐿 𝑑 2 log 𝑑 ) —

General (𝑙 new tokens, seq. length 𝑠 ) Prefill (𝑙 = 𝑠 ) Decode (𝑙 = 1) Init W (one-time, amortised)

100 WAN

WAN (far)

Internet

1ms/100Gb

10ms/100Gb

20ms/10Gb

20ms/1Gb

Table 6: Communication volume per phase (1k-token system prompt). init W is a one-time model setup cost; sys-prefill runs once and is amortized over subsequent runtime prompts. rt-prefill is a query of 35 tokens. “Send” is C to G; “Recv” is G to C.

Secure forward-pass complexity

Table 5 summarises the per-layer and full-forward-pass complexity for an 𝐿-layer transformer with model dimension 𝑑 model , intermediate (MLP) dimension 𝑑 ff ≈ 3.5 𝑑 model , total sequence length 𝑠, and 𝑙 new tokens per query. Per-layer GPU work is dominated by the 2 seven weight projections at 𝑂 (𝑑 model 𝑙), while per-layer trustedclient work is dominated by per-head attention at 𝑂 (𝑑 model · 𝑠 · 𝑙) for any context 𝑠 ≥ log 𝑑 model . The 𝐿-layer forward pass and the prefill (𝑙 = 𝑠) and autoregressive decode (𝑙 = 1) cases follow directly. A one-time per-weight Init W 2 2 cost of 𝑂 (𝑑 model log 𝑑 model ) totals 𝑂 (𝐿 𝑑 model log 𝑑 model ) across the model and is amortised across every subsequent forward pass.

Llama-3-70B Send Recv init_W (one-time) sys_prefill (one-time) rt_prefill (per query)

278.0 GB 17.1 GB 0.60 GB

— 31.6 GB 1.11 GB

Qwen2.5-72B Send Recv 285.9 GB 17.4 GB 0.61 GB

— 32.2 GB 1.13 GB

• In H0 , 𝑌 = 𝑊obf 𝐻 −1 +(𝑊pre −𝑊vic ) = 𝑊vic +𝑊pre −𝑊vic = 𝑊pre deterministically, so Pr[D (𝑌 ) = 1 | H0 ] = 1. • In H1 , 𝑌 = 𝑊vic +𝐸 equals 𝑊pre only when the fresh Gaussian 𝐸 lands on the single fixed point 𝑊pre −𝑊vic , which happens with probability negligible in 𝜆. Thus D achieves distinguishing advantage 1 − negl(𝜆), contradict𝑐 ing the indistinguishability step H0 ≈ H1 their reduction relies on.

Broken obfuscation security

We show a formal break in the security reduction of ArrowCloak [29] (USENIX’25). The scheme protects the privacy of a fine-tuned model 𝑊vic derived from a pre-trained model 𝑊pre , where the pre-trained model is explicitly public and static in the adversary’s view. The obfuscated weight is 𝑊obf = 𝑊vic 𝐻 for a secret “encryption” key matrix 𝐻 . The authors claim a reduction to search-LWE hardness and, as part of that reduction, introduce the following hybrid step (reproducing eq. 12, section 6.2 of [29]): H0 : 𝑌 = 𝑊obf 𝐻 −1 + (𝑊pre − 𝑊vic )

Regional

0.05ms/100Gb

Figure 14: Network sensitivity: system-prompt prefill at 1k / 4k / 7k tokens (dashed) and per-query runtime prefill (solid) at 35-token across various simulated network scenarios. While the (one-time) 7k-token sys prefill approaches 45 minutes on a 1 Gbps Internet link, the per-query runtime stays close to 20 s.

to the rotation. By corollary 1, this term remains negligible relative to the linear error terms both before and after rotation.

F

Datacenter

Network scenario

Table 5: Asymptotic per-layer and full-forward-pass complexity of ΠSec-FwdPass . Notation: 𝑑 = 𝑑 model , 𝐿 transformer layers, total sequence length 𝑠, 𝑙 new tokens per query.

E

NVLink 0.001ms/400Gb

G

H1 : 𝑌 = 𝑊obf 𝐻 −1 + 𝐸,

Supplementary deployment figures

We complement the main body figures with per-query prefill latencies across different network environments in the remote decisional setting (Figure 14) for 1/4/7k context sizes. We show communication volume for a 1k prefill required for our proposed protocol (Table 6) applied to 70B models.

where 𝐸 is a fresh discrete-Gaussian LWE sample. Computational indistinguishability of H0 and H1 is the essential step in their reduction; the authors support it with empirical evidence that𝑊pre − 𝑊vic is Gaussian-like.

H

Trivial distinguisher. However, the two hybrids are trivially distinguishable because𝑊obf and𝑊pre are part of the adversary’s public view per the ArrowCloak threat model. Define the deterministic distinguisher D (𝑌 ) := [ 𝑌 =? 𝑊pre ]. Then:

Outsourcing remote classifications

We also consider a case-study of an application that is not latency critical, and thus permits interactive outsourcing of computation across public networks, between a local trusted client and a remote 17

Chiang et al.

Trusted GPU

Untrusted GPU

diagnostic reasoning to a large language model hosted on an untrusted cloud accelerator. The system prompt encodes the full diagnostic knowledge base: equipment-specific failure mode signatures, vibration and temperature thresholds, oil-analysis limits, multisymptom escalation rules, and historical case studies. This domain context is lengthy (1k–7k tokens in our benchmarks) but changes infrequently—at most when maintenance procedures are revised. The runtime prompt, by contrast, is a short field report (∼35 tokens) written by a technician on the factory floor: a handful of sensor readings and a one-line observation such as “Coupling very hot, faint burnt smell from motor end.” The model’s task is to return a bounded decision, e.g. a severity classification and recommended action, not an open-ended generation. We choose this scenario precisely because it is not latencycritical: a maintenance classification that completes in seconds is well within operational requirements, unlike interactive chat or real-time control. Importantly, the task requires only a single forward pass at runtime over a bounded number of tokens, avoiding the compounding communication cost of autoregressive decode rounds. The expensive yet stable system-prompt setup is performed once and amortized over many lightweight runtime queries.

Comm (10 ms / 100 Gbps)

Latency (seconds)

87

100 66

80 45

60

17

40 4.7

20

7k )

k Sy s7 Rt

(S ys

1/ 4/

k Sy s4

in

Sy s

it_ W

1k

0

Figure 15: System prefill and runtime latency under simulated WAN (10 ms RTT, 100 Gbps, 𝐺 = 3 untrusted GPUs) for Qwen2.5-72B. Trusted GPU

Untrusted GPU

Comm (10 ms / 100 Gbps)

85

Latency (seconds)

100 80 63

Evaluation results. Figures 15 and 16 show that latency for decisional inference can remain practical. Despite ≈ 80 × 4 communication rounds, a single decisional inference query can be completed in under 5 seconds over a fast public network (10ms, 100Gbps). One can observe greatly reduced transfer volumes for the short runtime prompt compared to the system prompt, shown in Table 6. Overall run-time is clearly affected by communication latency; still for the short, real-time diagnostic prompt, the system prompt length does not have a measurable effect. The system prompt prefill of 7k tokens can be completed in about a minute, implying that frequent background, prefill updates are permissible. Simulated runtimes for a wider range of realistic network settings are detailed in Figure 14.

43

60

16

40 4.7

20

7k

)

k

4/

s7 Rt

(S

ys

1/

Sy

Sy

s4

k

k s1 Sy

in

it_ W

0

Figure 16: System prefill and runtime latency under simulated WAN (10 ms RTT, 100 Gbps, 𝐺 = 3 untrusted GPUs) for LLaMA-3-70B.

I

We supplement the main body with error accumulation metrics on final model layers in Table 7 and error accumulation (cosine relative ℓ2 error) across layers for Qwen2.5-32B (Figure 18), Qwen2.5-72B (Figure 19), LLaMA3-70B (Figure 20), DeepSeek-R1-Distill-LLaMA70B (Figure 21).

cloud accelerator (Figure 17). In addition to emulating the ring integer arithmetic (as mentioned at the beginning of this section) to measure client and remote GPU computation runtimes, we simulate network latency induced by our protocol for different network settings (Figures 15 and 16). Local site Cloud Field report ∼ 35 tok

TCB Trusted client

Decision (severity, action)

Public network e.g. 10 ms RTT, 100 Gbps 1 forward pass / query

Layer-by-layer error accumulation

Cloud GPU

Figure 17: Remote outsourcing topology: a trusted client at the local site outsources a single forward pass per query to an untrusted cloud GPU over a public network (e.g. 10 ms RTT, 100 Gbps). Industrial diagnostics. Consider an industrial predictive maintenance scenario in which a company operates a fleet of equipment— pumps, compressors, motors, gearboxes—and wishes to outsource 18

MOSAIC: Masked Outsourcing of Secure AI Computations

Noisy vs. ref

BF16 reference max |𝑥 |

𝜎𝑥

|𝑥 |

max |𝑥 |/𝜎𝑥

Qwen2.5-32B, protocol noise 𝜎 = 1.0 ℓ = 62 0.9991 0.0391 5480 ℓ = 63 0.9984 0.0532 4340 post-RMSNorm 0.9987 0.0448 173

23.4 24.1 2.86

11.98 13.95 1.37

236 177 61

Qwen2.5-72B, protocol noise 𝜎 = 1.0 ℓ = 78 0.9960 0.0724 1602 ℓ = 79 0.9941 0.0938 2850 post-RMSNorm 0.9949 0.0812 335

14.2 15.7 3.6

7.75 8.90 1.67

116 183 93

0.66 0.88 2.03

0.46 0.60 1.42

356 95 46

DeepSeek-R1-Distill-LLaMA-70B, protocol noise 𝜎 = 1.0 ℓ = 78 0.8289 0.5464 400 0.70 0.45 ℓ = 79 0.8387 0.5399 60 0.77 0.55 post-RMSNorm 0.8466 0.5098 87 2.00 1.46

572 78 44

Step

cos

rel-ℓ2

Quantization-friendly models

Quantization-unfriendly models LLaMA-3-70B, protocol noise 𝜎 = 1.0 ℓ = 78 0.9795 0.1774 233 ℓ = 79 0.9813 0.1771 86 post-RMSNorm 0.9822 0.1654 94

Table 7: Reference (BF16, noise-free) residual-stream amplitude and matched perturbed-pass metrics at the last two transformer model layers and after the final RMSNorm.

19

Chiang et al.

0.3

1

Relative ℓ2 error

Cosine similarity

0.99

0.98

4-bit NF4 INT8 INT16-rot 𝜎 = 0 INT16-rot 𝜎 = 0.5 INT16-rot 𝜎 = 0.7 INT16-rot 𝜎 = 1.0 INT16-rot 𝜎 = 1.5 INT16-rot 𝜎 = 2.0

0.97

0.96

0.2

0.1

0 0

20

40 Layer index (ℓ)

0

60

20

40 Layer index (ℓ)

60

Figure 18: Per-layer error accumulation: Qwen2.5-32B (64 model layers). For 20 recorded runs, the 1-standard deviation band is shown for selected quantization schemes. 1 0.25

Relative ℓ2 error

Cosine similarity

0.99

0.98 4-bit NF4 INT8 INT16-rot 𝜎 = 0 INT16-rot 𝜎 = 0.5 INT16-rot 𝜎 = 0.7 INT16-rot 𝜎 = 1.0 INT16-rot 𝜎 = 1.5 INT16-rot 𝜎 = 2.0

0.97

0.96

0.2

0.15

0.1 5 · 10 −2

0 0

20

40 Layer index (ℓ)

60

0

20

40 Layer index (ℓ)

Figure 19: Per-layer error accumulation: Qwen2.5-72B (80 model layers).

20

60

MOSAIC: Masked Outsourcing of Secure AI Computations

1.5

1

Relative ℓ2 error

Cosine similarity

0.9

0.8

4-bit NF4 INT8 INT16-rot 𝜎 = 0 INT16-rot 𝜎 = 0.5 INT16-rot 𝜎 = 0.7 INT16-rot 𝜎 = 1.0 INT16-rot 𝜎 = 1.5 INT16-rot 𝜎 = 2.0

0.7

0.6

0.5

1

0.5

0 0

20

40 Layer index (ℓ)

60

0

20

40 Layer index (ℓ)

60

Figure 20: Per-layer error accumulation: LLaMA-3-70B (80 layers). INT8 baseline collapses at ℓ = 79 and goes off-scale. 2

1

1.5 Relative ℓ2 error

Cosine similarity

0.8

0.6 4-bit NF4 INT8 INT16-rot 𝜎 = 0 INT16-rot 𝜎 = 0.5 INT16-rot 𝜎 = 0.7 INT16-rot 𝜎 = 1.0 INT16-rot 𝜎 = 1.5 INT16-rot 𝜎 = 2.0

0.4

1

0.5

0 0

20

40 Layer index (ℓ)

60

0

20

40 Layer index (ℓ)

60

Figure 21: Per-layer error accumulation: DeepSeek-R1-Distill-LLaMA-70B (80 layers).

21