arXiv CSopen access

An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG

An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG Chenxin Mao† , Shangyu Liu† , Zhenzhe Zheng† , Fan Wu† , Jie Wu‡ , Guihai Chen† ‡ Cloud Computing Research Institute, China Telecom

arXiv:2605.25716v1 [cs.CR] 25 May 2026

† Shanghai Jiao Tong University

Abstract

capabilities continue to evolve, LLMs will progressively penetrate more complex, multi-domain applications. Despite possessing vast pre-trained knowledge bases and exhibiting remarkable generalization, generalized LLMs encounter significant challenges when tasked with domainspecific or highly time-sensitive knowledge. A prominent issue is "hallucination," whereby models, lacking adequate context, are prone to generating plausible yet factually incorrect information [56]. Furthermore, maintaining the currency of the model’s knowledge via continual training entails prohibitive computational and temporal costs. To mitigate these inherent limitations, the Retrieval-Augmented Generation (RAG) architecture has emerged as a promising paradigm [29]. By dynamically integrating external, authoritative knowledge bases, RAG endows LLMs with the ability to access specialized and current information without requiring parameter retraining. This approach effectively curtails the incidence of hallucinations while simultaneously ensuring the accuracy and timeliness of the generated content. Beyond open-domain applications that leverage publicly accessible web data, many organizations increasingly rely on their proprietary data to construct specialized RAG systems. However, the efficacy of such RAG systems is intrinsically contingent upon the scale and comprehensiveness of their underlying knowledge bases [4]. In highly specialized domains, a single organization typically possesses limited and fragmented proprietary data, which is insufficient to support robust, well-rounded LLM reasoning. Consequently, there is a compelling need to collaborate and synthesize knowledge across multiple institutions to construct a more capable and comprehensive RAG system. For example, healthcare providers can enhance complex diagnostics by integrating electronic medical records from multiple hospitals, financial institutions can conduct robust risk assessments utilizing transaction logs across various banks, and the manufacturing sector can optimize production processes by analyzing operational data from multiple factories. Despite a strong mutual willingness to collaborate, organizations are strictly precluded from directly sharing raw documents due to stringent privacy

Retrieval-Augmented Generation (RAG) empowers LLMs with external knowledge, making cross-institutional domainspecific knowledge base integration a highly promising deployment paradigm. Despite this potential, strict privacy regulations create severe "data silos" that obstruct such collaboration. Building federated RAG systems requires distributed inference, but the Transformer’s self-attention mechanism fundamentally conflicts with this by mandating cross-node access to distributed Key-Value caches. To address this challenge, we present FedRAG, a high-throughput, privacy-preserving federated RAG framework. At its core is a novel Scrambled Distributed Attention protocol that utilizes numerically stable feature scrambling and token permutation. By dynamically delegating scrambled computations to collaborating nodes, our system successfully decouples attention execution from data localization without exposing plaintext. Crucially, our approach requires no specialized hardware or model retraining, circumventing the prohibitive latency and communication overheads of cryptographic solutions while robustly defending against intermediate state inversion attacks. Extensive evaluations demonstrate our framework preserves negligible (<0.1%) model utility degradation and achieves up to a 62× latency reduction over existing secure baselines, sustaining practical, human-reading throughput for cross-institutional knowledge synergy.

1

Introduction

In recent years, transformer-based Large Language Models (LLMs) have achieved unprecedented breakthroughs across both academia and industry, demonstrating exceptional capabilities in multilingual interaction, text generation, and logical reasoning [58]. This paradigm shift has led to their seamless integration into a myriad of applications, significantly optimizing various workflows, such as software development [6], conversational agents [36], knowledge management [38], and healthcare [42]. It is foreseeable that as their foundational 1

regulations and the imperative to protect commercial secrets, which inevitably leads to the formation of "data silos." Therefore, it is imperative to explore and establish privacypreserving Federated RAG frameworks to dismantle these informational barriers and enable secure, cross-institutional knowledge synergy. However, developing such collaborative systems entails addressing three primary challenges: • Privacy Protection. Early works attempted to preserve privacy by transmitting model intermediate states rather than raw text. However, recent studies reveal that the intermediate states of LLMs exhibit high semantic sparsity, rendering them vulnerable to inversion attacks capable of reconstructing partial original text [11]. For instance, Vocabulary Mapping Attacks (VMA) [44], which rely on traversing the vocabulary, can effectively bypass simple permutation-based protection schemes on intermediate states, including both sequencedimension and hidden-dimension permutations. The emergence of such sophisticated attacks imposes significantly more stringent requirements on the design of privacy-preserving algorithms. • Inference Overhead. Existing privacy-preserving distributed LLM inference architectures based on cryptographic methods suffer from prohibitive inference overhead. Even state-of-the-art frameworks utilizing Secure Multi-Party Computation (MPC), such as PUMA [12], require several minutes to generate a single token using an 8-billion parameter model. Furthermore, excessive communication overhead remains a pressing issue. Taking Homomorphic Encryption (HE) schemes (e.g., Paillier [37] or CKKS [7]) as an example, the ciphertext volume typically expands by tens to hundreds of times compared to the plaintext. For massive intermediate state tensors, such transmission costs are practically unacceptable. The severe latency induced by both computation and transmission renders cryptography-based secure inference impractical for real-world RAG systems. • Universality and Rapid Deployment. Many distributed secure inference architectures impose stringent prerequisites on the execution environment or the model itself. Hardwarebased solutions relying on Trusted Execution Environments (TEEs) [41] necessitate specialized hardware support and complex environmental attestation configurations, which are often difficult to satisfy in heterogeneous distributed systems. Conversely, other secure inference architectures require retraining the model [26, 39] or substituting specific operators with approximate counterparts. However, retraining hinders the seamless adoption of the latest high-performance models, while approximate operators inevitably lead to a degradation in inference accuracy. To address these challenges, we propose FedRAG, a highthroughput, privacy-preserving federated RAG framework designed specifically for cross-institutional knowledge synergy. At its core, FedRAG introduces a novel Scrambled Distributed Attention protocol. Instead of transmitting highly vulnerable intermediate states or relying on heavyweight cryp-

tographic protocols, participating institutions dynamically negotiate a numerically stable feature scrambling mechanism. This specially designed scrambling structure provides robust privacy guarantees while sustaining highly efficient and precise computation on hardware accelerators. By transmitting these scrambled tensors over the network to dynamically delegate computations to an independent compute node, FedRAG successfully decouples the self-attention execution from data localization. This innovative design ensures that no single entity can access the plaintext contexts of others, while fully leveraging the abundant computational resources and highspeed interconnects typical of institutional consortiums. Our core contributions can be summarized as follows: • We present FedRAG, a collaborative generative system that breaks down cross-institutional data silos. Our framework requires no model retraining, relies on no approximate operators, and eliminates the need for specialized hardware (e.g., TEEs), enabling seamless and rapid deployment of state-ofthe-art LLMs. • We design a highly efficient, non-cryptographic privacypreserving attention protocol. It provides robust mathematical and empirical defense against sophisticated intermediate state inversion attacks while replacing prohibitive cryptographic overhead with hardware-friendly linear transformations. • Extensive evaluations across various open-source LLMs and diverse benchmarks validate the promise of our system. FedRAG achieves up to a 62× latency reduction over existing secure baselines, sustaining practical, human-reading throughput. Crucially, this immense efficiency gain comes with near-zero model utility degradation across six question answering and summarization benchmarks.

2

Background

2.1

Transformers and Self-Attention

The Transformer [46] serves as the foundational architecture underlying contemporary LLMs, representing a highly efficient and high-performance neural network architecture for processing long sequences. A typical Transformer model is composed of a stack of layers, with each layer primarily consisting of a multi-head self-attention mechanism and a feed-forward network. At the core of this multi-head mechanism is the scaled dot-product attention. During computation, the model first linearly projects each token’s hidden state into three matrices1 : Queries (Q), Keys (K), and Values (V ). Since Q, K, and V are all linearly projected from the same input sequence, this mechanism is referred to as self -attention, and the attention output is computed as:  √  Attention(Q, K,V ) = softmax QK ⊤ / d V. (1) 1 Throughout this paper, vectors are represented as row vectors, and d

denotes the dimension of the attention head.

2

While this dense cross-token interaction enables the model to effectively understand long contexts, it inherently introduces a strict data dependency: the inference for any individual token dictates mandatory access to the representations (i.e., K and V ) of all other relevant tokens in the sequence. This intrinsic architectural trait forms a fundamental bottleneck when attempting to scale LLM inference across distributed or privacy-sensitive data silos, making the decoupling of attention computation and data localization a critical prerequisite for collaborative generation.

2.2

gets a highly practical and increasingly prevalent deployment scenario: cross-institutional collaborative RAG within a consortium. In this scenario, institutions are willing to share information to leverage LLMs for generating macroscopic insights, but they must strictly avoid exposing granular, recordlevel details to other participating entities. Consequently, we define the primary privacy objective of this work as preventing leakage of the documents’ raw plaintext. In such a cross-institutional setting, the participating nodes are large organizations rather than individual users. They typically operate within enterprise-grade infrastructure characterized by three main features: • Powerful Hardware Resources. Each participating node operates enterprise-grade servers equipped with high-end AI accelerators and massive storage systems, which provide the computational capacity required to execute large-scale LLM inference. • High-Speed and Predictable Interconnects. These institutions are often interconnected via dedicated optical networks or co-located within Data Center Colocation facilities, which provide high-speed connections of multi-Gbps and stable millisecond-level low latency. • Multi-Node Participation. The system inherently involves three or more independent participating nodes. According to the threat model introduced later, we assume no collusion occurs between any pair of these nodes. Existing privacy-preserving LLM inference frameworks predominantly focus on the traditional “edge-cloud” paradigm, which typically involves a resource-constrained edge device communicating over high-latency public networks [30, 60]. As a result, most existing work overlooks the critical need for inter-institutional federated inference. To fill this void, our method fully leverages the aforementioned advantages in system resources and multi-participant architectures, proposing a privacy-preserving federated RAG framework designed to satisfy the high-throughput and long-context demands of modern LLMs.

Retrieval-Augmented Generation

RAG [29] equips LLMs with external knowledge to mitigate the poor performance and hallucinations caused by static internal weights on domain-specific or time-sensitive tasks, all without requiring costly retraining. A standard query-based RAG pipeline typically executes in three steps. First, an embedding model maps the query into a dense vector space to perform coarse-grained document retrieval via approximate nearest neighbor search. Next, a cross-encoder reranker conducts precision filtering by evaluating fine-grained semantic interactions between the query and candidate documents. Finally, during response synthesis, the top-ranked documents are concatenated with the user query and jointly fed into the LLM to generate a grounded answer. However, deploying this pipeline in federated scenarios introduces a fundamental architectural conflict. When knowledge bases are distributed across mutually distrustful institutions, centralizing retrieved documents violates strict data privacy constraints. This data fragmentation severely impairs the reranking and generation stages, both of which mandate intensive cross-token attention computation between the user query and the retrieved documents to capture fine-grained semantic dependencies. Because the Transformer’s self-attention strictly demands access to all context tokens, executing this computation across distributed, plaintext-sensitive contexts naturally transforms into a challenging cross-node privacy problem, motivating the federated RAG pipeline formalized in §3.

3

3.2

To elucidate the system architecture, we formally define the problem of Federated RAG. Consider a consortium of M institutions, where each institution i maintains a private, localized document corpus Di . The global conceptual corpus is the S union of these isolated silos: D = M i=1 Di . When an authorized user submits a query q, the federated RAG pipeline aims to generate a comprehensive response y conditioned on both q and the most relevant documents distributed across D . The end-to-end pipeline can be broadly abstracted into two primary stages: global retrieval and distributed generation. During the global retrieval stage, the system collaboratively retrieves and aggregates the most relevant documents across the global conceptual corpus D to

System Model and Problem Formulation

In this section, we delineate the unique characteristics of our target deployment scenario, formalize the federated RAG pipeline, and define the threat model that guides our system design.

3.1

Problem Formulation

System Model and Target Scenario

While RAG significantly enhances the performance of LLMs in specialized domains, organizations possessing vast amounts of domain-specific data often face severe “data silo” problems due to privacy concerns. Bridging this gap, our work tar3

form a globally optimal context pool: C f inal = Retrieve (q, D1 , D2 , . . . , DM ) .

such malicious coordination exposes each colluding party to the catastrophic risk of being reported or audited by the other. Thus, rational, reputation-bound institutions will inherently refrain from collusion.

(2)

Subsequently, in the distributed generation stage, an LLM synthesizes the final response. For each generation step t, the model computes the attention outputs over the distributed context C f inal and the query q, generating the next token yt based on the previously generated tokens y<t : yt = LLM(C f inal , q, y<t ).

4 4.1

(3)

Distributed Attention Computation

In distributed RAG systems, the KV Cache of the LLM is partitioned across multiple nodes, requiring cross-node collaboration to execute the attention mechanism. To minimize cross-node communication, we build on the online-softmax decomposition widely adopted in efficient attention implementations [10, 32], which decomposes attention into two stages: per-node local computation followed by a lightweight global aggregation. In what follows, we formalize this decomposition in our federated setting; this framework will serve as the substrate on which our privacy-preserving protocol operates (§4.4). Per-Node Local Computation. Assume that the relevant KV Cache is distributed across M nodes. Let Ki and Vi denote the key and value matrices stored on node i, respectively. Given a query matrix Q, node i directly applies the standard attention computation to its local (Ki ,Vi ) producing two quantities: • Node-Local Attention Output. Applying Eq. (1) to the locally available K and V matrices gives   QK ⊤ Oi = softmax √ i Vi . (4) d

Problem Statement: During the distributed generation (and collaborative re-ranking, if applied), the core operation is the cross-node attention computation Attention(Q, K,V ). The fundamental problem our system addresses is how to compute this cross-node attention collaboratively when Q and K,V belong to mutually distrustful nodes. The system must ensure that neither party can reconstruct the other’s exact plaintext tensors, all while maintaining a strict end-to-end latency constraint suitable for real-time applications.

3.3

Scrambled Distributed Attention

Threat Model and Security Assumptions

Given the consortium-based scenario, our system operates under a highly practical threat model tailored for crossinstitutional collaborations. Honest-but-Curious (Semi-Honest) Adversaries: We assume all participating nodes are honest-but-curious. Nodes will strictly adhere to the predefined cryptographic protocols, correct execution of neural network forward passes, and timely transmission of intermediate states. However, they are “curious” and may attempt to passively log the exchanged intermediate activations (e.g., scrambled Q, K, V , and attention outputs) to infer or reconstruct the original private text of other institutions. Our primary defensive objective is to thwart such passive reconstruction attacks, notably intermediate state inversion [11] and Vocabulary Mapping Attacks (VMA) [44]. Reputation-Aware and Non-Collusion Constraints: Unlike anonymous P2P networks, institutions in our scenario (e.g., hospitals, banks) are bound by stringent legal frameworks (e.g., HIPAA, GDPR) and commercial contracts. Therefore, we introduce a Reputation-Aware assumption: participants are highly risk-averse regarding their institutional reputation. An adversary will actively refrain from any malicious behavior (such as deliberately feeding poisoned data, intentionally altering model weights, or violating protocol rules) if such actions have a high probability of leaving verifiable traces subject to future auditing. Active attacks are thus considered out of scope. Furthermore, this reputationaware nature inherently enforces a Non-Collusion assumption among participating institutions. Any attempt to collude— where two organizations explicitly coordinate to share cryptographic keys or intermediate tensors to decrypt a third party’s data—inevitably leaves severe digital footprints (e.g., abnormal out-of-band communication records). More importantly,

This output is normalized only over the keys stored on node i and thus does not yet reflect the global attention distribution. • Local Normalization Factor. To recover the global attention distribution across all nodes, we also compute the corresponding normalization factor. Let wi ∈ RLQ denote the vector whose ℓ-th entry aggregates the unnormalized softmax weights on node i:   QK ⊤ LVi [wi ]ℓ = ∑t=1 exp √ i . (5) d ℓ,t Global Aggregation. Finally, the local results from all nodes are communicated and aggregated to reconstruct the global attention output. The global attention output is reconstructed as Attention(Q, K,V ) =

∑M i=1 wi ⊙ Oi , ∑M i=1 wi

(6)

where ⊙ and the division are applied row-wise. This two-stage decomposition decouples the attention computation across nodes while preserving mathematical equivalence to the centralized formulation. Crucially, it exposes 4

a clean interface—each node only exchanges (Oi , wi ) with the aggregator—on which we layer our privacy-preserving transformation in §4.4.

Our construction. We construct the scrambling matrix as: Φ = S1 P1 HP2 S2 ,

where H ∈ Rd×d is a normalized Hadamard matrix, P1 , P2 are independent random permutation matrices, and S1 , S2 are random diagonal matrices whose entries possess random signs and variable magnitudes. Each component serves a distinct role: • Normalized Hadamard matrix H simultaneously addresses both goals from above. Its base {−1, +1} entries allow matrix multiplication to be computed via √ stable signflipping additions followed by a single global 1/ d scaling, avoiding the floating-point errors of arbitrary dense multiplications (numerical stability). At the same time, its√ orthogonality ensures that each output coordinate is a ±1/ d-weighted sum of all input coordinates (sufficient mixing). • Permutation matrices P1 , P2 randomize the otherwise deterministic H. A fixed H alone is publicly known; sandwiching it between two random permutations yields a distribution over mixing matrices. Permutations are exact operations, so they preserve numerical stability. • Scaling matrices S1 , S2 distort distances. Pure permutation-and-Hadamard preserves L2 distances between rows, exposing topological information (e.g., kNN structure) that enables graph-matching attacks (§6). The random scaling factors severely disrupt these distance relationships. Their magnitudes are sampled from an empirical range (e.g., [1/8, 8]), representing a fundamental trade-off between privacy enhancement and floating-point precision. The two-layer sandwich structure (S1 P1 · H · P2 S2 ) ensures that both the input and output sides of H are randomized, so partial knowledge of one layer does not help an attacker invert the other. Security under structured Φ. A natural concern is whether this structured Φ—which has fewer effective degrees of freedom than a fully random matrix—provides sufficient privacy. In §6, we show that it does: the highly collinear geometry of LLM hidden states [15, 16] fundamentally limits any algebraic inversion attack, making the structural constraints on Φ largely inconsequential in practice. We defer the detailed analysis to §6. Random Token Permutation. In addition to scrambling the feature dimension, we introduce random permutation in the token dimension. Due to the permutation-equivariant nature of attention computation, applying a consistent permutation to the K and V matrices yields an attention output from which the original result can be exactly recovered. Similarly, permuting the Q matrix results in an attention output that, upon applying the inverse permutation, matches the original output. This token-level shuffling significantly enhances system resilience against known-plaintext attacks and brute-force enumeration. Scrambled Attention Protocol. Consider three matrices Q = [q1 ; . . . ; qLQ ] ∈ RLQ ×d , K = [k1 ; . . . ; kLK ] ∈ RLK ×d , and

4.2 Privacy-Preserving Attention Computation Random Feature Scrambling. Let Φ ∈ Rd×d be an invertible matrix, which we call a scrambling matrix. Given a vector x ∈ Rd , we refer to xΦ or xΦ−⊤ as a scrambled vector; the choice between Φ and Φ−⊤ depends on the role of x in the downstream attention computation, as will become clear in §4.4. The KV Cache of the LLM is partitioned across multiple nodes, requiring cross-node collaboration to execute the attention mechanism. We defer the concrete construction of Φ to the next paragraph, since the following two algebraic properties hold for any invertible Φ. Lemma 1 (Inner Product Preservation). For any two vectors x1 , x2 ∈ Rd and any invertible matrix Φ ∈ Rd×d , scrambling with the dual matrices Φ and Φ−⊤ preserves the inner product: ⟨x1 Φ, x2 Φ−⊤ ⟩ = (x1 Φ)(x2 Φ−⊤ )⊤ = ⟨x1 , x2 ⟩.

(9)

(7)

Lemma 2 (Linearity). For any vectors x1 , . . . , xn ∈ Rd scrambled with the same matrix Φ ∈ Rd×d , and any scalar weights w1 , . . . , wn ∈ R:
n n (8) ∑i=1 wi (xi Φ) = ∑i=1 wi xi Φ. These two properties are chosen by design: they correspond precisely to the two core operations of scaled dotproduct attention. Lemma 1 preserves the Q-K inner products that determine attention logits, while Lemma 2 preserves the V-weighted summation that produces attention outputs. Together, they enable a computation party to execute attention on scrambled (Q, K,V ) and return a result that, after output-side descrambling, equals the plaintext attention—without ever observing the plaintext tensors. We formalize this protocol in §4.4. Constructing a Numerically Stable Scrambling Matrix. While Lemmas 1 and 2 hold for any invertible Φ, a naive choice—such as a fully random dense matrix sampled from a continuous distribution—fails in practice. Modern LLM inference runs in low-precision floating-point formats (BF16 or FP16), which offer only 7-10 bits of mantissa. A random dense Φ produces entries of widely varying magnitudes, causing the matrix-vector products xΦ and xΦ−⊤ to suffer from severe round-off and cancellation errors. These errors accumulate across the dozens of transformer layers, degrading generation quality to the point of unusability (please refer to Appendix A for an empirical demonstration). A practical scrambling matrix must therefore reconcile two seemingly conflicting goals: sufficient mixing to obscure the plaintext, and numerical stability under low-precision arithmetic. 5

① Embedding Retrieve ② Re-ranking Query Candidate Collaborative Nodes Documents 𝐾𝐾/𝑉𝑉/𝑄𝑄 Attn. Output

Coordinator

Embedding Vector

Similarity

Distributed Database Collaborative Nodes Docs LLM ③ Generation

Attn. Output

Attn. Output Ranking Coordinator Scores 𝐾𝐾/𝑉𝑉/𝑄𝑄

𝐾𝐾/𝑉𝑉

Attention KV Cache

Coordinator 𝑄𝑄

Attn. Output

Answer

Figure 1: System architecture of the federated RAG system. V = [v1 ; . . . ; vLV ] ∈ RLV ×d , where LV = LK . We generate two random scrambling matrices ΦKQ , ΦV ∈ Rd×d and two permutation matrices PQ ∈ RLQ ×LQ , PKV ∈ RLV ×LV to form the key set Θ. The workflow is as follows: 1. Plaintext Matrix Scrambling: Enc(Q, K,V, Θ) = (PQ QΦKQ , PKV KΦ−⊤ KQ , PKV V ΦV ) = (Q′ , K ′ ,V ′ ).

(10)

2. Scrambled Attention Computation:   ⊤ √ Attention(Q′ , K ′ ,V ′ ) = softmax Q′ K ′ / d V ′  √  ⊤ = softmax PQ QΦKQ (PKV KΦ−⊤ KQ ) / d (PKV V ΦV )  √  ⊤ = PQ softmax QK ⊤ / d PKV (PKV V ΦV ) = PQ Attention(Q, K,V )ΦV .

(11)

3. Output Descrambling: Dec(Attention(Q′ , K ′ ,V ′ ), Θ) = PQ−1 Attention(Q′ , K ′ ,V ′ )ΦV−1 = Attention(Q, K,V ).

(12)

In this design, no single node can access the plaintext data of matrices from other nodes, yet the system correctly executes the attention computation.

5

System Overview and Node Roles

A complete RAG request lifecycle consists of three main phases: vector retrieval, re-ranking, and generation. The system architecture is illustrated in Figure 1. In the vector retrieval phase, the Coordinator first converts the received user query into a vector representation using an embedding model and broadcasts it to all collaborating nodes. Each node performs vector retrieval against its local documents, returning the identifiers and similarity scores of the top-k most relevant documents to the Coordinator. The Coordinator then aggregates and ranks these scores to select the top-k global candidates for the subsequent re-ranking stage. During reranking, the Coordinator and the collaborating nodes jointly execute the re-ranking model inference to determine the final top-m (m < k) documents that constitute the context for the RAG request. Finally, in the generation phase, the Coordinator collaboratively performs LLM inference with other nodes holding the retrieved contexts to generate the final response. Since both the re-ranking and generation phases require cross-node attention computation, we first formalize the roles of participating nodes. To execute the privacy-preserving attention mechanism across the federated network, we categorize the nodes into three distinct roles based on their functions during each cross-node attention computation: • Inquirer. The node currently running the forward pass for unprocessed tokens. It possesses the plaintext Query required for the attention computation. • Context Owner. The node possessing the plaintext K and V matrices required for the attention computation. • Compute Node. The node responsible for executing the actual attention computation. It receives scrambled matrices from the Inquirer and Context Owner, computes the result, and returns it to the Inquirer. For each cross-node attention computation, the Inquirer and Context Owner share the scrambling key set Θ. The Compute Node does not possess these keys and thus cannot reconstruct the information, ensuring the security of the privacy-preserving protocol. These roles are applied dynamically across both the re-ranking and generation stages.

Compute Node

Compute Node

𝑄𝑄

5.1

5.2

Collaborative Re-ranking

Building upon the vector retrieval results, the re-ranking stage employs specialized re-ranking models (e.g., BGE Reranker [5], mGTE [55]) to recalibrate the relevance scores between candidate documents and the query. Unlike the biencoder architecture used in the preceding vector retrieval phase, which independently maps queries and documents into a shared vector space to compute coarse-grained similarities, this stage adopts a cross-encoder approach. It directly models the fine-grained, token-level interactions between each querydocument pair, thereby significantly improving retrieval quality.

The RAG Pipeline

This section details the lifecycle of a single user request within the federated RAG system. When an authorized user from a participating organization requires generative services, they submit a query to their local node. This receiving node assumes the role of the Coordinator for the request, orchestrating subsequent operations alongside other collaborating nodes in the federation system. 6

Layer Input

Context Owner

Hidden State

pass of a decoder-only transformer, each token only computes attention with its preceding tokens, rather than all tokens in the sequence as is characteristic of encoder architectures. During the generation process, the intermediate state (i.e., the KV Cache) of each token remains unchanged once computed. For distributed generation in our system, this implies that the roles of nodes participating in the decoder’s attention computation are asymmetric: a Context Owner does not need to perform layer-by-layer attention computation synchronously with the Inquirer possessing subsequent tokens. Instead, it can complete the KV Cache computation for its local tokens in advance and transmit them to the Compute Node ahead of time. We illustrate the forward pass of a single decoder layer and our distributed attention computation in Figure 2. The generation process comprises a prefill phase and a decoding phase. The former builds the KV Cache for the input tokens, while the latter generates new tokens step-bystep based on the prefilled cache. Unlike centralized RAG systems, the prefill and decoding attention computations in a federated RAG system must be coordinated across nodes due to the distributed nature of documents and queries. We define a continuous sequence of plaintext tokens held by a single node as an input segment. The global input sequence consists of ordered input segments T1 , T2 , . . . , Tns , owned by nodes O (T1 ), O (T2 ), . . . , O (Tns ), respectively. Nodes owning these segments sequentially perform the forward pass to complete attention caching based on the segment’s position in the sequence. Prefill Phase. When the tokens in segment Ti undergo the forward pass, the owners of all preceding segments, O (T1 ), . . . , O (Ti−1 ), act as Context Owners. They scramble the K and V of their already-prefilled tokens and transmit them to the Compute Node in advance. Concurrently, the owner of Ti assumes the role of the Inquirer. For each transformer layer, the Inquirer computes the local Q, K, and V matrices for its current segment and performs a partitioned attention calculation. As mathematically formulated in §4.1, it first derives the local attention output utilizing its own K, V , and Q matrices. Next, the Inquirer scrambles its local Query matrix and dispatches it to the Compute Node. The Compute Node executes the attention computation between this scrambled Query and the scrambled K, V previously received from the Context Owners, and subsequently returns the result to the Inquirer. Upon receiving this result, the Inquirer descrambles it and aggregates it with the previously computed local attention output to finalize the attention computation for the current segment. The aggregated attention output is then fed into the subsequent FFN to complete the forward pass for that specific layer. This layer-wise process is iterated until the forward computation for all layers is concluded. Finally, during transmission idle periods, the Inquirer scrambles the K and V matrices of its newly processed segment and forwards them to the Compute Node, making them available as context for subsequent prefill and decoding stages.

K/V

Linear Projection Scaled Dot-Product Attention

K/V

Q

Scrambling Scrambling

Local Attention Feed Forward Network Layer Output

Local Attention Compute Node

Descrambling & Aggregation

Figure 2: Forward pass of a decoder layer and the distributed attention computation in the generation phase.

Input Construction. In this stage, the model input is a concatenation of the query and the candidate document: [CLS] ⊕ query ⊕ [SEP] ⊕ document ⊕ [SEP], where [CLS] and [SEP] are special tokens functioning as the global sequence representation and the segment separator, respectively. In a federated RAG system, the query and documents may reside on different nodes, necessitating cross-node attention computation. We utilize the scrambling-based privacypreserving attention method introduced in §4 to execute collaborative re-ranker inference without leaking private information. Privacy-Preserving Forward Pass. Most classical reranking models typically adopt an encoder-only architecture utilizing bidirectional attention [48, 52]. This implies that the attention computation for every single token requires full access to the intermediate states of the entire sequence (i.e., both preceding and succeeding contexts). Consequently, participating nodes simultaneously act as both Inquirers (seeking attention outputs for their own tokens) and Context Owners (providing their K and V matrices to others) during the forward pass of each layer. Nodes independently compute the Q, K, and V matrices for their local tokens based on the output of the previous layer. These matrices are scrambled and transmitted to the designated Compute Node responsible for the attention calculation. Once the computation is complete, this node sends the corresponding attention outputs back to the respective Inquirer nodes owning the tokens. The participating nodes then descramble the results and proceed with the subsequent Feed-Forward Network (FFN) computations to complete the layer’s inference. This process repeats until all layers are processed, requiring bidirectional communication with the Compute Node Lreranker times (where Lreranker is the number of model layers). Finally, the hidden state of the [CLS] token is mapped to a relevance score.

5.3

Collaborative Generation

In the RAG generation phase, the retrieved results are fed into the LLM along with the user input. Modern LLMs predominantly rely on a decoder-only architecture. In the forward 7

Decoding Phase. This phase commences after the last input segment completes prefill. Since the final input segment exactly corresponds to the user’s initial query, its owner, O (Tns ), is inherently the Coordinator of the RAG request. Having already assumed the Inquirer role during the prefill of this final segment, the Coordinator naturally retains this identity to commence generating new tokens, while the owners of all preceding segments serve as Context Owners. The decoding forward pass mirrors the prefill phase, with the distinction that it processes one newly generated token at a time to predict the next token. Furthermore, the Inquirer does not need to send the newly computed K and V of the generated token to the Compute Node immediately. For each layer, the Inquirer scrambles the new token’s Query vector, sends it to the Compute Node (which already holds the scrambled KV Cache from the prefill phase), receives the result, descrambles it, and proceeds with the FFN. Upon completing the final decoder layer, the Inquirer samples the new token and initiates the next generation step. This process repeats until a termination condition is met (e.g., reaching a maximum length limit or generating an end-of-sequence token).

5.4

pute Node. This strategy eliminates the need to redundantly transmit the K and V information of any input segment.

6

Privacy Analysis

State Inversion & Text Recovery. State inversion attacks aim to recover original tokens by accessing the model’s intermediate activation states. Representative methods include VMA [44], white-box optimization-based inversion [11, 14], and black-box generation-based inversion [11, 35]. VMA relies on traversing the vocabulary and utilizing the sorted L1 distance for ANN collisions. However, the dense mixing introduced by the Hadamard matrix H and the non-isometric scaling by S1 , S2 in our scrambling scheme thoroughly destroy the distance relationships between the plaintext and ciphertext, rendering VMA ineffective. White-box optimization-based inversion requires calculating the loss via forward propagation to construct a differentiable distance function; since the computational node lacks the scrambling matrix Φ, backpropagation cannot be performed. Furthermore, black-box generation-based inversion relies on a stationary feature space to train a Seq2Seq inverse mapping model. In contrast, our system dynamically negotiates Φ for each layer at the beginning of every RAG request, preventing the attacker from obtaining a stable feature space for model convergence. Consequently, to execute the aforementioned text recovery attacks, an adversary must satisfy an absolute prerequisite: unmixing the dense scrambled representation manifold to extract the plaintext intermediate states. This forces the attacker to resort to low-level algebraic unmixing attacks. Defeating Independent Component Analysis (ICA). ICA is a classic blind source separation algorithm that aims to extract original underlying signals from their linear mixtures without prior knowledge of the mixing parameters [21]. We evaluated the resistance of our method against ICA attacks, as shown in Figure 3a. The five curves represent the Hungarian Mean Absolute Cosine Similarity between the attack results and the targets under different scenarios: no leakage (knowing only the scrambling structure), known P2 , known S2 , a random guess baseline based on the empirical distribution of intermediate states, and a positive control using independent Laplace sources. These results demonstrate that even under pessimistic assumptions of partial information leakage, the recovery similarities plateau near the random guess baseline and fail to approach an exploitable degree. In contrast, the IID Laplace positive control can be almost perfectly recovered. This disparity demonstrates that our scheme naturally resists ICA as the highly collinear intermediate states (i.e., the anisotropy problem) of modern LLMs intrinsically violate the independent source assumption [15, 16] upon which the algorithm relies. Graph Matching Attacks. Graph matching attacks are non-parametric attack schemes that attempt to reconstruct features by aligning the relative distance matrices of the scram-

Dynamic Role Assignment

Our privacy-preserving attention scheme employs a symmetric-like scrambling mechanism, implying that any node capable of scrambling is also capable of descrambling. This necessitates a strict constraint: a node acting as a Compute Node in a specific attention computation must not simultaneously be a Context Owner or an Inquirer. Otherwise, the Compute Node could use the shared key to reverse the counterparty’s data, violating privacy. This constraint implies that our federated RAG system requires a minimum of three independent physical nodes. If there exists a node in the system that does not own any input segments, it can permanently serve as the Compute Node for all cross-node attention computations in that request. However, in scenarios with few nodes and a uniform document distribution, all nodes may own input segments. This requires dynamically assigning the Compute Node role to at least two different physical nodes during a request. When physical node N1E acts as a Context Owner, N2E must serve as the Compute Node, and vice versa. This implies that the KV Cache for certain segments must be replicated to both N1E and N2E , incurring additional communication overhead. Let maxSeg(Ni ) = maxO (T j )=Ni j denote the highest index of an input segment owned by physical node Ni . Consider two physical nodes assigned the Compute Node role, N1E and N2E , with maxSeg(N1E ) < maxSeg(N2E ). To minimize the overhead of redundant transmission, we strategically select the node with the smallest maximum segment index as one of the Compute Nodes. Specifically, if there exists a physical node Ni such that maxSeg(Ni ) = 1 (i.e., it owns only the very first input segment), we can permanently assign it as a Com8

Sequence Length

(a) Resistance to ICA.

0.8

Neighbor Overlap

Matched Mean |Cos|

1.0 No Leakage Known P2 0.8 Known S2 0.6 IID Laplace Rand. Guess 0.4 0.2 24 25 26 27 28 29 210 211 212

0.6 0.4

Key Query Value Random

Table 1: Length configurations in throughput evaluations. Config Short Medium Long

0.2 0.0 4 5 6 7 8 9 10 11 12 2 2 2 2 2 2 2 2 2

Output 32 64 128

(b) Resistance to graph matching.

ers, network latency, and network bandwidth—on system efficiency. Next, we benchmark the generation quality of our method across six diverse datasets to demonstrate that our scrambling mechanism incurs negligible degradation in model accuracy. Finally, we investigate the accuracy-efficiency tradeoffs of quantizing intermediate states to provide insights for future optimizations.

bled space and the known plaintext spaces. Since exact subgraph matching is fundamentally an NP-Hard problem [8], existing efficient graph matching algorithms heavily rely on the topological stability of local spatial features [43]. As illustrated in Figure 3b, experiments demonstrate that the kNN overlap rate between the plaintext and ciphertext spaces drops rapidly as the sequence length L increases. At L = 4096, the local neighborhood overlap for Q, K, and V all plummet below 15%, which is vastly below the theoretical threshold that existing efficient alignment methods can exploit [9]. Malicious Prompt. To counter the threat of request initiators attempting to steal other nodes’ documents via maliciously crafted prompts, our system employs a game-theoretic defense mechanism based on ciphertext auditing. The initiating node of a RAG request (i.e., the Coordinator) must send the scrambled query to the computational nodes, and its keys are shared with at least one other node in the system. In the event of a dispute, the federation can collaboratively audit these scrambled records. This ensures proactive suppression of malicious attacks under a “reputation-aware” mechanism.

7.1

Query 32 64 128

Sequence Length

Figure 3: Security evaluation of the scrambling mechanism against algebraic and topological attacks.

7

Context 256+256 1024+1024 4096+4096

Baselines. We select SCX [53] and PermLLM [59] as our primary baselines, as they share similar characteristics with our approach: they do not require model retraining, do not mandate TEEs, and avoid approximate operators. For SCX, we implement it according to the non-TEE environment settings described in their paper. For PermLLM, to ensure a fair comparison, we strictly apply its cryptographic operations solely to the attention computation. Models. We evaluate our system using multiple state-ofthe-art open-source LLMs, including the Qwen 3 [49], Llama 3.1 [17], Ministral 3 [31], and GPT-OSS series [1], with parameter sizes ranging from 4B to 20B. All models are obtained from the Hugging Face model hub and executed using their default data types. For the RAG retrieval phase, we uniformly employ bge-m3 and bge-reranker-v2-m3 [5] as the embedding and reranking models, respectively. During the reranking stage, we apply 4-bit quantization to the intermediate states, which we discuss in further detail in §7.5.

Evaluation

Datasets. For throughput and latency evaluations, we generate input token sequences of fixed lengths, with configurations detailed in Table 1. Our experimental setup distributes context tokens across two nodes, each hosting two equal-length documents. For the reranking phase, we use 10 candidate documents for recalibration.

Evaluation Setup

We implement our privacy-preserving mechanism as an APIcompatible, drop-in replacement for the standard attention function in the Hugging Face transformers library. We conduct our evaluations on a server equipped with one NVIDIA PRO 6000 and two NVIDIA RTX 5090 GPUs. To enforce inter-node isolation, we deploy the participating nodes within separate Docker containers, and we employ Toxiproxy as the network proxy between containers to simulate diverse network environments with varying latency and bandwidth constraints. In this section, we first evaluate the end-to-end latency, throughput, and communication volume of our method compared to existing privacy-preserving LLM inference techniques in RAG scenarios. Subsequently, we analyze the impact of various factors—such as the number of protected lay-

For accuracy evaluation, we select several prominent question-answering (QA) and summarization datasets equipped with reference contexts. These benchmarks cover a diverse range of domains, context lengths, and reasoning complexities. Specifically, we evaluate our system on five standard QA datasets—SQuAD [40], HotpotQA [51], MuSiQue [45], MS MARCO [3], and AmbigQA [34]—along with one long-form meeting summarization benchmark, QMSum [61]. Detailed descriptions of these datasets are deferred to Appendix B. 9

103

103

Ours 103

102

102

102

102

101

101

101

101

Total Time (s)

103

SCX

PermLLM

102

102

102

101

101

101

101

100

100

100

102

Short Med. Long Short Med. Long Short Med. Long Short Med. Long Short Med. Long Short Med. Long Short Med. Long Short Med. Long

Qwen3 4B

Fully Protected

Llama3.1 8B

Fully Protected

Ministral3 14B Fully Protected

GPT-OSS 20B

Fully Protected

Qwen3 4B

2 Protected Layers

Llama3.1 8B

2 Protected Layers

Ministral3 14B

2 Protected Layers

GPT-OSS 20B

2 Protected Layers

Figure 4: End-to-end latency of RAG requests. Table 2: Performance evaluation for fully protected generation. Length

meta-llama/Llama-3.1-8B-Instruct

Method

openai/gpt-oss-20b

TTFT (s) ↓ Dec. TPS ↑ Traff. (MiB) ↓ Comm. Rounds ↓ TTFT (s) ↓ Dec. TPS ↑ Traff. (MiB) ↓ Comm. Rounds ↓ Ours PermLLM SCX

3.23 11.93 6.22

11.70 1.62 5.13

301.17 4721.73 1791.25

1147 10851 3458

1.39 9.68 3.52

10.19 1.79 5.94

220.66 4224.67 1181.29

867 8163 2618

Ours Medium PermLLM SCX

4.83 89.23 15.09

11.79 1.46 5.48

1107.61 45815.71 6752.25

2171 21091 6530

2.45 73.47 10.13

10.28 1.62 5.98

804.73 38419.13 4454.75

1635 15843 4922

Ours PermLLM SCX

12.00 1243.09 60.15

11.85 0.96 5.50

4238.54 587393.74 26202.25

4219 41571 12674

7.72 966.51 39.13

13.54 1.16 6.01

3037.73 483843.42 17274.69

1647 31203 9530

Short

Long

7.2 End-to-End Latency and Communication Overhead

maximize the computational payload delegated to the Compute Node within a single cross-node communication round, and we rely exclusively on hardware-efficient matrix multiplications for both plaintext encryption and ciphertext computation, thereby minimizing latency induced by cryptography and communication.

Figure 4 illustrates the total end-to-end latency of a single RAG request for our method and the baselines across varying prompt lengths. Although prior studies have demonstrated that transmitting plaintext intermediate states from any single layer can lead to privacy leakage (as discussed in §6), we follow the partial-layer protection strategy adopted by SCX and introduce an additional configuration that only protects two layers for comparison, with results presented in the second row of Figure 4. In practice, this partial-layer protection strategy serves as a viable trade-off option between system efficiency and privacy, allowing users to configure the system according to their specific operational requirements. As shown in Figure 4, our method achieves the lowest endto-end latency across all six configurations and four models, yielding a 1.71× to 62.96× speedup over the baselines. Table 2 details the comprehensive efficiency metrics for the Llama3.1-8B model, with results for other models provided in Appendix C. The table highlights that our approach achieves significantly superior results in Time to First Token (TTFT), Decode TPS, communication rounds, and communication volume. Crucially, the overhead of our method exhibits a linear growth trend with respect to sequence length, endowing our framework with excellent scalability for increasingly prevalent long-context workloads. We attribute this substantial efficiency improvement to the core design principle that we

7.3

Throughput and Network Sensitivity

Figure 5 plots the Decode TPS metrics under varying network latencies. Here, the number of nodes denotes the physical machines that act as Compute Nodes during a given RAG request; as articulated in §5.4, if a request requires all nodes in the system to supply tokens, we must dynamically assign the Compute Node role to two distinct nodes to guarantee privacy protection. The empirical results demonstrate that with a Round-Trip Time (RTT) of approximately 1ms, our method sustains Decode TPS greater than 10 under the full-layer protection configuration. For collaborative RAG scenarios deployed across physical servers within the same data center, across availability zones in the same region, or across departments within a research institution, our system can deliver output speeds matching human reading rates while providing uncompromising privacy [27]. When the RTT increases to 10ms, the Decode TPS drops to single digits; in such scenarios, operators can resort to offline generation or intentionally trade a portion of privacy for enhanced throughput. Neverthe10

Decode TPS

(a) Qwen3 4B, 1 node 40 30 20 10 0 24 8

(b) Llama 3.1 8B, 1 node

60 45 30 15 36 0 24 8

16

Decode TPS

(f) Qwen3 4B, 2 nodes

(c) Qwen3 14B, 1 node

32 24 16 8 32 0 24 8 16

16

(d) Ministral 3 14B, 1 node (e) GPT-OSS 20B, 1 node

32 24 16 8 40 0 24 8 16

20 15 10 5 40 0 2 4 8

16

24

(g) Llama 3.1 8B, 2 nodes (h) Qwen3 14B, 2 nodes (i) Ministral 3 14B, 2 nodes (j) GPT-OSS 20B, 2 nodes

60 32 40 32 20 45 24 30 24 15 30 16 20 16 10 15 8 10 8 5 0 24 8 16 0 0 0 36 24 8 16 32 24 8 16 40 24 8 16 40 0 2 4 8 16 24 Protected Layers Protected Layers Protected Layers Protected Layers Protected Layers 0ms 0.5ms 1ms 5ms 10ms Unprotected Local Decoding Baseline Figure 5: Decode throughput under different protected-layer counts and network settings. Llama 3.1 8B

Qwen3 14B

1000

GPT-OSS 20B

7.4 Utility Preservation on Downstream Tasks In §4, we theoretically proved that our privacy-preserving method is mathematically equivalent to the standard attention computation. However, in practical deployments, floatingpoint arithmetic inevitably introduces numerical precision errors, which can cause deviations in the model’s final output. To evaluate the impact of these deviations on model capabilities, we run benchmark evaluations across six datasets. We select three instruction-tuned models from the previous tests and additionally introduce the Qwen2.5-32B-Instruct model; possessing 64 hidden layers, it better exposes the cumulative effects of precision loss during the forward pass. We design ablation baselines for the benchmark, sequentially introducing the distributed attention computation and the intermediate state scrambling mechanism to analyze their respective impacts on model utility. The experimental results are summarized in Table 3. The accuracy reported in the table denotes the proportion of samples where the ground truth answer appears within the model’s output; for the meeting summarization dataset QMSum, we use the F1 score instead of accuracy. Furthermore, we measure the ROUGE-L overlap between the generated output and the ground truth, as well as the perplexity (PPL) of the ground truth given the model. The empirical results reveal that we maintain a virtually identical PPL compared to the standard attention baseline, with accuracy and ROUGE-L experiencing negligible average drops of roughly 0.06% and 0.05%, respectively. These findings confirm that our precision-preserving scrambling implementation enables privacy protection with near-zero accuracy degradation.

10.0

500 0

Ministral 3 14B

12.5

Decode TPS

Prefill TPS

Qwen3 4B 1500

100

10

1

Bandwidth (Gbps)

(a) Prefill TPS.

0.1

7.5 5.0100

10

1

Bandwidth (Gbps)

0.1

(b) Decode TPS.

Figure 6: Throughput vs. bandwidth.

less, our throughput remains significantly higher than existing cryptographic solutions, which typically require several minutes to generate a single token [12]. The throughput for configurations employing two Compute Nodes is depicted in the second row of Figure 5. Compared to the single Compute Node setup, the throughput exhibits an average degradation of roughly 10%. This decline primarily occurs because, during the decoding phase, the Inquirer must transmit the query vector of the token to the second Compute Node and wait for both nodes to finalize their computations before proceeding to the subsequent inference step. We further investigate the impact of network bandwidth on throughput, with the results shown in Figure 6. The two subfigures illustrate the throughput trends for the prefill and decoding phases, respectively. We observe that network bandwidth significantly affects the prefill phase, whereas the Decode TPS does not exhibit noticeable degradation until the bandwidth drops to 100Mbps. This discrepancy is due to the fact that the prefill phase involves fewer communication rounds but transmits massive volumes of data per round, whereas the decoding phase is the exact opposite; consequently, network latency serves as the predominant performance bottleneck for decoding.

7.5 Accuracy-Efficiency Trade-offs via Quantization Quantization serves as a vital optimization to reduce communication overhead, which is particularly prominent during 11

Table 3: Impact of Distributed Attention and Feature Scrambling on Downstream Task Performance. Method

SQUAD

HOTPOTQA

MARCO

AMBIGQA

MUSIQUE

QMSUM

Acc ↑ R-L ↑ PPL ↓ Acc ↑ R-L ↑ PPL ↓ Acc ↑ R-L ↑ PPL ↓ Acc ↑ R-L ↑ PPL ↓ Acc ↑ R-L ↑ PPL ↓ F1 ↑ R-L ↑ PPL ↓

Acc@3 (FP32) Acc@1 (FP32) Acc@3 Acc@1

16 8 7 6 5 4 3 2

Quantization Bits

(a) Accuracy under quantization.

69.40 78.26 69.35 78.24 69.37 78.21

3 2

3.07 3.07 3.07

22.16 45.92 22.09 45.84 22.11 45.84

Score L1 Dist Rank Edit Dist

1 0

16 8 7 6 5 4 3 2

4.23 4.23 4.24

8 6 4 2 0

49.94 64.02 49.89 63.95 49.79 63.91

17.0 17.0 16.9

63.93 66.79 63.82 66.66 63.79 66.67

3.33 3.33 3.34

26.29 20.24 26.55 20.38 26.53 20.32

16.1 16.1 16.1

allowing systems to dynamically trade rigorous privacy for communication efficiency.

Rank Edit Distance

1.0 0.8 0.6 0.4 0.2

3.06 3.06 3.06

Score L1 Distance

Rerank Accuracy

Baseline 89.43 89.67 +Distributed 89.42 89.68 +Scrambler 89.42 89.68

8

Discussion

System Scalability. In a typical RAG pipeline, the number of documents utilized during the reranking and generation phases is bounded by a predefined constant (i.e., top-k and top-m documents). Consequently, the number of participating nodes in these two stages does not scale indefinitely with the overall system size. As the institutional consortium expands, the primary performance bottleneck shifts to the global vector retrieval phase. Fortunately, a substantial body of existing work has extensively investigated efficient distributed vector search in large-scale systems [24, 47]. These established techniques can be orthogonally integrated into our framework to alleviate retrieval bottlenecks. Architectural Compatibility. Since our privacypreserving mechanism exclusively modifies the attention computation, it is inherently compatible with a wide spectrum of multimodal models that rely on the standard Transformer architecture. Furthermore, for emerging sub-quadratic architectures (e.g., Mamba [18], DeltaNet [50]), our framework remains adaptable. For layers utilizing these alternative mechanisms, the intermediate states can be directly transmitted to the Inquirer node for local computation, while the remaining standard attention layers can continue to leverage our distributed, scrambled attention protocol. Stronger Privacy Guarantees. While our previous evaluations primarily explored trading a portion of privacy for enhanced throughput under constrained networks, our scrambling methodology can be naturally extended to provide stronger privacy guarantees in scenarios with abundant bandwidth or stringent confidentiality requirements. By deliberately exchanging communication overhead for heightened security, we can apply more aggressive obfuscation to the intermediate states, particularly in shallower layers. Potential extensions include elevating the transmission precision of intermediate states to 32-bit floating-point coupled with a fully random dense scrambling matrix, as well as applying dimensional expansion to the K and Q matrices padded with orthogonal random noise. These enhancements would further resist sophisticated algebraic unmixing attempts. Future Work. Our work represents an early exploration into privacy-preserving federated RAG systems, leaving sev-

Quantization Bits

(b) Deviation from the FP32 baseline.

Figure 7: Impact of intermediate activation quantization on the Reranker.

the reranking phase (consuming ∼70% of the total transmission using the model’s default precision). To investigate this accuracy-efficiency trade-off, we apply standard affine min-max quantization [23] to the intermediate states during reranking. Using the SQuAD dataset, we retrieve the top-10 candidate documents via dense embeddings and re-score them using the quantized reranker. Figures 7a and 7b illustrate the impact of varying quantization bit-widths on retrieval quality and raw output deviations. While absolute score deviations (L1 and Rank Edit distances) increase progressively below 8-bit precision, the actual reranking accuracy (Acc@1 and Acc@3) remains remarkably robust down to 4-bit quantization, maintaining performance comparable to the FP32 baseline. This robustness stems from the fact that reranking efficacy depends strictly on the relative ordering of scores rather than their absolute logits, providing a sufficiently large margin to absorb quantization errors. At 3-bit and below, however, inconsistencies rapidly spike, rendering the results unusable. Thus, we strongly recommend 4-bit quantization for the reranking phase as the optimal balance between communication efficiency and retrieval quality. Furthermore, we briefly investigate intermediate state quantization during the generation phase. Directly applying 8-bit quantization to the scrambled states severely degrades generation quality because the two random scaling operations (S1 and S2 ) significantly expand the tensor’s numerical distribution range. For deployments under extremely constrained bandwidth, operators can adopt a workaround by deliberately removing the post-Hadamard scaling matrix (S2 ). Under this relaxed privacy configuration, 8-bit quantization incurs only marginal utility penalties (average drops of ∼0.26% in accuracy and ∼0.23% in ROUGE-L, as detailed in Appendix E), 12

eral avenues for future research. Currently, our decoding phase processes the generation of a single token sequentially. Future iterations could incorporate advanced mechanisms such as Speculative Decoding [28] to generate multiple tokens per cross-node communication round, thereby significantly boosting system throughput. Additionally, while our current defense against malicious prompts relies on reactive ciphertext auditing logs, future designs could integrate automated detection and mitigation mechanisms to proactively block adversarial queries prior to execution.

9

magnitude latency reduction and sustains practical, humanreading throughput with near-zero model utility degradation. Ultimately, this work eliminates critical infrastructure barriers, paving the way for secure, cross-institutional knowledge synergy in highly regulated and privacy-sensitive domains.

References [1] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gptoss-20b model card. arXiv preprint arXiv:2508.10925, 2025.

Related Work

Retrieval-Augmented Generation. RAG was initially proposed to mitigate LLM hallucinations and bypass costly retraining by fetching relevant, non-parametric knowledge from external data stores prior to generation [29]. Existing RAG architectures can be fundamentally categorized into three paradigms based on their augmentation methodologies [57]. Query-based RAG directly concatenates retrieved plaintext information with user inputs at the prompt level (e.g., REALM [19], SELF-RAG [2]). Latent representationbased RAG deeply fuses retrieved contexts into the generative model’s hidden states via cross-attention mechanisms (e.g., FiD [22], RETRO [4]). Finally, Logit-based RAG interpolates retrieval probabilities into the generator’s step-wise decoding distribution during inference (e.g., kNN-LM [25], TRIME [62]). Privacy-Preserving LLM Inference. Recent work has actively explored privacy-preserving LLM inference to safeguard data in collaborative deployments. Cryptography-based methods [12, 20] deliver rigorous mathematical security by executing neural network operations over encrypted data or secret shares, primarily leveraging Fully Homomorphic Encryption (FHE) and Secure Multi-Party Computation (SMPC). Alternatively, TEE-based methods [33] leverage hardware enclaves to protect critical data and sensitive intermediate states. Furthermore, statistical perturbation mechanisms [13, 54] mitigate feature inversion risks while avoiding extensive cryptographic computational overhead by applying Differential Privacy (DP) principles to inject controlled noise into intermediate LLM activations.

10

[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023. [3] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. [4] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206– 2240. PMLR, 2022. [5] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5), 2024. [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.

Conclusion

[7] Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information security, pages 409–437. Springer, 2017.

In this paper, we present a privacy-preserving federated RAG framework designed to overcome the prevalent data silo problem and the prohibitive latency constraints of existing cryptographic inference techniques. By leveraging a numerically stable feature scrambling mechanism alongside dynamic role delegation, our system facilitates secure, cross-node attention computation without exposing plaintext contexts. Extensive evaluations confirm that our approach delivers orders of

[8] Donatello Conte, Pasquale Foggia, Carlo Sansone, and Mario Vento. Thirty years of graph matching in pattern recognition. International journal of pattern recognition and artificial intelligence, 18(03):265–298, 2004. 13

[9] Daniel Cullina and Negar Kiyavash. Improved achievability and converse bounds for erdos-rényi graph matching. ACM SIGMETRICS performance evaluation review, 44(1):63–72, 2016.

[18] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [19] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR, 2020.

[10] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344– 16359, 2022.

[20] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. Advances in neural information processing systems, 35:15718–15731, 2022.

[11] Tian Dong, Yan Meng, Shaofeng Li, Guoxing Chen, Zhen Liu, and Haojin Zhu. Depth gives a false sense of privacy:{LLM} internal states inversion. In 34th USENIX Security Symposium (USENIX Security 25), pages 1629–1648, 2025.

[21] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411–430, 2000.

[12] Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, Wen-Guang Chen, et al. Puma: Secure inference of llama-7b in five minutes. Security and Safety, 4:2025014, 2025.

[22] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 874–880, 2021.

[13] Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Finetuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665–2679, 2023.

[23] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704– 2713, 2018.

[14] Ege Erdoğan, Alptekin Küpçü, and A Ercüment Çiçek. Unsplit: Data-oblivious model inversion, model stealing, and label inference attacks against split learning. In Proceedings of the 21st Workshop on Privacy in the Electronic Society, pages 115–124, 2022.

[24] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billionscale similarity search with gpus. IEEE transactions on big data, 7(3):535–547, 2019.

[15] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 55–65, 2019.

[25] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019. [26] Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, 34:4961–4973, 2021.

[16] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6894– 6910, 2021.

[27] Woosuk Kwon. vLLM: An Efficient Inference Engine for Large Language Models. PhD thesis, UC Berkeley, 2025.

[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

[28] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023. 14

[29] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020.

tions on Knowledge and Data Engineering, 36(7):3580– 3599, 2024. [39] Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy-preserving, accurate and efficient inference for transformers. In 2024 IEEE Symposium on Security and Privacy (SP), pages 4753–4771. IEEE, 2024.

[30] Senyao Li, Haozhao Wang, Wenchao Xu, Rui Zhang, Song Guo, Jingling Yuan, Xian Zhong, Tianwei Zhang, and Ruixuan Li. Collaborative inference and learning between edge slms and cloud llms: A survey of algorithms, execution, and open challenges. arXiv preprint arXiv:2507.16731, 2025.

[40] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392, 2016.

[31] Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026.

[41] Mohamed Sabt, Mohammed Achemlal, and Abdelmadjid Bouabdallah. Trusted execution environment: What it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/Ispa, volume 1, pages 57–64. IEEE, 2015.

[32] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023.

[42] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.

[33] Zhifan Luo, Shuo Shao, Su Zhang, Lijing Zhou, Yuke Hu, Chenxu Zhao, Zhihao Liu, and Zhan Qin. Shadow in the cache: Unveiling and mitigating privacy risks of kv-cache in llm inference. arXiv preprint arXiv:2508.09442, 2025.

[43] Anders Søgaard, Sebastian Ruder, and Ivan Vulić. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778–788, 2018.

[34] Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 5783–5797, 2020.

[44] Rahul Krishna Thomas, Louai Zahran, Erica Choi, Akilesh Potti, Micah Goldblum, and Arka Pal. Hidden no more: Attacking and defending private third-party llm inference. In Forty-second International Conference on Machine Learning, 2025.

[35] John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embeddings reveal (almost) as much as text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12448–12460, 2023.

[45] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539– 554, 2022.

[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[37] Pascal Paillier. Public-key cryptosystems based on composite degree residuosity classes. In International conference on the theory and applications of cryptographic techniques, pages 223–238. Springer, 1999.

[47] Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. Milvus: A purposebuilt vector data management system. In Proceedings of the 2021 international conference on management of data, pages 2614–2627, 2021.

[38] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transac15

[48] Zhichao Xu, Fengran Mo, Zhiqi Huang, Crystina Zhang, Puxuan Yu, Bei Wang, Jimmy Lin, and Vivek Srikumar. A survey of model architectures in information retrieval. arXiv preprint arXiv:2502.14822, 2025.

[57] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. Data Science and Engineering, pages 1–29, 2026.

[49] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

[58] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2):1– 124, 2023.

[50] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024.

[59] Fei Zheng, Chaochao Chen, Zhongxuan Han, and Xiaolin Zheng. Permllm: Private inference of large language models within 3 seconds under wan. arXiv preprint arXiv:2405.18744, 2024.

[51] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018.

[60] Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. A review on edge large language models: Design, execution, and applications. ACM Computing Surveys, 57(8):1–35, 2025. [61] Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905–5921, 2021.

[52] Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. Pretrained transformers for text ranking: Bert and beyond. In Proceedings of the 14th ACM International Conference on web search and data mining, pages 1154–1156, 2021. [53] Mu Yuan, Lan Zhang, Liekang Zeng, Siyang Jiang, Bufang Yang, Di Duan, and Guoliang Xing. Scx: Stateless kv-cache encoding for cloud-scale confidential transformer serving. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 39–54, 2025.

[62] Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5657–5673, 2022.

[54] Ziqian Zeng, Jianwei Wang, Junyao Yang, Zhengdong Lu, Haoran Li, Huiping Zhuang, and Cen Chen. Privacyrestore: Privacy-preserving inference in large language models via privacy removal and restoration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10821–10855, 2025. [55] Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. mgte: Generalized longcontext text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1393–1412, 2024. [56] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, 51(4):1373–1418, 2025. 16

A

Numerical Stability Evaluation

C

Overall Performance Details

Detailed performance data for the evaluations conducted in § 7.3 are presented in Table 5. We report the following evaluation metrics: Time to First Token (TTFT, in seconds), Decode Throughput (Dec. TPS, in tokens/s), Network Traffic (Traff., in MiB), and Communication Rounds . The optimal value for each metric is highlighted in bold. As demonstrated, our method consistently outperforms the baselines across all configurations and evaluation metrics.

We evaluate the numerical stability of using a random dense matrix for feature scrambling. Table 4 presents the relative error between the scrambled attention outputs and the original plaintext results. We observe that scrambling with a random dense matrix introduces a relative error of approximately 12%. In contrast, our proposed structured scrambling method (S1 P1 HP2 S2 ) significantly reduces this error to roughly 1.5%, maintaining consistent stability across varying sequence lengths.

D

Benchmark Accuracy Details

Table 4: Relative attention-output error. Method

Seq=128

Seq=512

Seq=2048

Dense Random Matrix S1 P1 HP2 S2

12.61% 1.49%

11.68% 1.52%

12.62% 1.63%

B

Detailed results for the benchmark evaluations discussed in § 7.4 are provided in Table 6. For the HotpotQA, MuSiQue, and MS MARCO datasets, we default to filtering the reference contexts using the supporting evidence (ground truth) provided by the datasets. This filtering mechanism simulates a practical RAG pipeline where a retriever isolates relevant passages from irrelevant distractors. For comprehensive comparison, we also evaluate the models by feeding all provided contexts (including distractors); these results are denoted with an unfiltered suffix. Regarding our accuracy metric, a response is considered correct if it contains the ground truth string as a substring. Consequently, the reported accuracy may occasionally exceed the ROUGE-L score. Note that QMSum is a meeting summarization dataset; since strict substring matching yields an accuracy of 0 across all models, we report the F1 score instead. Additionally, we conducted identical evaluations using the closed-source GPT-5.4 model, with results summarized in Table 7. It is important to note that model performance on these benchmarks is highly sensitive to prompt design and text preprocessing. Therefore, the accuracy figures reported herein are intended solely for relative comparison within the scope of our experimental setup.

Dataset Descriptions

For accuracy evaluation, we select several prominent questionanswering (QA) and summarization datasets equipped with reference contexts. These benchmarks cover a diverse range of domains, context lengths, and reasoning complexities: SQuAD [40]. SQuAD is a large-scale reading comprehension dataset comprising over 100,000 Wikipedia-based questions, where each answer is a continuous text span extracted directly from the provided passage. MuSiQue [45]. MuSiQue is a challenging multihop QA dataset constructed by composing single-hop questions. It minimizes dataset biases to ensure models perform genuine reasoning across multiple contexts rather than relying on shortcuts. QMSum [61]. QMSum is a query-based meeting summarization benchmark containing annotated transcripts from various domains, designed to evaluate the extraction and summarization of relevant context from long-form conversations. HotpotQA [51]. HotpotQA is a diverse, explainable multihop QA dataset that requires models to reason across multiple Wikipedia documents and synthesize answers while providing sentence-level supporting facts. MS MARCO [3]. MS MARCO is a large-scale reading comprehension dataset built from anonymized Bing search queries, featuring real-world questions paired with humangenerated answers derived from multiple web documents. AmbigQA [34]. AmbigQA focuses on ambiguous opendomain questions that yield multiple valid interpretations, evaluating a system’s ability to identify ambiguity and generate comprehensive answers covering various plausible contexts.

E

Evaluation of Quantization Effects

Table 8 illustrates the utility degradation resulting from 8-bit quantization of LLM intermediate states, as discussed in § 7.5. For these experiments, we evaluated only the first 128 samples from each dataset. As noted in the main text, naively applying quantization introduces significant performance penalties. However, deliberately removing the outermost scaling matrix S2 effectively mitigates these quantization-induced losses. Furthermore, our evaluation currently employs a rudimentary affine min-max quantization scheme; integrating more advanced quantization techniques could potentially yield even better accuracy-efficiency trade-offs in future implementations.

17

Table 5: Performance evaluation detailed results. Length

protected_layers = 2

Method

protected_layers = full

TTFT (s) ↓ Dec. TPS ↑ Traff. (MiB) ↓ Comm. Rounds ↓ TTFT (s) ↓ Dec. TPS ↑ Traff. (MiB) ↓ Comm. Rounds ↓ Qwen/Qwen3-4B-Instruct-2507 Ours PermLLM SCX

0.25 0.85 5.20

36.15 16.38 25.66

156.47 452.59 1418.02

75 683 628

3.65 12.97 5.20

9.91 1.42 4.86

328.01 5135.92 1677.80

1287 12195 3878

Ours Medium PermLLM SCX

0.37 6.02 14.44

36.51 15.32 25.84

619.11 3650.17 5494.15

139 1323 820

5.39 92.52 15.20

9.96 1.28 4.84

1210.85 49936.56 6345.42

2439 23715 7334

Ours PermLLM SCX

1.05 77.53 52.82

36.79 11.75 26.22

2462.82 42196.11 21620.40

267 2603 1204

13.15 1341.59 55.41

10.02 0.86 4.83

4639.35 640416.59 24644.67

4743 46755 14246

Short

Long

meta-llama/Llama-3.1-8B-Instruct Ours PermLLM SCX

0.25 0.86 4.86

47.04 18.18 30.97

141.01 439.03 1526.88

75 683 580

3.23 11.93 6.22

11.70 1.62 5.13

301.17 4721.73 1791.25

1147 10851 3458

Ours Medium PermLLM SCX

0.42 5.93 17.44

46.99 16.94 28.67

555.65 3592.90 5891.63

139 1323 772

4.83 89.23 15.09

11.79 1.46 5.48

1107.61 45815.71 6752.25

2171 21091 6530

Long

Ours PermLLM SCX

1.15 77.51 58.06

44.49 12.74 31.44

2207.37 41963.67 23159.13

267 2603 1156

12.00 1243.09 60.15

11.85 0.96 5.50

4238.54 587393.74 26202.25

4219 41571 12674

Short

Ours PermLLM SCX

0.20 0.93 9.09

33.77 19.68 24.45

173.14 466.64 2117.09

75 558 676

1.64 14.73 9.06

9.12 1.29 4.07

357.67 5620.91 2406.17

1427 13539 4298

Ours Medium PermLLM SCX

0.43 6.07 19.91

33.05 14.62 24.56

683.78 3721.02 8172.84

139 1323 868

3.51 106.02 21.65

9.23 1.52 4.18

1316.92 53448.83 9083.67

2707 20139 8138

Ours PermLLM SCX

1.29 76.65 79.65

31.59 11.23 24.75

2719.49 42475.80 32134.34

267 2603 1252

12.51 1464.34 80.99

9.17 3.39 4.25

5042.98 673616.17 35278.67

5267 12839 15818

Short

mistralai/Ministral-3-14B-Instruct-2512-BF16

Long

openai/gpt-oss-20b Ours PermLLM SCX

0.22 0.98 3.13

20.67 13.25 17.79

46.00 379.52 941.40

75 683 484

1.39 9.68 3.52

10.19 1.79 5.94

220.66 4224.67 1181.29

867 8163 2618

Ours Medium PermLLM SCX

0.44 6.21 9.29

20.66 12.56 17.90

160.84 3295.09 3643.90

139 1323 676

2.45 73.47 10.13

10.28 1.62 5.98

804.73 38419.13 4454.75

1635 15843 4922

Ours PermLLM SCX

1.35 77.91 36.42

21.88 10.24 18.06

611.33 40677.89 14331.90

140 2603 1060

7.72 966.51 39.13

13.54 1.16 6.01

3037.73 483843.42 17274.69

1647 31203 9530

Short

Long

18

Table 6: Detailed Benchmark Performance under Distributed Attention and Feature Scrambling. Model

SQUAD

HOTPOTQA

MARCO

Acc (%) ↑ Rouge-L ↑ PPL ↓

Acc (%) ↑ Rouge-L ↑ PPL ↓

Acc (%) ↑ Rouge-L ↑ PPL ↓

Method

Llama 3.1 8B

Baseline +Distributed +Scrambler

88.33 88.29 88.30

89.75 89.74 89.72

1.85 1.85 1.85

66.58 66.47 66.50

75.11 75.01 75.00

1.92 1.92 1.92

24.11 24.13 24.18

48.14 48.11 48.16

2.05 2.05 2.05

Qwen 3 4B

Baseline +Distributed +Scrambler

89.21 89.23 89.18

88.22 88.23 88.24

6.43 6.43 6.43

69.63 69.68 69.67

79.30 79.38 79.32

5.94 5.97 5.95

24.25 24.11 24.23

46.34 46.25 46.34

7.01 7.02 7.05

Baseline Ministral 3 +Distributed 14B +Scrambler

86.44 86.40 86.41

89.76 89.75 89.74

1.60 1.60 1.60

68.41 68.43 68.47

78.51 78.52 78.50

1.72 1.72 1.72

18.03 18.09 18.02

42.44 42.46 42.35

2.35 2.35 2.35

Qwen 2.5 32B

Baseline +Distributed +Scrambler

93.72 93.75 93.80

90.94 91.00 91.02

2.35 2.35 2.35

72.99 72.82 72.83

80.14 80.06 80.03

2.69 2.69 2.69

22.25 22.00 22.00

46.75 46.56 46.49

5.50 5.51 5.50

Model

Method

AMBIGQA

MUSIQUE

QMSUM

Acc (%) ↑ Rouge-L ↑ PPL ↓

Acc (%) ↑ Rouge-L ↑ PPL ↓

F1 (%) ↑

Rouge-L ↑ PPL ↓

Llama 3.1 8B

Baseline +Distributed +Scrambler

56.79 56.59 56.44

63.78 63.64 63.53

3.92 3.92 3.92

59.33 59.21 59.21

62.63 62.47 62.64

1.98 1.98 1.97

30.02 29.97 29.87

23.08 23.07 22.92

11.61 11.61 11.60

Qwen 3 4B

Baseline +Distributed +Scrambler

58.04 58.04 57.79

61.09 61.05 61.02

46.02 45.93 45.40

57.10 57.34 57.10

62.13 62.30 62.03

7.12 7.13 7.16

25.75 25.70 25.65

19.66 19.81 19.60

16.65 16.65 16.69

Baseline Ministral 3 +Distributed 14B +Scrambler

59.24 59.29 59.29

64.26 64.22 64.37

4.09 4.10 4.11

65.70 65.58 65.87

66.80 66.60 66.93

1.91 1.91 1.91

22.59 22.52 22.54

17.73 17.65 17.65

16.49 16.46 16.46

Qwen 2.5 32B

Baseline +Distributed +Scrambler

25.67 25.62 25.62

66.97 66.90 66.72

14.11 14.11 14.13

73.60 73.15 72.98

75.58 75.27 75.06

2.31 2.31 2.31

26.81 28.01 28.09

20.50 20.98 21.09

19.80 19.82 19.83

Model

Method

HOTPOTQA (Unfiltered)

MUSIQUE (Unfiltered)

MARCO (Unfiltered)

Acc (%) ↑ Rouge-L ↑ PPL ↓

Acc (%) ↑ Rouge-L ↑ PPL ↓

Acc (%) ↑ Rouge-L ↑ PPL ↓

Llama 3.1 8B

Baseline +Distributed +Scrambler

56.87 56.96 56.87

66.01 66.02 65.92

2.14 2.15 2.15

35.79 35.79 35.79

40.11 40.15 40.00

2.93 2.93 2.93

18.66 18.77 18.66

34.04 34.04 34.03

2.22 2.22 2.22

Qwen 3 4B

Baseline +Distributed +Scrambler

61.01 61.08 60.93

69.59 69.61 69.56

11.62 11.63 11.83

30.41 30.49 30.33

35.57 35.70 35.60

15.72 15.73 15.91

17.58 17.48 17.52

29.43 29.42 29.39

12.85 12.84 12.95

Baseline Ministral 3 +Distributed 14B +Scrambler

61.63 61.74 61.66

71.41 71.51 71.39

1.89 1.90 1.89

47.62 47.29 47.37

49.86 49.60 49.78

2.48 2.48 2.48

14.24 14.34 14.25

27.31 27.34 27.30

2.99 2.99 2.99

Baseline +Distributed +Scrambler

66.08 65.56 65.64

73.95 73.44 73.48

2.95 2.96 2.96

47.54 46.83 47.08

52.80 52.23 52.37

3.54 3.55 3.55

17.08 16.85 16.92

32.64 32.68 32.69

7.19 7.19 7.20

Qwen 2.5 32B

19

Table 7: Benchmark Performance of GPT-5.4. Dataset SQUAD HOTPOTQA MARCO AMBIGQA MUSIQUE QMSUM HOTPOTQA (Unfiltered) MUSIQUE (Unfiltered) MARCO (Unfiltered)

Acc (%) ↑ F1 (%) ↑ Rouge-L ↑ 94.24 76.41 21.28 65.93 82.25 0.00 75.21 78.03 16.94

91.98 84.19 45.32 69.50 83.39 24.38 83.09 79.59 29.34

91.79 84.66 46.86 70.23 83.49 17.89 83.54 79.97 30.31

20

Table 8: Benchmark accuracy degradation introduced by different scrambling methods under 8-bit quantization. Model

SQUAD

Method ∆Acc

HOTPOTQA

∆Rouge-L

∆PPL

∆Acc

MARCO

∆Rouge-L ∆PPL

∆Acc

∆Rouge-L

∆PPL

Llama 3.1 8B

S1 & S2 -29.69 S1 only +0.00

-31.12 -0.39

+0.49 +0.05

-39.84 +2.34

-37.06 +2.14

+3.09 +0.05

-12.50 -1.56

-28.12 -0.02

+0.84 +0.01

Qwen 3 4B

S1 & S2 -20.31 S1 only +0.78

-25.49 -1.95

-0.51 -0.47

-27.34 -0.78

-36.31 -1.01

+2.75 -0.72

-8.59 +0.00

-22.27 +0.97

-2.12 -0.12

Ministral 3 S1 & S2 14B S1 only

-3.13 +0.00

-1.03 +0.00

+0.03 +0.00

-2.34 +0.78

-3.33 +0.78

+0.09 +0.00

-1.56 -1.56

-2.52 -2.22

+0.08 -0.01

Qwen 2.5 32B

S1 & S2 S1 only

-4.69 +0.00

-3.31 +0.65

+0.30 -0.01

-3.13 +0.00

-0.82 -0.13

+1.03 -0.04

+0.00 +0.78

-5.26 +0.09

+0.04 -0.07

Model

Method

AMBIGQA ∆Acc

MUSIQUE

∆Rouge-L

∆PPL

∆Acc

QMSUM

∆Rouge-L ∆PPL

∆F1

∆Rouge-L

∆PPL

Llama 3.1 8B

S1 & S2 -39.06 S1 only +0.78

-57.97 +1.35

+83.23 +0.06

-43.75 +0.78

-38.78 -0.36

+2.55 +0.03

-21.53 -0.64

-11.94 -0.88

+71.53 +0.01

Qwen 3 4B

S1 & S2 -31.25 S1 only -1.56

-49.15 -1.20

+50.30 -7.22

-35.94 -5.47

-34.60 -3.45

+1.75 -0.48

-13.59 -0.27

-5.99 -0.05

+37.00 -0.25

-7.03 -0.78

-9.58 -1.04

-0.02 +0.01

-10.16 +0.00

-6.90 +0.52

+0.25 +0.01

-1.67 -0.05

-0.94 +0.13

+0.85 +0.06

S1 & S2 -10.94 S1 only +0.78

-16.86 +0.33

+10.81 -0.53

-14.06 +0.00

-9.56 +0.39

+1.79 -0.04

-12.06 +0.22

-5.82 +0.13

+35.23 -0.29

Ministral 3 S1 & S2 14B S1 only Qwen 2.5 32B

21