arXiv:2606.17421v1 [cs.CR] 16 Jun 2026
Bifrost: Hybrid TEE–FHE Inference for Privacy-Preserving Transformer and LLM Serving Chenghao Chen
Kailun Qin
Xiaolin Zhang
[email protected] Shanghai Jiao Tong University Shanghai, China
[email protected] Shanghai Jiao Tong University Shanghai, China
[email protected] Shanghai Jiao Tong University Shanghai, China
Chi Zhang
Dawu Gu
[email protected] Shanghai Jiao Tong University Shanghai, China
[email protected] Shanghai Jiao Tong University Shanghai, China
Abstract
the remaining boundary: fully cached GPT-2 decode is GEMM-dominated, while Qwen3 exposes a 34.4% fallback weight-encoding penalty when encoded weights no longer fully fit on the accelerator. The resulting lesson is selective encrypted execution: use FHE only where ciphertext-only accelerator delegation is required, and keep non-linear, refresh, and prompt-side work inside the CPU TEE.
Cloud-hosted transformer and large language model (LLM) inference creates an immediate confidentiality problem: user prompts may contain source code, business plans, personal data, or regulated documents, yet remote serving exposes intermediate state to the cloud software stack and accelerator runtime. Fully homomorphic encryption (FHE) keeps accelerator-side execution ciphertext-only, but end-to-end LLM inference remains expensive because linear layers are interleaved with non-linear, reduction-heavy, cache-state, and refresh-sensitive operators. CPU trusted execution environments (TEEs) can execute those operators natively, but trusting a CPU TEE does not by itself explain how an untrusted accelerator should participate. We present Bifrost, a hybrid TEE–FHE serving architecture for this trust-policy point: secrets are provisioned only to an attested CPU TEE, while the accelerator, device memory, driver/runtime stack, and host software remain outside the trusted computing base. FHE is therefore the secure delegation mechanism, not the sole trust anchor. Bifrost keeps regular projection and feed-forward linear layers on acceleratorbacked CKKS, while non-linear operators, attention-side control logic, KV-state transitions, and decrypt-then-encrypt (DtE) refresh execute inside the CPU TEE. Bifrost+ further applies a prefill/decode (PD) split to the same boundary. Instead of sending every prompt token through the encrypted accelerator loop, Bifrost+ builds prompt-side KV state inside the CPU TEE and hands only decode-side state to the hybrid ciphertext path. This does not make FHE kernels faster; it avoids prompt-side encrypted work when the CPU TEE can execute prefill directly. In an estimator-style comparison matching Euston’s methodology, Bifrost reduces projected latency by 9.25× on GPT-2 (1.5B) and 9.91× on LLaMA 3 (8B). In direct CKKS/FHE deployments, Bifrost+ reduces TTFT by 14.6–45.8× on GPT-2 (124M) and 15.3–53.4× on Qwen3 (0.6B) for prompt lengths 16 and 64. Cost breakdowns show
CCS Concepts: • Security and privacy → Systems security; • Computer systems organization → Architectures. Keywords: fully homomorphic encryption, trusted execution environment, large language model, privacy-preserving inference, heterogeneous secure execution
1
Introduction
Large language models (LLMs) are increasingly deployed through Model-as-a-Service (MaaS): users send prompts to a cloud provider, the provider hosts the model, and generated outputs are returned remotely. This deployment model is convenient and cost-effective, but it creates a direct privacy problem. Prompts may contain source code, business plans, personal information, legal text, or regulated documents, and remote inference exposes intermediate state to the cloud software stack, accelerator runtime, and service operator. Privacy-enhancing technologies offer two natural starting points. Fully Homomorphic Encryption (FHE) protects data cryptographically by enabling computation on ciphertexts [6, 15]. Trusted Execution Environments (TEEs), such as Intel TDX and AMD SEV, protect data architecturally by confining plaintext execution to hardware-isolated domains [1, 2]. For LLM serving, however, neither mechanism is sufficient by itself. FHE provides a narrow accelerator-side privacy boundary, but non-linear operators, depth growth, ciphertext refresh, and serving state make end-to-end inference expensive. CPU TEEs execute those operators natively, but CPU TEE-only execution becomes the trusted-CPU latency floor rather than a way to use untrusted accelerators. 1
TEE (TD) (Trusted Component)
This paper targets the remaining deployment point. A tenant provisions secrets to an attested CPU TEE, but does not place accelerator memory, firmware, drivers, or runtime in the trusted computing base. FHE is therefore a secure accelerator-delegation mechanism rather than the trust root. The question is not whether FHE beats plaintext CPU TEE execution; it does not in our prototype. The question is which operators and serving phases should still pay FHE cost when accelerator execution must remain ciphertext-only. Two properties of modern LLM serving make this question more than encrypted GEMM scheduling. First, model architectures evolve: decoder blocks combine normalization, RoPE, gating, grouped-query attention, and cache logic whose semantics vary across model families and are awkward to re-engineer as fixed homomorphic circuits. Second, serving is phase-structured: prefill determines time-to-firsttoken (TTFT), while decode repeatedly touches persistent KV state and small per-token control paths. A confidential serving system therefore needs both an operator-level placement policy and a phase-level serving policy. Recent CKKS systems, compilers, and accelerators have improved encrypted linear algebra [8, 9, 12, 14, 16, 31]; hybrid systems such as BLB, TEEFHE, and HT2ML show that selected computation can leave a pure-cryptographic path when the protection model allows it [26–28]. What remains missing, to our knowledge, is a TEE+FHE Transformer/LLM serving system that couples a trusted CPU to acceleratorbacked FHE, supports autoregressive cache semantics, respects prefill/decode asymmetry, and evaluates end-to-end behavior. The key observation behind Bifrost is that the trust boundary and the Transformer layer boundary should not coincide. Regular linear projections are stable, data-parallel, and FHE-friendly, so they are the right candidates for the ciphertext accelerator path. Non-linear operators, control logic, refresh, and serving-state transitions are precision-sensitive or control-heavy, so they should execute inside the CPU TEE when the trust policy allows it. This split adapts to changing model architectures while keeping the accelerator view ciphertext-only. Bifrost instantiates this observation as a hybrid TEE–FHE architecture. Large linear layers run on an untrusted accelerator under CKKS, while normalization, RoPE, Softmaxstyle reductions, attention-side control, gating, and ciphertext refresh execute inside the CPU TEE through decryptcompute-encrypt or decrypt-then-encrypt (DtE) windows. The prototype is built as an executable serving runtime, not a collection of isolated kernels: it includes request routing, policy control, encoded-weight caching, and KV-cache machinery for autoregressive inference. Figure 1 summarizes the end-to-end path. Bifrost alone still routes prompt prefill and steady-state decode through the same hybrid path. That is a poor serving
CPU Host (Handler)
Prefill Calc.
FHE Key Mgmt.
for Bifrost+
Scheduler
Encrypted Prompt
TEE Decrypt
FHE Encrypt
FHE Evaluation Ops
Non-linear Ops
Models
Encrypted Result
Frontend OpenAIcompatible API
GeLU
LN
FHE Accelerator(s) (FHE Linear Evaluation)
Softmax
FHE Linear Layers
DtE Noise Refresh
FHE Decrypt
FHE Encrypt
Conv
MatMul
FHE Output Aggregation Final TEE Encrypt
Figure 1. End-to-end hybrid inference architecture. The CPU TEE is the trusted control point; the accelerator is untrusted and evaluates FHE linear layers over ciphertexts. Trusted GPU execution is not assumed.
policy because every prompt token pays the encrypted accelerator cost before the first token can be emitted. Bifrost+ applies prefill/decode (PD) disaggregation [22, 32] to the TEE–FHE boundary. It completes prompt prefill and promptside KV construction inside the CPU TEE, then hands only decode-side state to the same hybrid ciphertext path. This is not a new cryptographic primitive and does not accelerate the FHE kernels; it removes prompt-side encrypted work. At the implementation level, our prototype builds on nano-vLLM, a lightweight derivative of the vLLM serving stack [17, 30]. This lets the evaluation exercise an end-to-end serving substrate, including request routing, policy selection, and KV state, rather than relying only on operator microbenchmarks or replay scripts. Contributions. We make four contributions. • CPU-TEE-rooted ciphertext-only accelerator delegation. We define and instantiate a deployment point where the CPU TEE is the only trusted execution boundary and the accelerator acts only as a ciphertext evaluator. • Operator-affinity TEE–FHE execution. Bifrost keeps regular linear layers on accelerator-backed FHE and moves non-linear, reduction-heavy, control-heavy, and refresh-sensitive operators into the CPU TEE. • Phase-aware PD split over a secure serving boundary. Bifrost+ keeps prompt prefill and prompt-side KV construction inside the TEE, then hands only decode-side state to the hybrid ciphertext path. • Executable serving substrate and deploymentboundary evidence. The nano-vLLM-derived prototype integrates policy routing, encoded-weight caching, and KV backends, exposing encoded-weight residency and accelerator memory pressure—not arithmetic alone—as the dominant scaling boundary for larger deployments. 2
2
Background and Motivation
2.1
Fully Homomorphic Encryption
confidential LLM inference across CPU and GPU TEEs and surveys of heterogeneous confidential computing quantify the performance, cost, and trust-model tradeoffs of bringing accelerators into the TCB [7, 25]. In both cases, TEEs are much better at executing branch-heavy or reduction-heavy operators than FHE, but they do not remove the problem of serving large models inside a limited trusted footprint.
Our prototype targets the CKKS approximate homomorphic encryption scheme [6]. CKKS operates over the ring 𝑅𝑄 = Z𝑄 [𝑋 ]/(𝑋 𝑁 + 1) for power-of-two ring dimension 𝑁 , and packs 𝑁 /2 complex slots into each ciphertext. This packing makes CKKS well suited to large linear algebra because one ciphertext can hold many vector or matrix fragments at once. The price is that scale management, key switching, and rotations are explicit costs rather than transparent implementation details. Three CKKS concepts are central for this paper. First, ring dimension controls both security and throughput: larger 𝑁 exposes more slots, but also increases ciphertext size and the cost of NTT-heavy kernels. Second, levels and rescaling govern multiplicative depth: after a ciphertext–ciphertext multiplication, CKKS rescales to keep the numerical scale bounded while consuming one level from the modulus chain. Third, bootstrapping refreshes a ciphertext once the remaining level budget is exhausted, but that refresh is expensive enough to be a first-order systems concern. This is why operator choice matters. Let 𝐷 lin denote the depth contribution from linear layers and 𝐷 NL the contribution from non-linear approximations. In a pure-FHE transformer block, the effective encrypted depth is 𝐷 FHE ≈ 𝐷 lin + 𝐷 NL .
2.3
This paper targets decoder-only autoregressive serving. Each block applies normalization, QKV projection, positional encoding, attention against the KV cache, output projection, another normalization, and feed-forward layers before producing logits for the next token. The same block structure is exercised during both prompt processing and generation, but the serving phases behave differently. Prefill processes all prompt tokens, is matrix-heavy, and benefits from batch-style linear algebra. Decode emits one token at a time, touches the KV cache repeatedly, and performs smaller per-step computation. The KV cache is therefore not an optional side detail: it is central to latency and to the placement of persistent state across the TEE–accelerator boundary. Any secure serving system has to explain not only where arithmetic runs, but also how cache state is materialized, updated, and handed across phases. 2.4
(1)
hybrid
≈ 𝐷 lin,
(2)
which directly reduces refresh pressure and allows the FHE path to focus on accelerator-friendly linear algebra. 2.2
Design Space and Motivation
The relevant design space has four endpoints: CPU TEE-only, CPU+GPU TEE, pure FHE, and CPU TEE+FHE accelerator delegation. CPU+GPU TEE designs form a complementary trusted-accelerator endpoint [7, 25]; Bifrost instead keeps the accelerator outside the TCB and uses FHE for ciphertextonly delegation. The systems opportunity exploited by this paper is the complementarity at the last point: large linear projections map naturally to accelerator-backed CKKS execution, while normalization, Softmax-style reductions, routing, and refresh are precisely the stages where FHE becomes depth-intensive or numerically fragile and where a TEE can execute the operator exactly. Autoregressive serving introduces a second asymmetry on top of the operator split. Prompt prefill is matrix-heavy and batched, while decode performs smaller per-token work with frequent KV accesses. In plaintext serving, PD disaggregation separates these phases for throughput and latency reasons. In our hybrid setting, a unified path still forces every prompt token through the expensive encrypted loop, so TTFT grows with prompt length even if the decode path is already well designed. This is why the paper applies PD disaggregation to the TEE–FHE boundary as a PD split: a second-layer trustdomain partitioning decision on top of the operator split, rather than a replacement for operator-affinity execution. Finally, memory residency shapes how far the design scales. We use parameterized model names throughout: the
For modern LLMs, 𝐷 NL is often the more fragile term because normalization, Softmax-style reductions, and gating paths must be approximated or otherwise lowered into expensive arithmetic circuits. Once those operators move into a trusted plaintext domain, the encrypted depth becomes 𝐷 FHE
Autoregressive Transformer Inference
Trusted Execution Environments
TEEs protect computation architecturally rather than cryptographically. In Intel TDX, guest-private memory is encrypted, access control is enforced by the CPU memory controller, and attestation binds the running software stack to platform measurements before secrets are provisioned [2]. For our purposes, TDX offers three useful properties: plaintext execution of complex operators, an attestation root for key release, and a deployment model that does not require trusting the host kernel or hypervisor. Those benefits come with systems constraints. TEE performance is still shaped by trusted-memory capacity, page-management overhead, and the limited availability of accelerator-class TEEs for general-purpose inference. AMD SEV offers a similar VM-level protection story with different isolation and attestation details [1]. Recent work on 3
common GPT-2 Small, Medium, and Large checkpoints correspond to GPT-2 (124M), GPT-2 (355M), and GPT-2 (774M) in our parameter-count convention [23]. GPT-2 (124M) behaves like a clean fully cached case, GPT-2 (355M) remains close to that regime, and Qwen3 (0.6B), a dense model from the Qwen3 family [29], already pays a measurable fallback encoding penalty. Once encoded weights, persistent KV state, and temporary workspace compete for the same accelerator budget, memory becomes a first-order boundary even if the architectural split remains beneficial.
3
state needed for the next operator window, evaluates it in plaintext, and immediately re-encrypts the outputs before resuming accelerator execution. Throughout inference, the untrusted accelerator observes only ciphertexts and deterministic kernel launches. Table 1. Execution paths and measurement roles. CPU TEEonly is the trusted-CPU latency floor; Bifrost targets the stricter setting where delegated accelerator work remains ciphertext-only.
System Design
Our goal is not to introduce a new primitive, but to present a systems-level execution strategy that makes privacy-preserving transformer and LLM serving more practical. This section focuses on operator placement, execution flow, scheduler policy, and threat model; Section 4 then maps those ideas onto the prototype runtime. 3.1
Path
Prefill
Decode
Accel.
Role
CPU TEE-only Pure FHE Bifrost Bifrost+
TEE FHE hybrid TEE
TEE FHE hybrid hybrid
none ct ct ct
latency floor projected baseline direct CKKS/FHE direct CKKS/FHE + PD
Within this architecture, Table 1 separates the trustedCPU endpoint from the three encrypted or delegated paths; ct denotes a ciphertext-only accelerator view. CPU TEE-only executes both phases inside the trusted CPU and serves as the latency floor under our CPU-TEE trust assumption. Pure FHE is the fully encrypted crypto baseline. Bifrost is the base hybrid architecture, which retains the operator split but still routes prompt prefill through the same hybrid loop used by decode. Bifrost+ keeps the same hybrid decode path while moving prompt prefill into the CPU TEE. It is built directly on top of Bifrost rather than being a separate architecture.
Architecture Overview
Operator-affinity split. Our design partitions transformer inference by operator affinity, as illustrated in Figure 2: • FHE-friendly (accelerator). Large, regular linear projections, including QKV/output projection GEMMs and FFN linear layers, are evaluated homomorphically on an accelerator-backed FHE path. These operators dominate FLOPs and remain structurally stable across model variants. • TEE-friendly (CPU). Operators that are precisionfragile, depth-hungry, or control-heavy under FHE, including RoPE, Softmax, attention control, gating, normalization, and refresh, are executed inside the TEE in plaintext and their outputs are immediately re-encrypted before re-entering the FHE path. This split keeps the encrypted backbone focused on throughput-critical linear layers while making the overall system more robust to evolving LLM motifs. End-to-end execution flow. Figure 1 shows the deployment pipeline. The client encrypts the prompt and sends ciphertexts to the server. A host-side handler forwards the request into the TEE, which manages secret keys, executes TEE-friendly operators, and orchestrates accelerator-backed execution for FHE-friendly linear layers. The same TEEside runtime also maintains the instantiated KV cache and governs when ciphertext-backed state is handed to the accelerator path. Execution alternates between TEE segments and accelerator-backed FHE segments. For an accelerator segment, the TEE prepares the required activations and dispatches ciphertexts to the accelerator, where homomorphic linear kernels run without accessing plaintext. For a TEE segment, the TEE decrypts only the minimal intermediate
3.2
Operator-Affinity Execution
Our offloading strategy follows a simple systems observation: in transformer inference, the arithmetic bulk lies in large linear algebra, whereas the fragile parts under FHE are non-linear and reduction-heavy operators together with ciphertext refresh. We therefore split execution by operator affinity: the accelerator-backed FHE path specializes in dataparallel linear layers, while the CPU TEE executes operators whose FHE realizations are depth-intensive, rotation-heavy, or accuracy-fragile. Offloading non-linear operators. Since FHE natively supports additions and multiplications, non-linear operators in LLMs are typically implemented via polynomial or rational approximations, or via comparison-oriented arithmetic circuits. Even with sufficient CKKS precision, replacing 𝑓 (·) by an approximation 𝑓˜(·) changes the model’s semantics. Our design instead executes these operators inside the CPU TEE in plaintext: when reaching a non-linear operator, the encrypted activation is transferred to the TEE, decrypted with the secret key sealed in the TEE, evaluated exactly using the native operator implementation, and immediately re-encrypted before returning to the FHE pipeline. This has three practical effects: 4
2) MLP Sublayer (SwiGLU)
Linear (GEMM)
Residual Add
Linear (GEMM)
Residual Add
Linear (GEMM)
DtE Support
(B) Trusted CPU domain plaintext computation + scheduler
(A) Untrusted Accelerator FHE ciphertext computation
1) Attention Sublayer (Pre-norm) and MHA)
(TEE ciphertext)
Normalization
Scheduler (TEE)
FHE Encrypt
FHE Decrypt
FHE Decrypt
RoPE (Q, K)
Softmax
FHE Encrypt
Non-linear operators executed exactly in TEE; no polynomial approximation
FHE Decrypt
FHE Decrypt
SwiGLU Gate + Elementwise Mul
FHE Encrypt
FHE Encrypt
DtE Refresh (Decrypt-thenEncrypt, replaces bootstrapping)
FHE Encrypt
TEE Offload (Decrypt-Compute-Encrypt)
Noise Refresh
GPU/FHE operations on ciphertext
TEE Offload
Data transfer
TEE plaintext operations
Decrypt-then-Encrypt (DtE) Refresh
Control flow
Scheduler in TEE, controlling when to offload and when to refresh FHE ciphertext
Figure 2. Operator-level hybrid dataflow of one representative modern LLM block. Linear and accelerator-friendly segments run on an untrusted FHE accelerator ciphertext path, instantiated by a GPU in our prototype. Non-linear, controlheavy, and refresh-sensitive operators, such as normalization, RoPE, Softmax, gating, and DtE refresh, are handled inside the CPU TEE through decrypt-compute-encrypt offloads. Table 2. Operator-to-domain mapping and measured decode-time fractions for GPT-2 (124M) in the fully cached regime.
1. No approximation gap. The remaining numerical effects are those already inherent to CKKS encoding and quantization. 2. Depth stays linear-dominated. The encrypted path spends its level budget primarily on the large linear kernels that benefit from accelerator execution. 3. Architecture adaptivity. As LLM blocks evolve, new non-linear semantics can be absorbed by the TEE without redesigning approximation circuits or re-calibrating the homomorphic backbone.
Operation
Domain
Linear (QKV, O, GateUp, Down) DtE boundary crossing Norm, RoPE, Softmax/control, activation
GPU FHE TEE ↔ GPU CPU TEE
% decode 97.5% 2.3% <0.2%
a TEE-assisted DtE primitive: ct′ = Encpk (Decsk (ct)) ,
The key insight is that only the four linear projections per layer—QKV, output, gate-up, and down—are candidates for FHE offload, and these projections account for 97.5% of pertoken decode time in the fully cached regime (Table 2). All remaining operators execute on the CPU TEE at negligible cost relative to the homomorphic GEMM.
(3)
where decryption and re-encryption both occur inside the TEE. Conceptually, DtE provides the same logical effect as bootstrapping—resetting noise and restoring decryptability— but performs it using the TEE trust boundary rather than evaluating the refresh homomorphically. DtE is beneficial when
Decrypt-then-encrypt (DtE) refresh. Bootstrapping is the standard mechanism to reset ciphertext noise in CKKS, but it is expensive due to large-scale transforms and modular arithmetic. Our system replaces homomorphic refresh with
FHE 𝑇transfer + 𝑇Dec + 𝑇Enc < 𝑇refresh .
(4)
Offloading non-linear operators and DtE refresh reinforce each other: removing non-linear approximations slows depth 5
Algorithm 1: Boundary-aware scheduler
growth, and when refresh is still required, DtE makes it cheaper. Managing domain-transition overhead. CPU– accelerator transfers could in principle offset the gains from hybrid execution. Our design mitigates this in two ways. First, ciphertexts remain resident on the accelerator across consecutive linear layers whenever possible, so transitions occur only at architectural boundaries or refresh points. Second, the runtime batches adjacent TEE-resident operators into a single offload window when possible. As a result, the number of cross-domain transitions scales with operator groups rather than with every elementwise step. 3.3
Input: operator graph 𝐺, tiling configurations T, cache budget 𝐵 Output: execution plan Π Π←[] 𝑏←𝐵 foreach operator group 𝑔 in topological order of 𝐺 do if 𝑔 is TEE-friendly then append a TEE offload window for 𝑔 to Π update ciphertext level, scale, and boundary state else choose 𝜏 ∈ T minimizing projected latency under budget 𝑏 if encoded weights for 𝑔 fit in persistent accelerator cache then pin 𝑔 in accelerator-resident cache else append fallback encode / repack action for 𝑔 to Π
Boundary-Aware Scheduler and Tiling
The scheduler is the control plane of the hybrid inference pipeline. It runs inside the TEE and coordinates operator routing, ciphertext-state bookkeeping, domain transitions, and noise management. Beyond routing, its key systems responsibility is to choose an execution granularity that balances accelerator throughput against TEE-boundary costs. Pack (CKKS Slots)
One huge polynomial / large ring (NTT domain)
if predicted level budget falls below threshold then append DtE refresh to Π and reset the noise state return Π
by homomorphic aggregation. It maintains a lightweight ciphertext state, including remaining levels, scale, and noise margin, and inserts DtE refresh when the predicted budget is insufficient: ct → Encpk Decsk (ct) .
(FHE Encrypted)
Ops: NTT, mul, rot(k), add, rescale/modswitch, keyswitch, bootstrap TEE: Scheduling + Refresh
append tiled FHE dispatch for 𝑔 with tile size 𝜏
Pure FHE
TEE-FHE Hybrid
small polynomial / ring small polynomial / ring
(TEE Encrypted)
Non-linear operators follow the same pattern: the scheduler transfers only the required ciphertext state into the TEE, executes the operator exactly in plaintext, and immediately re-encrypts the output before resuming FHE execution.
small polynomial / ring
Figure 3. Boundary-aware scheduler for large homomorphic linear layers. Instead of treating tiling as a GPUonly kernel decision, the scheduler jointly chooses tile size, persistent cache versus fallback layers, and refresh or repack insertion points. The resulting plan dispatches smaller CKKS tiles to the FHE accelerator while preserving the CPU TEE as the control point for cache and boundary management.
Boundary-aware planning. In practice, the scheduler optimizes more than raw operator order. It tries to maximize contiguous accelerator-resident linear segments, batch adjacent TEE-friendly operators into a single offload window, and avoid unnecessary re-encoding of weights or transient states. This is especially important for models whose encrypted decode path only partially fits in device memory. In those cases, the scheduler must jointly choose the tiling plan, which layers remain persistently cached on the accelerator, and whether the next boundary crossing should also be used to refresh or repack ciphertext state.
Tiling for large homomorphic GEMMs. A monolithic CKKS packing strategy for large matrices often forces a large ring dimension to accommodate enough slots, which directly increases ciphertext size and the cost of NTT and key-switch-heavy kernels. Tiling replaces one large-ring evaluation with many small-ring evaluations: the scheduler cuts activations into tiles, packs them into smaller CKKS instances, and launches a sequence of tile-level kernels, as shown in Figure 3. This improves accelerator efficiency and makes per-tile noise growth easier to track, at the cost of additional TEE-boundary crossings. Scheduling, assembly, and refresh. Given a tiling plan, the scheduler dispatches tile-level GEMM tasks to the accelerator backend and receives encrypted assembled outputs
3.4
From Bifrost to Bifrost+
Where the operator split (Section 3.2) partitions which operators cross the trust boundary, the PD split partitions which serving phase crosses it. Under Bifrost, prompt tokens and decode tokens both traverse the same hybrid loop, so the system materializes encrypted intermediate states and hybrid KV updates at every prompt step before the first token can be emitted. Under Bifrost+ , prompt processing and prompt-side KV construction are completed inside the TEE, and only the 6
CPU (TEE Orchestrator)
FHE Accelerator
(a) Bifrost
Prefill (inside Hybrid)
matrixheavy batched GEMM
matrixheavy batched GEMM
prepare req & enc params
prepare
...
...
Decode (inside Hybrid loop)
matrixheavy batched GEMM
prepare
per-token compute & update
...
dominated, whereas decode performs much smaller per-token compute while repeatedly touching cached state. Under Bifrost, every prompt token still pays the TEE-to-accelerator boundary cost and the encrypted linear path, so TTFT grows roughly with prompt length. Under Bifrost+ , TTFT is closer to “TEE prefill + one hybrid decode step,” while steady-state decode remains governed by the same encrypted backbone. Concretely, the offline phase pre-encodes model weights into GPU-resident BSGS-GEMM objects (approximately 48 s for Qwen3 (0.6B)). Online, Phase 1 runs prompt prefill entirely on the CPU TEE at approximately 58 ms/token, while Phase 2 invokes four FHE linear projections per layer per decode token at approximately 700 ms per call.
read / write KV Cache
next-token control, post-process
Time Prefill (TEE only)
CPU (TEE Orchestrator)
FHE Accelerator
(b) Bifrost+
Decode (inside Hybrid loop)
KV / state handoff
complete prompt prefill & build initial KV Cache (plaintext inside TEE)
KV / state handoff
per-token compute & update
...
read / write KV Cache
3.5 next-token control, post-process
Trusted and untrusted components. Bifrost trusts the attested CPU TEE, its firmware and attestation chain, and the in-TEE handling of FHE key material. We do not trust the host operating system, hypervisor, device runtime, GPU memory, PCIe-facing software stack, or the FHE accelerator. The accelerator may be a GPU, FPGA, or ASIC backend; in all cases, it is treated as an untrusted ciphertext evaluator rather than as a trusted execution environment. This is the sense in which Bifrost is a “CPU TEE + FHE accelerator” system: FHE protects the accelerator boundary, while the CPU TEE remains the root of trust.
Figure 4. PD-oriented serving split over the hybrid backend. Bifrost and Bifrost+ share the same hybrid decode path. The difference is phase-specific: Bifrost routes every prompt token through the hybrid loop, whereas Bifrost+ completes prompt prefill and prompt-side KV construction inside the CPU TEE before performing a single handoff into the same decode backend.
Protection goal. The primary confidentiality goal is userinput privacy. Bifrost keeps prompts, intermediate activations, persistent cache state selected for the ciphertext path, and TEE-sealed cryptographic material hidden from the untrusted host and accelerator stack. Plaintext appears only inside the CPU TEE. The accelerator observes ciphertexts, deterministic FHE kernel launches, and coarse traffic metadata, but not plaintext activations or prompts.
decode-side state handoff enters the hybrid path. In other words, PD does not change the encrypted decode kernels; it changes which state reaches those kernels and when the handoff occurs. This adapts the PD disaggregation idea from plaintext serving infrastructure [22, 32] to the TEE–FHE trust boundary. Why not route prefill through FHE? The PD split is motivated by a simple cost asymmetry. Let 𝑇 be the prompt length, 𝑐 cpu the cost of one prompt-token step in plaintext inside the CPU TEE, and 𝑐 fhe the cost of one hybrid encrypted step. The three relevant policies have the following first-order TTFT behavior: TTFTTEE ≈ (𝑇 + 1)𝑐 cpu,
Model weights. The default deployment assumes public model architecture and weights, as in many private-input inference settings. This matters because Bifrost’s FHE linear path uses ciphertext–plaintext multiplication with preencoded weight diagonals. Those diagonals are prepared inside the TEE, but the prototype should not be claimed to hide model weights from an accelerator that can inspect plaintext FHE operands. Private-model deployments can instead keep weight handling on the TEE path, use a trusted accelerator, or replace ciphertext–plaintext kernels with costlier ciphertext–ciphertext kernels. The main evaluation focuses on protecting user prompts and intermediate states.
(5)
TTFTBifrost ≈ (𝑇 + 1)𝑐 fhe,
(6)
TTFTBifrost+ ≈ 𝑇𝑐 cpu + 𝑐 fhe .
(7)
Threat Model and Leakage Contract
In our prototype, 𝑐 cpu ≪ 𝑐 fhe . CPU TEE-only is therefore expected to be faster than both Bifrost and Bifrost+ under the CPU-TEE trust assumption. Bifrost+ is not designed to beat that trusted-CPU endpoint. It is designed to avoid unnecessary prompt-side FHE work while preserving the ciphertextonly accelerator boundary during decode. Figure 4 makes the workload consequence explicit. Prompt prefill is matrix-heavy and batched-GEMM
Persistent cache state. The trusted-TEE / ciphertext-only accelerator boundary also covers persistent serving state. The ShadowCipherKV path is a simulated route used to validate serving-stack plumbing and boundary behavior; it is not a CKKS performance claim. Real CKKS KV and packed CKKS 7
4.2
KV overheads are reported separately in the evaluation and appendix. Leakage contract. The prototype hides prompt content and intermediate activations from the host and accelerator; plaintext appears only inside TEE windows or on the ciphertext path. It does not hide prompt/decode length, request size, ciphertext volume, coarse timing, or per-token kernel counts. Under the default public-weight setting, encoded plaintext FHE operands may also be accelerator-visible. Padding, bucketing, dummy kernels, ORAM-style padding, constant-time TEE routines, trusted accelerators, and ciphertext–ciphertext kernels are optional hardening mechanisms complementary to Bifrost’s operator-affinity and PD-split contributions. Out of scope. As in most TEE-backed systems, we do not claim protection against physical attacks, denial-of-service, rollback or fork attacks, supply-chain compromise, microarchitectural side channels, or coarse request metadata such as length, traffic volume, and timing. Defending these channels requires padding, oblivious memory primitives, constant-time TEE implementations, or dummy-kernel emission, which are orthogonal to the trust-boundary partitioning studied here. Why DtE and PD do not widen the trust boundary. The TEE already requires decryption authority to execute non-linear operators and attention-side control logic exactly. DtE therefore does not introduce a new trust root; it reuses the existing CPU TEE boundary to replace homomorphic refresh. Likewise, the PD split changes only when promptside state enters the hybrid decode path. It moves prefill into the same trusted TEE and leaves the accelerator-visible decode boundary unchanged.
4
Implementation
4.1
Software Stack
FHE Backend
The FHE backend builds on HEonGPU [8], an open-source GPU-accelerated CKKS library that provides the underlying homomorphic arithmetic, NTT kernels, and key-switching primitives used by both Bifrost and Bifrost+ . The prototype uses a CKKS parameter set with polynomial degree 𝑁 = 8192, yielding 4096 complex slots, coefficientmodulus primes of [60, 30, 30, 30] bits, initial scale Δ = 230 , and one 60-bit special prime for key switching. This configuration targets an estimated security level of approximately 128 bits under the RLWE hardness assumption used by the HE standard [6]. On top of that substrate, the runtime is organized around a context pool keyed by input dimension. For each unique projection shape, it provisions a CKKS context with the corresponding modulus chain and BSGS rotation parameters, then pre-encodes the weight matrix diagonals into GPU-resident BSGS-GEMM objects. For Qwen3 (0.6B) (𝐿=28 layers, four projections each), this produces 112 persistent objects and takes approximately 48 s on the H20 GPU at model-load time. Once encoded, these objects remain GPU-resident for all subsequent tokens, avoiding per-token re-encoding. Each per-operator FHE linear call then costs approximately 30 ms for CPU-side encrypt, 500–800 ms for the GPU BSGS matrix-vector multiply, and 30 ms for CPU-side decrypt. This organization matches the two dominant deployment regimes seen in evaluation. Fully cached models (e.g., GPT-2 (124M) with all 12 layers resident) behave like clean GEMM-bound cases with 3.1–4.7× speedup over per-token re-encoding. Partially cached models (e.g., Qwen3 (0.6B) with 23 of 28 layers cached) pay a measurable weight-encoding penalty for the remaining fallback layers. The implementation therefore exposes both persistent-cache and fallback-encode modes rather than assuming that all encrypted weights always reside on the device.
Our prototype is implemented atop nano-vLLM, a lightweight runtime that preserves the request/sequence handling, scheduler skeleton, and block-management substrate popularized by vLLM [17, 30]. The main extension is a SecureLLM engine that acts as a drop-in replacement for nanovllm.LLM. A SecureInferenceConfig object selects the secure policy, linear backend, KV mode, and run metadata. In the evaluation harness, the code-level policies pure_fhe, hybrid_unified, and hybrid_pd_split correspond directly to the paper-facing systems Pure FHE, Bifrost, and Bifrost+ . The runtime treats the KV cache as a first-class system component rather than as an external assumption. It exposes a pluggable KVBackend abstraction via PlaintextCPUKV and ShadowCipherKV backends. This is what lets Bifrost and Bifrost+ share the same serving substrate across prefill, decode, and PD handoff: they do not merely assume a logical cache, but instantiate it explicitly in the runtime.
4.3
TEE Integration
In the primary measurement configuration, the trusted side of the prototype runs under Intel TDX on the CPU. The TEE is responsible for attestation, FHE key release, policy control, boundary scheduling, and execution of all TEEresident operators. Decryption and re-encryption buffers are managed inside the trusted runtime so that host-visible components only observe ciphertext movement and fixed control flow. This integration model also explains why DtE is a systems feature rather than a paper-only abstraction. The TEE already needs access to plaintext activations for normalization, Softmax-style reductions, RoPE handling, and other controlheavy operators. Reusing that same authority for refresh keeps the trust boundary unchanged while avoiding the cost of homomorphic bootstrapping inside the accelerator path. 8
4.4
API and Serving Path
and Qwen3.5 (9B) [24] provide scale and CPU-endpoint anchors only; they are not main Bifrost or Bifrost+ FHE result rows. GPT-2 (1.5B) plus LLaMA 3 (8B) appear only as estimator-style continuity rows for Euston comparison. Metrics and prompts. We report time-to-first-token, total latency, and steady-state decode. The main body focuses on prompt lengths 16 and 64 because they expose the TTFT difference between unified hybrid execution and the PDoriented split. Prompt-length-1 rows and additional scale/KV rows appear in the appendix. Construction of the projected Pure FHE baseline. The Pure FHE rows are built from measured primitive latencies for homomorphic linear kernels, non-linear approximations, and refresh, combined with per-model operator counts and prompt/decode invocation structure. This preserves the Pure FHE execution semantics of the Euston-style methodology while avoiding the false impression that the Pure FHE rows are direct end-to-end measurements. KV cache in the serving stack. Bifrost and Bifrost+ do not treat the cache as a paper-only abstraction: the runtime materializes KV state through the same serving substrate that handles request routing and model execution. We report simulated ShadowCipherKV route checks separately from real CKKS encrypted-KV and packed-KV measurements. Packed CKKS KV on GPT-2 (124M) adds 0.248% end-to-end overhead in API mode, while the Qwen3.5 (9B) CKKS KV measurement reports 60.6 ms of KV overhead per step with all correctness checks passing.
The serving interface remains OpenAI-compatible. A hostfacing API server receives requests, forwards them into the trusted runtime, and returns generated tokens through the same request path used for ordinary serving. Inside the TEE, request handling, batching metadata, policy selection, and KV management share one control path; this is what allows Bifrost and Bifrost+ to be presented as routing policies inside a common runtime rather than as separate model rewrites. Phase-aware routing is implemented at the serving layer. Under Bifrost, prompt tokens and decode tokens both use the hybrid path. Under Bifrost+ , prompt prefill stays in the TEE, prompt-side KV state is materialized there, and only the decode-side handoff enters the FHE path. The decode kernels themselves are unchanged.
5
Evaluation
5.1
Measurement Methodology and Baselines
Measurement hierarchy. The evaluation separates direct CKKS/FHE measurements, projected Pure FHE baselines, estimator-style comparison rows, and CPU endpoint rows. Direct CKKS/FHE measurements support the main Bifrost and Bifrost+ results. Pure FHE rows are projections from measured primitive costs and model operator counts, rather than direct serving deployments. Estimator-style rows follow Euston’s methodology for external comparison. CPU endpoint rows characterize the plaintext latency floor under the CPU-TEE trust assumption.
5.2 CPU TEE decode latency (ms/token)
Why direct models are smaller than estimator rows. Direct deployment materializes FHE serving state, not just raw parameters: encoded BSGS diagonals, rotation material, workspace, persistent KV state, and fallback paths compete for accelerator memory. This is why Qwen3 (0.6B) already enters a partially cached regime and GPT-2 (774M) reaches the memory wall on a 96 GiB H20 after only a subset of layers are encoded. Hardware. All primary measurements run on Alibaba Cloud using a single server with 24 vCPUs, 128 GiB system memory, and one NVIDIA H20 GPU with 96 GiB device memory. The CPU side runs under Intel TDX. The H20 instance is CC-capable as provisioned, but Bifrost’s trusted computing base remains the CPU TEE. The GPU remains an untrusted FHE accelerator and sees only ciphertexts and accelerator-side metadata.
Cross-Model Latency Landscape
9B is 8% faster than 8B 466–490 429 ms/tok ms/tok
103
102
101 GPT-2 (124M)
GPT-2 (355M)
GPT-2 (774M)
Qwen3 (0.6B)
Qwen3 (8B)
Qwen3.5 (9B)
Figure 5. Cross-model CPU endpoint decode landscape from 124M to >8B. These rows characterize the CPU-TEE latency floor under the paper’s trust assumption. They are not Bifrost targets to outperform.
Models. Primary direct measurements use GPT-2 checkpoints [23] and Qwen3 (0.6B) [29]. For GPT-2 (124M) and Qwen3 (0.6B), Bifrost and Bifrost+ are direct CKKS/FHE rows, while Pure FHE rows are projected. Qwen3 (8B) [29]
Figure 5 establishes the trusted-CPU latency floor. CPU endpoint decode scales from 13.74 ms/tok on GPT-2 (124M) 9
Table 3. Representative operator and refresh microbenchmarks. These are primitive/operator provenance rows, not Bifrost end-to-end serving results. FHE rows use the GPU backend; TEE rows execute the exact plaintext operator or DtE refresh inside the trusted CPU boundary.
to 490 ms/tok on Qwen3 (8B). These rows are included to show the plaintext floor; encrypted Bifrost, Bifrost+, and Pure FHE rows are reported elsewhere with separate measured/projected roles. Qwen3.5 (9B) reaches 429 ms/tok, 8% faster than Qwen3 (8B) despite a larger parameter count; a separate warm serving-path measurement at 371–388 ms/tok corroborates the benchmark figure. 5.3
GEMM Tiling via TEE Scheduling
Homomorphic matrix multiplication dominates the runtime of transformer inference. We benchmark a representative CKKS GEMM workload of size 1024 × 1024 under different tiling granularities. The baseline performs one monolithic homomorphic GEMM. Tiling instead partitions the matrices into smaller tiles, executes tile-level homomorphic GEMMs on the GPU, and aggregates the outputs homomorphically.
GPU FHE (s)
TEE (s)
Speedup
GELU LayerNorm Softmax Refresh
0.02694 0.01559 0.00935 0.08701
0.000068 0.000678 0.000537 0.00659
398× 23× 17× 13.2×
TEE offloading substantially reduces both runtime and numerical risk for non-linear operators. Compared to GPUbased FHE evaluation, TEE execution is about 398× faster for GELU, 23× faster for LayerNorm, and 17× faster for Softmax in our measured setting. For refresh, DtE is 13.2× faster than GPU bootstrapping. The latency advantage alone would already justify hybrid execution, but the qualitative effect is equally important: the TEE path executes the native operator exactly, avoiding the approximation drift that becomes especially problematic for operators such as Softmax.
105 Time (ms)
Primitive
104 103
5.5 baseline
2x2
4x4
8x8
Table 4. Estimator-style continuity comparison against the Pure FHE SOTA baseline Euston. These rows are projected from the older methodology and serve only as an external baseline reference; they are not direct Bifrost end-to-end deployment results.
Configuration (1024 × 1024) Total
GPU compute
Encrypt
Decrypt
Figure 6. Runtime breakdown of a 1024×1024 homomorphic GEMM under different tiling granularities. The 2×2 configuration is the best point in our setting, reducing GPU compute time from 269.7 s to 28.2 s and end-to-end time from 270.6 s to 30.0 s.
Model GPT-2 (1.5B) LLaMA 3 (8B)
Pure FHE (s)
Bifrost (s)
Speedup
555.544 2804.887
60.048 283.157
9.252× 9.906×
Table 4 restores the external Pure FHE comparison in the same estimation setting used by Euston. Under that continuity setting, Bifrost achieves a consistent ≈ 9× reduction in estimated end-to-end latency. We keep this comparison as projection context only; direct Bifrost claims use direct CKKS/FHE measurements instead of Euston operator or estimator rows.
For this CKKS GEMM, 2×2 tiling is the sweet spot: it improves GPU homomorphic compute efficiency enough to outweigh the extra encryption and decryption work, yielding a 9.01× speedup. More aggressive tiling (4×4, 8×8) leads to rapidly increasing tile-level operations that erode the benefit. This is precisely why the scheduler must choose tiling jointly with TEE-boundary cost rather than treating tile size as a pure accelerator-kernel decision.
5.6 5.4
Comparison to Pure-FHE Baselines
Non-Linear Operators and DtE Refresh
Real End-to-End Hybrid Deployment
Figure 7 and Table 5 present the stepwise end-to-end result. Against the projected Pure FHE reference, Bifrost reduces total latency substantially (e.g., GPT-2 (124M) from 1090 s to 121 s at 𝑃=16) by removing non-linear operators and refresh from the homomorphic path. Bifrost+ then further reduces it to 63 s by moving prompt prefill into the TEE. The decode
A key motivation for hybrid execution is that non-linear operators and ciphertext refresh are expensive, and sometimes numerically fragile, under pure FHE. Table 3 summarizes representative operator-level measurements using the GPU FHE path and the TEE path used by our hybrid system. 10
Table 5. Detailed stepwise end-to-end comparison for the main prompt lengths. Each cell reports TTFT / total latency / steady-state decode in seconds. Pure FHE is projected from measured primitive costs; Bifrost and Bifrost+ are direct CKKS/FHE measurements. Model
Prompt
GPT-2 (124M) GPT-2 (124M) Qwen3 (0.6B) Qwen3 (0.6B)
16 64 16 64
Pure FHE
Bifrost
Bifrost+
579.190 / 1090.240 / 34.070 2214.550 / 2725.600 / 34.070 1946.840 / 2748.480 / 114.520 7443.800 / 8245.440 / 114.520
62.199 / 120.579 / 3.892 249.849 / 308.859 / 3.934 352.302 / 506.631 / 22.047 1419.378 / 1574.631 / 22.179
4.269 / 63.399 / 3.942 5.459 / 63.659 / 3.880 23.023 / 177.261 / 22.034 26.585 / 182.335 / 22.250
Total latency (s)
104
encrypted decode kernels, and steady-state decode changes by at most 2%. The gain comes from deciding when not to use the encrypted accelerator: prompt prefill stays in the CPU TEE, while only the decode-side handoff enters the hybrid path.
103
102
6
Discussion
6.1
Cost Structure and Memory Boundary GEMM
101 GPT-2 (124M) 𝑃 =64
Pure FHE (proj.)
Qwen3 (0.6B) 𝑃 =16 Bifrost (meas.)
DtE / other
97.5% GEMM
GPT-2 (124M) GPT-2 (124M) 𝑃 =16
weight encode
Qwen3 (0.6B) 𝑃=64
2.3%
64.2% GEMM
Qwen3 (0.6B)
34.4%
Bifrost+ (meas.) fallback weight encode 0%
Figure 7. Stepwise end-to-end comparison. Pure FHE bars are projected; Bifrost and Bifrost+ bars are direct CKKS/FHE measurements.
Table 6. Bifrost+ TTFT measurements for direct CKKS/FHE PD-split rows. Unified TTFT is the same hybrid decode path without the prompt-side PD split. PD split removes promptside encrypted passes; it does not speed up the FHE kernels.
GPT-2 (124M) GPT-2 (124M) GPT-2 (124M) Qwen3 (0.6B) Qwen3 (0.6B)
Prompt
Bifrost+ TTFT
Unified TTFT
Gain
1 16 64 16 64
3.888 s 4.269 s 5.459 s 23.023 s 26.585 s
7.769 s 62.199 s 249.849 s 352.302 s 1419.378 s
2.0 × 14.6 × 45.8 × 15.3 × 53.4 ×
75%
100%
Table 7. FHE decode cost breakdown. GPT-2 (124M) is fully cached; Qwen3 (0.6B) pays fallback weight encoding because not all encoded weights fit in accelerator-resident cache.
Bifrost+ over Bifrost
Model
50%
Figure 8. Cost structure and systems boundary. In fully cached GPT-2 (124M) FHE decode, GEMM dominates runtime and DtE is small. In Qwen3 (0.6B), five fallback layers cause a substantial weight-encoding penalty, exposing GPU memory capacity as a first-order systems constraint.
column confirms that neither hybrid policy changes the pertoken decode cost—the gain comes from doing less encrypted work, not from faster encrypted kernels. 5.7
25%
Model
Total
GEMM
DtE
Fallback
GPT-2 (124M), 12/12 3.88s 3.78s (97.5%) 0.09s (2.3%) 0 Qwen3 (0.6B), 23/28 22.01s 14.14s (64.2%) 0.31s (1.4%) 7.56s (34.4%)
Table 7 summarizes the FHE decode breakdown. In fully cached GPT-2 (124M) FHE decode, GEMM consumes 97.5% of runtime and DtE only 2.3%, so PD mainly removes repeated prompt-side GEMM. Qwen3 (0.6B) exposes the next constraint: with 23 of 28 layers cached, fallback weight encoding still contributes 34.4% of runtime. This is encodedweight/cache pressure, distinct from reference-route GPU memory utilization. Full accelerator-resident caching is therefore first-order. In the H20 96 GiB budget, GPT-2 (774M) already reaches the memory wall: 23 of 36 layers
Interpretation. Table 6 shows that the gain grows with prompt length because Bifrost+ avoids repeated promptside encrypted passes. This is a scheduling result rather than a kernel-speed result. Bifrost and Bifrost+ use the same 11
LLM serving. vLLM demonstrates paged KV management for LLM deployment [17], and DistServe/Splitwise establish PD disaggregation as a serving optimization [22, 32]. We adapt PD as a split across the TEE–FHE trust boundary.
occupy ∼92.1 GiB after encoding. CPU TEE anchors remain faster, with Qwen3 (8B) at 466–490 ms/token and Qwen3.5 (9B) at 429 ms/token. 6.2
Limitations
8
The prototype has four limitations: (1) memory, not arithmetic, is the dominant boundary—once encoded weights and persistent state stop fitting on device, fallback encoding becomes a first-order cost; (2) the evaluation is single-accelerator and single-tenant—multi-GPU partitioning and admission control remain future work; (3) our threat model excludes rollback, denial-of-service, and microarchitectural side channels; (4) encrypted KV overhead (∼1.25 MiB/token/layer for GPT-2 (124M)) is modest for short prompts but grows with longer contexts.
7
Conclusion
This paper studies confidential Transformer/LLM serving where the CPU TEE is the only trusted execution boundary and the accelerator acts only as a ciphertext evaluator. Under this assumption, FHE is not a latency optimization over plaintext CPU TEE-only execution; it is the mechanism that lets an untrusted accelerator participate without seeing prompts or intermediate activations. Bifrost instantiates this design end to end by keeping linear layers on acceleratorbacked FHE and moving non-linear operators and refresh into the CPU TEE. Bifrost+ then applies PD splitting to keep prompt prefill inside the trusted CPU domain. The systems lesson is selective encrypted execution: remove non-linear operators and refresh from the homomorphic critical path, then remove prompt-side encrypted passes from TTFT. In an estimator-style comparison matching Euston’s methodology, this yields 9.25× and 9.91× projected latency reductions on GPT-2 (1.5B) and LLaMA 3 (8B). In direct deployments, Bifrost+ reduces TTFT from 62.2 s to 4.3 s on GPT-2 (124M) and from 352.3 s to 23.0 s on Qwen3 (0.6B) at prompt length 16. The remaining boundary is memory, so future work should focus on encoded-weight residency, multiaccelerator partitioning, long-context KV management, and optional side-channel hardening.
Related Work
FHE for neural network inference. Private neural inference has progressed from interactive MPC/PHE systems such as MUSE [18] to non-interactive FHE systems. BOLT, NEXUS, BumbleBee, EncryptedLLM, and Euston improve communication, packing, and GPU utilization for secure transformer inference [9, 14, 19, 21, 31]. They provide the strongest pure-FHE baselines but remain constrained by non-linear evaluation, level management, and refresh cost. Multi-scheme and hybrid cryptographic approaches. BLB combines MPC with CKKS-style HE to split transformer operators across schemes [28]. EncryptedLargeLanguage2025 pushes attention into server-side FHE while keeping the rest client-side [5]. TEEFHE and HT2ML show that TEEs can absorb bootstrapping or HE-unfriendly stages [26, 27]. TEE-only and CPU+GPU TEE systems are complementary endpoints for confidential LLM inference, and heterogeneous confidential-computing surveys highlight broader trust and attack-surface considerations for GPU TEEs [7, 25]. TEE+MPC, masking, and trusted-accelerator designs occupy alternative points in the design space and may be better latency choices under different trust assumptions. Our narrower claim is that, to our knowledge, we are the first to instantiate TEE+FHE transformer/LLM inference with a CPU TEE plus accelerator-backed FHE path, and to study both operator-affinity and PD splitting while keeping the accelerator view ciphertext-only.
References [1] [n. d.]. AMD Secure Encrypted Virtualization (SEV). https://www. amd.com/en/developer/sev.html Intel® Trust Domain Extensions (Intel® TDX). [2] [n. d.]. https://www.intel.com/content/www/us/en/developer/tools/trustdomain-extensions/overview.html [3] Rashmi Agrawal, Leo de Castro, Guowei Yang, Chiraag Juvekar, Rabia Yazicigil, Anantha Chandrakasan, Vinod Vaikuntanathan, and Ajay Joshi. 2023. FAB: An FPGA-based Accelerator for Bootstrappable Fully Homomorphic Encryption. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 882–895. doi:10. 1109/HPCA56546.2023.10070953 [4] Asra Ali, Jaeho Choi, Bryant Gipson, Shruthi Gorantala, Jeremy Kun, Wouter Legiest, Lawrence Lim, Alexander Viand, Meron Zerihun Demissie, and Hongren Zheng. 2025. HEIR: A Universal Compiler for Homomorphic Encryption. arXiv:2508.11095 [cs.CR] https://arxiv. org/abs/2508.11095 [5] Roman Bredehoft and Jordan Frery. 2025. Towards Encrypted Large Language Models with FHE. https://huggingface.co/blog/encryptedllm. [6] Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. 2017. Homomorphic Encryption for Arithmetic of Approximate Numbers. In Advances in Cryptology – ASIACRYPT 2017, Tsuyoshi Takagi and Thomas Peyrin (Eds.). Springer International Publishing, Cham, 409– 437. doi:10.1007/978-3-319-70694-8_15 [7] Marcin Chrapek, Marcin Copik, Etienne Mettaz, and Torsten Hoefler. 2025. Confidential LLM Inference: Performance and Cost Across CPU
FHE acceleration. WarpDrive, Neo, and HEonGPU target GPU-accelerated CKKS; Trinity, Alchemist, FAB, and FAST explore domain-specific accelerators; and HEIR plus Orion reduce the compilation cost of encrypted execution [3, 4, 8, 10–13, 16, 20]. Building on HEonGPU, we ask which operators and phases should still pay encrypted execution cost once accelerator-backed encrypted linear algebra is practical. 12
and GPU TEEs. In 2025 IEEE International Symposium on Workload Characterization (IISWC). 84–98. doi:10.1109/IISWC66894.2025.00017 [8] Ali Şah Özcan and Erkay Savaş. 2024. HEonGPU: a GPU-based Fully Homomorphic Encryption Library 1.0. Cryptology ePrint Archive, Paper 2024/1543. https://eprint.iacr.org/2024/1543 [9] Leo De Castro, Daniel Escudero, Adya Agrawal, Antigoni Polychroniadou, and Manuela Veloso. 2025. EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon LacosteJulien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 12677–12688. [10] Xianglong Deng, Shengyu Fan, Zhicheng Hu, Zhuoyu Tian, Zihao Yang, Jiangrui Yu, Dingyuan Cao, Dan Meng, Rui Hou, Meng Li, Qian Lou, and Mingzhe Zhang. 2024. Trinity: A General Purpose FHE Accelerator. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 338–351. doi:10.1109/MICRO61859.2024. 00033 [11] Austin Ebel, Karthik Garimella, and Brandon Reagen. 2025. Orion: A Fully Homomorphic Encryption Framework for Deep Learning. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 734–749. doi:10.1145/3676641.3716008 [12] Guang Fan, Mingzhe Zhang, Fangyu Zheng, Shengyu Fan, Tian Zhou, Xianglong Deng, Wenxu Tang, Liang Kong, Yixuan Song, and Shoumeng Yan. 2025. WarpDrive: GPU-Based Fully Homomorphic Encryption Acceleration Leveraging Tensor and CUDA Cores. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1187–1200. doi:10.1109/HPCA61900.2025.00091 [13] Shengyu Fan, Xianglong Deng, Liang Kong, Guiming Shi, Guang Fan, Dan Meng, Rui Hou, and Mingzhe Zhang. 2025. FAST:An FHE Accelerator for Scalable-parallelism with Tunable-bit. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 92–106. doi:10.1145/3695053.3731407 [14] Xinwen Gao, Shaojing Fu, Lin Liu, Zhuotao Liu, Yuchuan Luo, and Yongjun Wang. 2026. Euston: Efficient and User-Friendly Secure Transformer Inference with Non-Interactivity. In 2026 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 899–918. doi:10.1109/SP63933.2026.00048 [15] Craig Gentry. 2009. Fully Homomorphic Encryption Using Ideal Lattices. In Proceedings of the Forty-First Annual ACM Symposium on Theory of Computing (STOC ’09). Association for Computing Machinery, New York, NY, USA, 169–178. doi:10.1145/1536414.1536440 [16] Dian Jiao, Xianglong Deng, Zhiwei Wang, Shengyu Fan, Yi Chen, Dan Meng, Rui Hou, and Mingzhe Zhang. 2025. Neo: Towards Efficient Fully Homomorphic Encryption Acceleration Using Tensor Core. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 107–121. doi:10.1145/3695053.3731408 [17] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). ACM, 611–626. doi:10.1145/ 3600006.3613165 [18] Ryan Lehmkuhl, Pratyush Mishra, Akshayaram Srinivasan, and Raluca Ada Popa. 2021. Muse: Secure Inference Resilient to Malicious Clients. In 30th USENIX Security Symposium (USENIX Security 21). 2201–2218.
[19] Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and Wenguang Chen. 2025. BumbleBee: Secure Two-party Inference Framework for Large Transformers. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025. The Internet Society. [20] Jianan Mu, Husheng Han, Shangyi Shi, Jing Ye, Zizhen Liu, Shengwen Liang, Meng Li, Mingzhe Zhang, Song Bian, Xing Hu, Huaiwei Li, and Xiaowei Li. 2024. Alchemist: A Unified Accelerator Architecture for Cross-Scheme Fully Homomorphic Encryption. In Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC ’24). Association for Computing Machinery, New York, NY, USA, 1–6. doi:10.1145/ 3649329.3657331 [21] Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. 2024. BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers. In 2024 IEEE Symposium on Security and Privacy (SP). 4753–4771. doi:10.1109/SP54263.2024.00130 [22] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA ’24). ACM, 118–132. doi:10.1145/3620665.3640401 [23] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019). [24] Qwen Team. 2026. Qwen3.5: Accelerating Productivity with Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5 [25] Qifan Wang and David Oswald. 2026. Confidential Computing on Heterogeneous CPU-GPU Systems: Survey and Future Directions. Comput. Surveys 58, 9 (Feb. 2026), 230:1–230:35. doi:10.1145/3793532 [26] Qifan Wang, Lei Zhou, Jianli Bai, Yun Sing Koh, Shujie Cui, and Giovanni Russello. 2023. HT2ML: An Efficient Hybrid Framework for Privacy-Preserving Machine Learning Using HE and TEE. Computers & Security 135 (Dec. 2023), 103509. doi:10.1016/j.cose.2023.103509 [27] Wenhao Wang, Yichen Jiang, Qintao Shen, Weihao Huang, Hao Chen, Shuang Wang, XiaoFeng Wang, Haixu Tang, Kai Chen, Kristin Lauter, and Dongdai Lin. 2019. Toward Scalable Fully Homomorphic Encryption Through Light Trusted Computing Assistance. arXiv:1905.07766 [cs] doi:10.48550/arXiv.1905.07766 [28] Tianshi Xu, Wen-jie Lu, Jiangrui Yu, Yi Chen, Chenqi Lin, Runsheng Wang, and Meng Li. 2025. Breaking the layer barrier: remodeling private transformer inference with hybrid CKKS and MPC. In Proceedings of the 34th USENIX Conference on Security Symposium (Seattle, WA, USA) (SEC ’25). USENIX Association, USA, Article 137, 20 pages. [29] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] doi:10. 48550/arXiv.2505.09388 [30] Xingkai Yu. 2026. GeeeekExplorer/Nano-Vllm. [31] Jiawen Zhang, Xinpeng Yang, Lipeng He, Kejia Chen, Wen-jie Lu, Yinghao Wang, Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. 2025. Secure Transformer Inference Made Non-interactive. In Proceedings 2025 Network and Distributed System Security Symposium. Internet Society, San Diego, CA, USA. doi:10.14722/ndss.2025.230868
13
[32] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’24). USENIX Association, 193–210.
14
A
Generalization
Algorithm 3: Hybrid PD-split serving protocol Input: prompt tokens[1..𝑇 ], generation budget 𝑛, policy Output: generated response tokens // Offline: encode weights once at model load 𝑊 ← LoadModel( ) 𝐺, 𝐶 ← EncodeWeights(𝑊 , Θ) // GPU-persistent BSGS objects
The architecture is broader than the current GPU-backed prototype. Because the accelerator sees ciphertexts only, the same split applies to FPGA- and ASIC-based FHE backends as well as to future GPU generations. Likewise, the trusted-side logic is not tied specifically to Intel TDX: the same CPU-TEE role could be instantiated with AMD SEV-style deployments so long as attestation and key release support the same trust assumptions. The operator-affinity argument should also generalize beyond the exact models in this paper. GPT-2, Qwen3, and Qwen3.5 already span classical transformer blocks, RMSNorm, RoPE, grouped-query attention, and mixed architectural motifs. Qwen3.5 (9B) is particularly informative as an SSM-heavy architecture-mix anchor [24]: SSM-style layers can remain entirely on the CPU TEE without FHE overhead, yielding a 3.7–4.4× FHE speedup over all-attention architectures of comparable size.
// Phase 1: fast CPU TEE prefill (no FHE) for 𝑡 = 1 to 𝑇 do ℎ ← Block(ℎ,𝑊 , 𝑡 ) // CPU plaintext matmul KV [𝑙 ] ← Append(𝑘𝑙 , 𝑣𝑙 ) for each layer 𝑙 𝑡 0 ← Sample(LMHead(ℎ) ) // Phase 2: FHE decode loop for 𝑖 = 1 to 𝑛 − 1 do if 𝑡𝑖 −1 = EOS then break foreach layer 𝑙 do TEE ops: Norm, RoPE, Softmax/control, Gating GPU FHE: FHE_Linear(ℎ, 𝐺 [𝑙 ], 𝐶 ) for each projection 𝑡𝑖 ← Sample(LMHead(ℎ) ) return [𝑡 0 , 𝑡 1 , . . .]
B
Algorithmic Details
This section provides formal pseudocode for two systemlevel procedures that complement the boundary-aware scheduler presented in the main text (Algorithm 1). Operator partitioning. Algorithm 2 formalizes how each transformer operator is assigned to an execution domain. Non-linear operators that require plaintext access (normalization, positional encoding, attention, and activations) are routed to the CPU TEE, while the four linear projections per layer are routed to GPU FHE when the accelerator is untrusted. This partition is the basis of the operator-domain mapping summarized in Table 2 of the main text. PD-split serving protocol. Algorithm 3 describes the end-to-end Bifrost+ serving protocol. The offline phase preencodes model weights into GPU-resident BSGS-GEMM objects once at model-load time. The online phase then runs in two stages: Phase 1 executes prompt prefill entirely on the CPU TEE using plaintext matmul (no FHE), while Phase 2 enters the hybrid decode loop where each layer invokes
four FHE linear projections on the GPU and all remaining operators execute on the CPU TEE.
C
Supporting Evaluation Tables
This section collects supplementary evaluation data that supports the main-text results but is not essential to the primary narrative. Prompt-length-1 four-way matrix. Table 8 reports TTFT, total latency, and steady-state decode for a single-token prompt across all four execution policies. These rows complement the prompt-16 and prompt-64 results in the main text by showing the limiting case where prefill cost is minimal and the PD advantage is smallest. Table 8. Prompt-length-1 TTFT / total / decode (s).
Algorithm 2: Operator partitioning by execution domain Input: transformer model 𝑀 with 𝐿 layers, threat model 𝑇 Output: partition 𝑃 mapping each operator to a domain foreach operator op in 𝑀 do if op.type ∈ {Norm, RoPE, Softmax, AttentionControl, Activation} then 𝑃 [op] ← TEE_PLAINTEXT // must see plaintext else if op.type ∈ {Linear_QKV, Linear_O, Linear_GateUp, Linear_Down} then if 𝑇 .gpu_trusted then 𝑃 [op] ← GPU_PLAIN else 𝑃 [op] ← GPU_FHE
rows.
Each
cell:
Model
CPU TEE
Pure FHE
Bifrost
Bifrost+
GPT-2 (124M) Qwen3 (0.6B)
.05 / .33 / .02 .11 / .52 / .06
68 / 579 / 34 229 / 1031 / 115
7.8 / 66 / 3.9 45 / 202 / 22
3.9 / 62 / 3.9 22 / 177 / 22
CPU TEE and large-model scaling references. Table 9 reports CPU endpoint latency across model scales. These rows are included as the plaintext latency floor under the CPU-TEE trust assumption, not as Bifrost measurements. Notably, Qwen3.5 (9B) is 8% faster than Qwen3 (8B) despite its larger parameter count, likely because its SSM-heavy architecture mix [24] reduces the attention-heavy workload that dominates at this scale. KV measurement separation. Table 10 separates the three KV measurement types used in the paper. Simulated
return 𝑃 15
Table 9. CPU endpoint latency-floor reference across scales. Bifrost does not try to outperform this endpoint; it targets ciphertext-only accelerator delegation.
main body treats exact TEE execution of non-linear operators as both a latency optimization and a numerical-stability advantage.
Model
CPU endpoint
Role
Table 12. Representative non-linear operator latencies across backends.
GPT-2 (124M) Qwen3 (0.6B)
13.74 ms/tok 58.55 ms/tok
Qwen3 (8B)
466–490 ms/tok
Qwen3.5 (9B)
429 ms/tok
primary CPU baseline JSON primary CPU baseline JSON, BF16 row scale/reference CPU endpoint; not a Bifrost/FHE main result scale/reference CPU endpoint; not a Bifrost/FHE main result
ShadowCipherKV rows exercise route plumbing and boundary behavior only; real CKKS KV rows support encryptedcache overhead and correctness claims; packed CKKS KV rows support packing and memory-reduction claims. Table 10. KV-cache measurement classes. Simulated ShadowCipherKV, real CKKS encrypted KV, and packed CKKS KV are separate classes and should not be collapsed into one performance claim. KV measurement
Metric and role
ShadowCipherKV route
Simulated infrastructure for route plumbing and boundary behavior only; no CKKS performance claim. Real CKKS encrypted KV microbenchmark; append 2.07 ms, read 4.74 ms, correctness passing. Packed real CKKS KV; 0.248% overhead, 5.32 × memory reduction, and 16/16 token agreement. Real CKKS encrypted KV; 60.6 ms/step, 0.046% overhead, and 32/32 token agreement.
GPT-2 (124M) CKKS KV GPT-2 (124M) packed KV Qwen3.5 (9B) CKKS KV
Ciphertext refresh microbenchmark. Table 11 compares three refresh strategies: GPU-based bootstrapping, CPUbased bootstrapping, and TEE-assisted DtE. DtE is 13.2× faster than GPU bootstrapping and over 4 300× faster than CPU bootstrapping, which is the primary reason the hybrid architecture replaces homomorphic refresh with DtE. Table 11. Ciphertext refresh microbenchmark. Primitive
Backend
Avg. time (s)
Bootstrapping Bootstrapping DtE refresh
GPU FHE CPU FHE CPU TEE
0.0870 28.623 0.00659
Non-linear operator latencies. The final table reports per-operator latency for GELU, LayerNorm, and Softmax across CPU FHE, GPU FHE, and TEE plaintext backends. TEE execution is orders of magnitude faster than either FHE backend and preserves native operator semantics exactly, whereas FHE approximations introduce numerical deviation that grows especially large for Softmax. This is why the 16
Operator
Backend
Avg. time (s)
GELU GELU GELU
FHE (CPU) FHE (GPU) TEE plaintext
2.766 0.0269 6.77 × 10−5
LayerNorm LayerNorm LayerNorm
FHE (CPU) FHE (GPU) TEE plaintext
2.241 0.0156 6.78 × 10−4
Softmax Softmax Softmax
FHE (CPU) FHE (GPU) TEE plaintext
1.090 0.00935 5.37 × 10−4