Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints

Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints Abid Mohamed Nadhira,∗ , Ahmad Al Hanbalib,c,∗∗ and Beggas Mounira,∗ a El Oued University, El Oued, Algeria b King Fahd University of Petroleum and Minerals (KFUPM), Dhahran, Saudi Arabia

arXiv:2610.02074v1 [cs.AI] 1 Oct 2026

c Interdisciplinary Research Center For Smart Mobility and Logistics, KFUPM, Dhahran, Saudi Arabia

ARTICLE INFO

ABSTRACT

Keywords: Homomorphic encryption Reinforcement learning Polynomial activation Privacy-preserving machine learning Bellman operator Advantage function

Privacy-preserving machine learning presents significant deployment challenges on the cloud for intelligent systems with confidential data. Fully Homomorphic Encryption (FHE) offers a compelling solution for secure computation, preserving data confidentiality of cloud computations. However, applying FHE to reinforcement learning (RL) requires replacing non-linear operations with polynomial approximations, which diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift. This article introduces the Homomorphic Advantage Operator (HAO), a stabilization framework designed to prevent polynomial approximation divergence in FHE-based deep RL. HAO adapts the zero-mean centering projection from advantage-based value estimation directly to temporal-difference (TD) targets. This linear projection annihilates the uniform statevalue baseline that drives the Bellman drift, maintaining per-state action rankings while requiring zero additional non-linear multiplicative depth and avoiding expensive ciphertext bootstrapping. The proposed HAO framework was evaluated using a three-tier experimental methodology, including a tabular Markov Decision Process (MDP), an encrypted CartPole environment using real CKKS cryptographic operations, and a 20-node logistics routing benchmark with dense continuous features. The results demonstrate that the proposed HAO strictly bounds network pre-activations within the safe polynomial approximation domain. The proposed HAO RL agents achieved 0% boundary breaches across all random seeds used, whereas regularization alone (L2 weight decay and gradient clipping) breached the bound on 3 of 5 seeds and the unstabilized baseline did so in 83.8% of episodes. Finally, HAO agents improve optimal policy accuracy by 18.0 percentage points in tabular domains and remain stable when DP-SGD-style Gaussian noise is added to the clipped gradients.

1. Introduction Reinforcement learning (RL) has emerged as a powerful tool for optimizing complex, dynamic systems, from autonomous robotics to large-scale industrial supply chains and logistics [31, 7, 28]. However, as these intelligent systems are increasingly deployed in cloud environments, they face stringent data privacy and confidentiality requirements. In scenarios such as smart logistics routing, transmitting sensitive operational data, like proprietary demand forecasts, financial data, or real-time vehicle GPS coordinates, to centralized servers presents critical security vulnerabilities [18, 17]. Recent work has shown that CKKS-based encryption can protect such location data in outsourced services such as ride-sharing and traffic management [33]. Fully Homomorphic Encryption (FHE) offers a compelling solution to this privacy bottleneck by enabling computation directly on encrypted data [8]. Under an FHE paradigm, a client can encrypt their operational state and transmit ciphertexts to a cloud-hosted neural network, which then computes and returns encrypted policy decisions without ever exposing the underlying raw data [9, 15]. While we refer to this broad paradigm as FHE, our practical implementation utilizes Leveled Homomorphic Encryption (LHE). By

exploiting the fixed depth of the neural network architecture, LHE avoids the prohibitive computational cost of ciphertext bootstrapping, enabling efficient real-time policy evaluation. Despite its cryptographic guarantees, a fundamental limitation of FHE schemes such as CKKS is that they natively support only addition and multiplication [8]. Consequently, standard non-linear activation functions like ReLU or sigmoid must be replaced with polynomial approximations [27, 3]. The standard approach utilizes degree-𝑑 Chebyshev polynomials, which provide minimax approximation over a strictly bounded convergence domain [−𝐵, 𝐵] [9, 3]. For a degree-2 polynomial evaluated on [−5, 5]: 𝑃2 (𝑧) = 0.469 + 0.5𝑧 + 0.094𝑧2 ≈ max(0, 𝑧)

(1)

abid-mohamednadhir@univ-eloued.dz (A.M. Nadhir); ahmad.alhanbali@kfupm.edu.sa (A.A. Hanbali); beggas-mounir@univ-eloued.dz (B. Mounir)

If the neural network’s pre-activation functions move out-of-bounds [−𝐵, 𝐵], the higher-order polynomial terms cause catastrophic numerical explosion [3, 16]. For example, at 𝑧 = 50, 𝑃2 (50) = 260.47 compared to ReLU(50) = 50. This failure mode is well-documented in FHE-based supervised learning, where solutions include boundary loss penalties [3], trainable polynomial coefficients [27], and modified training methodologies [4]. None of the supervised-learning solutions addresses the unique dynamics of RL, and existing privacy-preserving RL frameworks [24, 25] do not target the polynomial activation divergence caused by recursive temporal-difference (TD) bootstrapping. In supervised learning, target labels are fixed,

A. M. Nadhir et al.: Preprint submitted to Elsevier

Page 1 of 12

∗ Corresponding author

ORCID (s):

Homomorphic Advantage Operator

static constants. In RL, the TD targets are computed sequentially from the network’s own outputs via bootstrapping [31]: 𝑦𝑡 = 𝑟𝑡 + 𝛾 max 𝑄(𝑠𝑡+1 , 𝑎; 𝜃) 𝑎

(2)

where 𝑟𝑡 is the transition reward, 𝛾 ∈ [0, 1) is the discount factor, and 𝑄(𝑠𝑡+1 , 𝑎′ ; 𝜃) evaluates next-state actions under current network parameters 𝜃. Because each forward pass through the encrypted network injects a small polynomial approximation error, 𝜖𝑎𝑝𝑝 . Because the target depends on the 𝑄 values computed with the polynomial, the error accumulates recursively throughout training. The resulting steady-state value function error is bounded by ̂ ∞ ≤ 𝜖𝑎𝑝𝑝 ∕(1 − 𝛾) [5, 22]. At a standard dis‖𝑄∗ − 𝑄‖ count factor of 𝛾 = 0.99, this results in a 100× amplification of the polynomial error. The error amplification creates a positive feedback loop—which we formally term the Bellman drift—that pushes pre-activation values out of the Chebyshev boundary, rendering standard FHE architectures numerically unstable. To address this instability, the zero-mean advantage centering principle was applied directly to the temporaldifference bootstrapping targets while employing dueling architectures [32]. Note that the dueling architectures perform internal stream centering for network identifiability; however, they still output drift-prone 𝑄-values. Our approach tackles this shortcoming by introducing a holistic architectural framework for privacy-preserving RL. Our specific contributions are as follows:

4. We show that the HAO framework remains stable when Gaussian noise of the kind used in DP-SGD is added to the clipped gradients (𝜎 up to 1.0), which makes it compatible with differentially private training. The remaining part of the paper is organized as follows. In Section 2, the related literature is reviewed. Section 3 explains the proposed methodology in detail. Section 4 evaluates the proposed framework numerically. In section 5, we discuss the main findings and key insights. Finally, Section 6 concludes the paper, including possible future extensions.

2. Related Work

This section reviews the existing literature surrounding secure machine learning and cryptographic protocols. We categorize prior contributions into three distinct areas: techniques for managing polynomial activations under HE constraints, foundational cryptographic methodologies, and prior efforts in privacy-preserving RL. As summarized in Table 1, while extensive research addresses stability in supervised learning and static inference, the unique challenges of applying FHE to RL remain unexplored. CryptoNets [9] introduced the use of square activations (𝑥2 ) for HE-compatible inference. Subsequent work improved accuracy through self-learning activation functions (SLAF) [27, 34], HE-friendly training with knowledge distillation [4], and boundary loss penalties with selective gradient clipping [3]. The ReBoot framework [26] enables fully encrypted training via depth-minimized architectures 1. We identify and formally define the Bellman drift as with CKKS bootstrapping. an RL-specific failure mode under FHE polynomial Crucially, the previous stabilization methods are explicconstraints, distinguishing it from the static approxiitly designed for supervised learning paradigms. Boundary mation errors found in supervised learning literature. loss penalties and SLAF rely on the assumption that target la2. We propose the Homomorphic Advantage Operator bels are static, ground-truth constants provided by a dataset. (HAO), a tripartite stabilization framework. It adapts In RL, the agent generates its own dynamic targets via the zero-mean advantage centering principle of Dutemporal-difference (TD) bootstrapping. The static trainingeling architectures [32], but shifts its application ditime regularizers are insufficient to prevent the subsequent rectly to the value recursion. Whereas dueling netBellman drift because the targets themselves are impacted works perform internal stream centering for network by the polynomial approximation error. identifiability while still outputting full drift-prone 𝑄The CKKS scheme [8] supports approximate arithmetic values, HAO applies a zero-mean linear projection on encrypted real numbers, serving as the foundation for matrix 𝑃 directly to the temporal-difference bootstrapour continuous state-space operations. Alternate approaches ping targets. This projection purges the uniform stateinclude GAZELLE [15], which combines HE linear layvalue baseline 𝑉 (𝑠) from the recursive error loop, thus ers with garbled circuits for exact non-linear operations, eliminating Bellman drift with no additional multiand Glyph [20], which switches between CKKS and TFHE plicative depth. When paired with client-side reward schemes to optimize different operation types. While mathescaling and decoupled weight decay, HAO strictly matically robust, these mixed-protocol techniques incur sigbounds pre-activations within degree-2 polynomial nificant communication and computational overhead, renlimits without requiring ciphertext bootstrapping. dering them difficult to scale for highly sequential RL en3. We demonstrate through a three-tier experimental vironments. evaluation, including a tabular MDP, a CartPole enPrior work on privacy-preserving RL can be categorized vironment using real TenSEAL CKKS cryptographic into three distinct paradigms as follows: operations, and a single-depot capacitated routing (I) Several cryptographic data-protection approaches utibenchmark, that the HAO framework achieves 0% boundary breaches, whereas regularization alone breaches lizing Multi-Party Computation (MPC) or Differential Privacy (DP) are proposed to protect distributed parameters. the bound on 3 of 5 random seeds and the unstabilized Namely, federated RL frequently deploys secure aggregation baseline in 83.8% of episodes.

A. M. Nadhir et al.: Preprint submitted to Elsevier

Page 2 of 12

Homomorphic Advantage Operator

to mask client weight updates [6], while DP-SGD bounds gradient privacy leakage [1]. Homomorphic encryption has likewise been used to protect gradients in distributed and federated deep learning [2, 10]. However, these approaches do not provide a mechanism for an untrusted cloud server to compute policy forward passes directly on encrypted continuous state features. (II) A growing body of work explores direct evaluation on encrypted data. For example, Park et al. [25] proposed a privacy-preserving Q-learning framework using LWE-based FHE in cloud computing, validating feasibility on tabular problems but without neural-network function approximation or polynomial activations. Liu et al. [19] introduced the Preyer framework for dynamic treatment regimes, designing a plaintext length control protocol to manage bit-length overflow during repeated encrypted operations in small discrete state spaces. Jesu et al. [14] studied DQN training over AESencrypted state inputs; notably, they evaluated homomorphic encryption (CKKS) but concluded that its computational latency was infeasible for iterative RL, falling back to symmetric block ciphers where performance collapsed on complex environments. Most recently, Nguyen et al. [24] introduced a groundbreaking privacy-preserving Soft ActorCritic (SAC) framework using CKKS with full bootstrapping, achieving performance within 10% of plaintext baselines on CartPole and Pendulum cases. Their core innovation is an HE-compatible Adam optimizer that reparameterizes momentum to bypass high-degree polynomial approximations of inverse square roots. However, their framework requires high-degree minimax polynomials (degrees 15 and 31) and full bootstrapping (incurring ≈ 505 s per training step on CartPole), and does not analyze whether preactivations remain within the polynomial approximation domain. (III) RL is increasingly applied to security problems themselves [13], and recent studies investigate RL for FHE compilation rather than RL under FHE constraints. Sefsaf et al. [29] utilize deep RL as a compiler optimization tool to automate FHE code vectorization, where the RL agent operates in plaintext. Similarly, Sun et al. [30] deploy FHE strictly for secure gradient aggregation in usage-based insurance, while policy evaluations remain in plaintext. Table 1 summarizes the surveyed solutions across supervised learning, cryptographic schemes, and privacypreserving RL. While prior RL works address optimization bottlenecks [24] or tabular feasibility [25], none of them addresses the recursive polynomial activation explosion (Bellman drift) inherent to bootstrapped deep Q-learning. To the best of our knowledge, our work is the first to formalize this activation-domain failure mode and provide a zerodepth linear operator that enables stable deep RL under lightweight computations of degree-2 leveled HE, while avoiding costly bootstrapping evaluations.

A. M. Nadhir et al.: Preprint submitted to Elsevier

3. Proposed Methodology To overcome the theoretical limitations of standard regularization under FHE, we introduce a comprehensive architectural framework for privacy-preserving RL. First, this section formalizes the interaction between cryptographic mathematical constraints and the recursive nature of valuebased learning; second, it introduces the Homomorphic Advantage Operator (HAO) to resolve these constraints, then proves its formal properties, and lastly outlines the clientserver deployment architecture. Figure 1 provides a highlevel overview of the proposed framework. We define the privacy-preserving RL agent as a deep Qnetwork mapping environmental states to a vector of action values. To ensure compatibility with the HE scheme, the network is restricted to a degree-d polynomial activation function 𝑃𝑑 . For the mechanical constraint, we model the polynomial bottleneck at the primary hidden layer, where the forward pass is defined as: 𝑄(𝑠, ⋅; 𝜃) = 𝑃𝑑 (𝑠𝑊1 + 𝑏1 )𝑊2 + 𝑏2

(3)

where 𝜃 = {𝑊1 , 𝑏1 , 𝑊2 , 𝑏2 } denotes the network parameters, 𝑠 is the environmental state, and the dot (⋅) indicates that 𝑄(𝑠, ⋅; 𝜃) ∈ ℝ|| represents the vector of Qvalues evaluated across all possible actions in the action space  with cardinality ||. Additionally, 𝑃𝑑 is a degree𝑑 Chebyshev approximation of the ReLU function. This approximation is mathematically valid only within a strictly bounded domain [−𝐵, 𝐵]. Consequently, the pre-activation vector 𝑧1 = 𝑠𝑊1 + 𝑏1 must satisfy the hard constraint ‖𝑧1 ‖∞ ≤ 𝐵, ‖ ⋅ ‖∞ is the supremum norm, to prevent the polynomial divergence. Theorem 1 (Bellman Drift Bound). For a 𝛾-discounted MDP with a per-step polynomial approximation error bounded by 𝜖𝑎𝑝𝑝 , the steady-state value function error satisfies: ̂ ∞≤ ‖𝑄∗ − 𝑄‖

𝜖𝑎𝑝𝑝 1−𝛾

(4)

where 𝑄∗ represents the optimal action-value function, 𝑄̂ denotes the approximated value function under the polynomial evaluation, and ‖ ⋅ ‖∞ is the supremum norm. This theoretical upper bound adapts the finite-time analysis of approximate value iteration established by Munos and Szepesvári [22], proving that even marginal polynomial deviations are severely amplified by the recursive nature of the Bellman operator. To understand why traditional neural network regularization fails to prevent this amplification, we must examine the internal structure of the action values. Any Q-value can be structurally decomposed as 𝑄(𝑠, 𝑎) = 𝑉 (𝑠) + 𝐴(𝑠, 𝑎), 1 ∑ ′ where 𝑉 (𝑠) = || 𝑎′ 𝑄(𝑠, 𝑎 ) is the state-value baseline and 𝐴(𝑠, 𝑎) = 𝑄(𝑠, 𝑎) − 𝑉 (𝑠) represents the relative advantage of an action [32]. In practical environments, the magnitude of the state-value baseline typically dwarfs the advantage function, meaning |𝑉 (𝑠)| ≫ |𝐴(𝑠, 𝑎)|. Standard stabilization techniques such as L2 weight decay and gradient clipping successfully control overall weight Page 3 of 12

Homomorphic Advantage Operator Table 1 Taxonomy of existing solutions for polynomial activation and privacy-preserving learning under HE. PHE = Partially HE, LHE = Leveled HE (no bootstrapping), FHE = Fully HE (with bootstrapping). #

Solution

Reference

HE

Paradigm

Mechanism Summary Exponential penalty on |𝑧| > 𝐵; prevents explosion during training Clips gradients excluding BatchNorm 𝛾, 𝛽; preserves normalization Trainable polynomial coefficients; learns task-specific activations Knowledge distillation + polynomial BN folding Constrains weight norms; bounds preactivations indirectly Scales inputs to [−1, 1]; keeps first-layer activations in domain

Training-Time Stabilization (Supervised) 1

Boundary Loss

Araki et al. [3]

LHE

Supervised

2

Selective Grad. Clipping

Araki et al. [3]

LHE

Supervised

3

SLAF

Pulido-Gaytan & Tchernykh [27]

LHE

Supervised

4

HE-Friendly Training

Baruch et al. [4]

LHE

Supervised

5

Weight Clamping / L2

Standard technique

Any

Supervised

6

Input Normalization

Standard technique

Any

Supervised

Cryptographic & Architectural Techniques 7

CKKS Bootstrapping

Cheon et al. [8]

FHE

Inference

8

Leveled HE

BGV/BFV/CKKS

LHE

Inference

9

Scheme Switching

Lou & Jiang [20]

LHE+

Inference

10 11

CryptoNets (𝑥2 ) ReBoot Framework

Gilad-Bachrach et al. [9] Pirillo & Colombo [26]

LHE FHE

Inference Supervised

12

Hybrid HE + MPC

Juvekar et al. [15]

LHE

Inference

Refreshes ciphertexts to reset noise budget; unlimited depth Pre-allocates depth budget; eliminates bootstrapping latency Switches CKKS ↔ TFHE for arithmetic vs. comparison ops Replaces ReLU with 𝑥2 ; depth-1 activation Depth-minimized encrypted training with CKKS bootstrapping Linear layers in HE; non-linear activations in garbled circuits

Privacy-Preserving Reinforcement Learning 13

FHE Q-Learning

Park et al. [25]

FHE

Tabular RL

14

Preyer

Liu et al. [19]

Additive

Discrete RL

15

State-Encrypted DQN

Jesu et al. [14]

AES

Deep RL

16

HE-SAC

Nguyen et al. [24]

FHE

Deep RL

This paper

LHE

Deep RL

LWE encrypted Q-table updates; no neural network or poly activations Encrypted experience replay with plaintext length overflow control DQN on AES-encrypted states; abandoned HE due to latency HE-compatible Adam optimizer; uses degree 15/31 poly with bootstrapping

This Work 17

HAO (This Work)

magnitude but fail to eliminate the persistent directional pressure exerted by 𝑉 (𝑠). Every TD update systematically pulls the network weights toward this large uniform baseline. This asymmetric force inevitably overpowers symmetric weight decay. The vulnerability is particularly pronounced in environments with dense continuous state spaces, such as logistics routing. Because all input features activate simultaneously, they rapidly accelerate pre-activation growth, virtually guaranteeing a catastrophic breach of the boundary 𝐵.

A. M. Nadhir et al.: Preprint submitted to Elsevier

P-matrix centering eliminates 𝑉 (𝑠) Bellman drift; degree-2, no ct–ct multiplication, no bootstrapping

3.1. The Homomorphic Advantage Operator (HAO) To overcome the theoretical limitations of standard regularization, we introduce the Homomorphic Advantage Operator (HAO). This operator structurally eliminates the persistent Bellman drift without introducing prohibitive computational overhead to the HE circuit. The method integrates a zero-depth centering projection with symmetric regularization.

Page 4 of 12

Homomorphic Advantage Operator Encrypted forward pass Polynomial activation HAO Centering

Cloud Layer Layer 03

Encrypt State [[S]]

Encrypted Centred

DP protected weight [[W]]

Communication Layer Layer 02

5G

Satelite

RSU Encrypted State [[S]]

Encrypted Centred

DP protected weight [[W]]

Warehouse A

Warehouse B

State Layer 01

Destination Cargo Reward ...

CKKS Encrypt/ Decrypt

Figure 1: High-level overview of the HAO framework for privacy-preserving reinforcement learning. The system integrates three interdependent components: (1) CKKS-encrypted state transmission and polynomial forward evaluation, (2) the HAO centering projection that neutralizes Bellman drift, and (3) clipped weight updates with optional DP-SGD-style Gaussian noise.

The core insight is that subtracting the state-value baseline neutralizes the persistent directional drift that drives preactivations outside the polynomial domain [−𝐵, 𝐵]. Crucially, this centering is achieved via a constant linear projection matrix multiplication, which is natively supported by the CKKS scheme and adds zero non-linear multiplicative depth. Formally, we define the centering matrix: 𝑃 = 𝐼|| −

1 𝟏𝟏𝑇 ||

(5)

where  denotes the discrete action space, 𝐼|| is the identity matrix of size || × ||, 𝟏 is the all-ones row vector of length ||, and 𝟏𝑇 is its transpose. For any state 𝑠, the HAO modifies the TD target by projecting the Q-values through 𝑃 : ̃ ⋅) = 𝑄(𝑠, ⋅) ⋅ 𝑃 = 𝐴(𝑠, ⋅) 𝑄(𝑠, (6) ∑ 1 ′ where 𝐴(𝑠, 𝑎) = 𝑄(𝑠, 𝑎) − || 𝑎′ 𝑄(𝑠, 𝑎 ) is the advantage function. Applying this centering projection to the bootstrapping evaluation at the successor state 𝑠𝑡+1 , the modified TD target becomes: ̃ 𝑡+1 , 𝑎) 𝑦𝑡 = 𝑟𝑡 + 𝛾 max 𝑄(𝑠 𝑎

(7)

It is essential to distinguish the HAO projection from the advantage formulation in Dueling DQN [32]. Dueling networks decompose the internal neural network architecture into separate value and advantage streams, recombining 1 ∑ ′ them as 𝑄(𝑠, 𝑎) = 𝑉 (𝑠) + (𝐴(𝑠, 𝑎) − || 𝑎′ 𝐴(𝑠, 𝑎 )). While this achieves parameter identifiability, the final output A. M. Nadhir et al.: Preprint submitted to Elsevier

𝑄(𝑠, 𝑎) still retains the large, drifting 𝑉 (𝑠) baseline, which amplifies the subsequent TD targets error. Moreover, implementing dual-stream branching architectures under FHE incurs significant cryptographic memory and routing overhead. In contrast, HAO operates on a standard single-stream network and applies the centering projection 𝑃 directly to ̃ 𝑡+1 , 𝑎)). By the TD target computation (𝑦𝑡 = 𝑟𝑡 + 𝛾 max𝑎 𝑄(𝑠 feeding strictly de-meaned advantage values into the recursive backup, HAO completely severs the positive feedback loop of the Bellman drift while consuming zero non-linear multiplicative depth. While the centering projection mathematically secures the Bellman contraction, empirical stability in dense environments requires integrating it with scalar bounds and symmetric regularization. The full HAO framework combines three interdependent components: (1) HAO Centering (Eq. 6), which removes the 𝑉 (𝑠) directional bias; (2) Clientside Reward Scaling (𝛼), which proportionally compresses the active advantage stream |𝐴(𝑠, 𝑎)|, keeping the centered targets within polynomial bounds; and (3) Decoupled Weight Decay, which applies differential regularization across layers (𝜆1 > 𝜆2 ) alongside gradient clipping threshold 𝐺 and weight decay 𝜆. This is done to bound the per-step update magnitudes and prevent long-term parameter growth. Our ablation study in Section 4 demonstrates that all three are necessary: centering without regularization diverges (no gradient control), and regularization without centering violates the bound 𝐵 therefore it cannot counteract 𝑉 (𝑠) bias.

Page 5 of 12

Homomorphic Advantage Operator

3.2. Formal Properties and the Operator Distortion Identity Because the HAO modifies the Bellman target, we formalize its theoretical properties by analyzing both static policy evaluation and dynamic temporal-difference bootstrapping. Proposition 1 (HE-Compatibility). The HAO centering requires only a single matrix–vector multiplication 𝑄̃ = 𝑄 ⋅ 𝑃 , which is natively supported by the CKKS scheme. It adds zero additional non-linear multiplicative depth to the HE circuit. Proof. The matrix 𝑃 is constant and precomputed. The operation 𝑄 ⋅ 𝑃 decomposes into CKKS-native additions and multiplications by plaintext constants. No comparison, division, or branching operations are required. It requires no ciphertext–ciphertext multiplication; the plaintext–ciphertext product consumes at most one rescaling level, which fits within the leveled CKKS parameters used here, so no bootstrapping is needed. Proposition 2 (Per-State Action Ranking Preservation). For any fixed state 𝑠 and any action-value vector 𝑄(𝑠, ⋅) ∈ ℝ|| , the HAO centering preserves greedy action selection: ̃ 𝑎) = arg max 𝑄(𝑠, 𝑎). arg max 𝑄(𝑠, 𝑎

𝑎

(8)

Proof. For any state 𝑠, the centered action values are given 1 ∑ ′ ̃ 𝑎) = 𝑄(𝑠, 𝑎)−𝑄(𝑠), ̄ ̄ by 𝑄(𝑠, where 𝑄(𝑠) = || 𝑎′ 𝑄(𝑠, 𝑎 ) ≡ 𝑉 (𝑠) denotes the mean action-value (the state-value basē is independent of the action 𝑎, subtractline). Because 𝑄(𝑠) ing it is a uniform shift across all action values for state 𝑠: [ ] ̄ ̃ 𝑎) = arg max 𝑄(𝑠, 𝑎) − 𝑄(𝑠) arg max 𝑄(𝑠, 𝑎

𝑎

= arg max 𝑄(𝑠, 𝑎).

(9)

𝑎

Thus, greedy policy extraction at any evaluated state is invariant under the HAO projection. Theorem 2 (Span Contraction of Ideal Row Centering). Let  denote the standard Bellman optimality operator, and let row 𝑄(𝑠, ⋅) = ( 𝑄)(𝑠, ⋅) ⋅ 𝑃 denote the idealized rowcentered Bellman operator across all actions at state 𝑠. Then row is isometric in the span seminorm (𝑠𝑝), preserving the 𝛾-contraction: 𝑠𝑝(row 𝑄1 − row 𝑄2 ) ≤ 𝛾 𝑠𝑝(𝑄1 − 𝑄2 ).

(10)

Proof. For any state 𝑠, projecting by 𝑃 subtracts the uniform 1 ∑ ′ scalar mean 𝑦(𝑠) ̄ = || 𝑎′ ( 𝑄)(𝑠, 𝑎 ) from all action components at 𝑠. For any vector 𝑣, 𝑠𝑝(𝑣 ⋅ 𝑃 ) = max𝑖 (𝑣𝑖 − 𝑣) ̄ − min𝑖 (𝑣𝑖 − 𝑣) ̄ = max𝑖 𝑣𝑖 − min𝑖 𝑣𝑖 = 𝑠𝑝(𝑣). Because  is a 𝛾-contraction in the supremum norm, its composition with the span isometry 𝑃 strictly satisfies 𝑠𝑝(row 𝑄1 − row 𝑄2 ) ≤ 𝛾 𝑠𝑝(𝑄1 − 𝑄2 ).

A. M. Nadhir et al.: Preprint submitted to Elsevier

In sample-based model-free deep RL, transition samples (𝑠, 𝑎, 𝑟, 𝑠′ ) provide access only to the observed successor state 𝑠′ = succ(𝑠, 𝑎), leading to the sampled HAO target: ̃ ′ , 𝑎′ ) = (succ 𝑄)(𝑠, 𝑎). (11) 𝑦𝑡 (𝑠, 𝑎) = 𝑟(𝑠, 𝑎) + 𝛾 max 𝑄(𝑠 ′ 𝑎

We now establish the exact algebraic relationship between the sample-based realization succ and the ideal operator row . Theorem 3 (Operator Distortion Identity). For any state 𝑠 and any two actions 𝑎, 𝑎′ ∈  with successor states 𝑠′𝑎 = succ(𝑠, 𝑎) and 𝑠′𝑎′ = succ(𝑠, 𝑎′ ), the difference between the sample-based targets satisfies the exact identity: succ 𝑄(𝑠, 𝑎) − succ 𝑄(𝑠, 𝑎′ ) = 𝐷(𝑎, 𝑎′ ) − 𝐸(𝑎, 𝑎′ ) (12) where 𝐷(𝑎, 𝑎′ ) = row 𝑄(𝑠, 𝑎) − row 𝑄(𝑠, 𝑎′ ) is the true ̄ ′ ) − 𝑄(𝑠 ̄ ′ ′ )] is the relative advantage, and 𝐸(𝑎, 𝑎′ ) = 𝛾[𝑄(𝑠 𝑎 𝑎 successor baseline distortion. Proof. By definition of succ : ̄ ′) succ 𝑄(𝑠, 𝑎) = 𝑟(𝑠, 𝑎) + 𝛾 max 𝑄(𝑠′𝑎 , 𝑏) − 𝛾 𝑄(𝑠 𝑎 𝑏

̄ ′ ). = ( 𝑄)(𝑠, 𝑎) − 𝛾 𝑄(𝑠 𝑎 Subtracting the target for action 𝑎′ : succ 𝑄(𝑠, 𝑎) − succ 𝑄(𝑠, 𝑎′ ) = ̄ ′ ′ )]. ̄ ′ ) − 𝑄(𝑠 [( 𝑄)(𝑠, 𝑎) − ( 𝑄)(𝑠, 𝑎′ )] − 𝛾[𝑄(𝑠 𝑎

𝑎

Because row-centering at state 𝑠 subtracts the identical scalar 𝑦(𝑠) ̄ from all actions at 𝑠, ( 𝑄)(𝑠, 𝑎) − ( 𝑄)(𝑠, 𝑎′ ) ≡ row 𝑄(𝑠, 𝑎) − row 𝑄(𝑠, 𝑎′ ) = 𝐷(𝑎, 𝑎′ ). Setting 𝐸(𝑎, 𝑎′ ) = ̄ ′ ) − 𝑄(𝑠 ̄ ′ ′ )] completes the identity. 𝛾[𝑄(𝑠 𝑎 𝑎

Corollary 1 (Action-Selection Preservation and Scale Invariance). Let 𝑎∗ = arg max𝑎∈ row 𝑄(𝑠, 𝑎) denote the greedy optimal action under the ideal operator. The samplebased target preserves the optimal greedy selection arg max𝑎∈ succ 𝑄(𝑠, 𝑎 𝑎∗ if and only if: ∀𝑎 ≠ 𝑎∗ ,

𝐸(𝑎∗ , 𝑎) < 𝐷(𝑎∗ , 𝑎).

(13)

Furthermore, this criterion is strictly scale-invariant: for any positive linear reward scaling factor 𝛼 > 0, sign(𝛼𝐷 − 𝛼𝐸) ≡ sign(𝐷 − 𝐸). Corollary 1 delineates the exact operational mechanisms of the HAO framework: 1. Theoretical Foundation (row vs. succ ): Theorem 2 guarantees non-expansive span contraction for ideal row centering. In sample-based deep RL, succ serves as a computationally lightweight surrogate whose stability transfers conditionally upon satisfying (13). 2. Reward Scaling (𝛼): As proven by Corollary 1, 𝛼 is scale-invariant with respect to ordinal action rankings and cannot alter the sign of 𝐷 − 𝐸. Its sole role is to bound pre-activation magnitude (‖𝑧‖∞ ≤ 𝐵), preventing catastrophic polynomial domain escape. Page 6 of 12

Homomorphic Advantage Operator

3. Empirical Distortion Margin: At decision-critical bifurcation states (e.g., balance vs. terminal failure in CartPole), 𝐷(𝑎∗ , 𝑎) and 𝐸(𝑎∗ , 𝑎) track each other on the same order of magnitude. Whether 𝐷(𝑎∗ , 𝑎) > 𝐸(𝑎∗ , 𝑎) holds is an empirical property determined by the environment’s reward margins and downstream advantage spreads, which can be directly monitored 𝐸(𝑎∗ ,𝑎) via the distortion ratio 𝜌(𝑠) = max𝑎≠𝑎∗ 𝐷(𝑎 ∗ ,𝑎) < 1.

3.3. Client-Server Architecture and Privacy We consider a logistics company (client) outsourcing RL computation to a cloud service (server), see Figure 2 for a full illustration. We adopt the standard honest but curious threat model used throughout the HE-ML literature [9, 15, 26]. According to this model, the server faithfully executes the protocol but may attempt to infer private information from the observed data. The primary privacy goal is data confidentiality, where the server must not learn the client’s raw states, Q-values, rewards, and actions. The protocol operates as follows: • Forward Pass (Encrypted): The client encrypts the state 𝑠 → [[𝑠]] using CKKS and sends it to the server. The server computes the encrypted forward pass: [[𝑄]] = 𝑃2 ([[𝑠]] ⋅ 𝑊1 + 𝑏1 ) ⋅ 𝑊2 + 𝑏2 , and returns [[𝑄]] to the client. This is a Ciphertext-Plaintext (CxP) protocol where model weights remain in plaintext on the server. • Training Update: The 𝑃 -matrix centering (𝑄̃ = 𝑄 ⋅ 𝑃 ) can be applied either by the server before decryption as a CxP operation to support thin-client deployments, or by the client after the PxP decryption operation to minimize server operations. The client then decrypts the values if not already plaintext, computes TD targets and gradients in plaintext, applies clipping and weight decay, and sends weight updates Δ𝑊 to the server. The weight updates Δ𝑊 transmitted to the server constitute a potential privacy leakage channel, as gradient inversion attacks can reconstruct training data from observed gradients [35]. It is common in privacy-preserving RL to combine FHE with DP. Therefore, the proposed HAO framework is also designed to seamlessly integrate with DP mechanisms. We address gradient leakage with the Gaussian mechanism of DP-SGD [1]. In DP-SGD, the client adds calibrated noise to clipped gradients before data transmission. The gradient clipping bound 𝐺 from the proposed HAO framework serves a dual purpose by stabilizing the Bellman drift and bounding the L2 sensitivity required by the Gaussian mechanism: ̃ = clip(Δ𝑊 , 𝐺) +  (0, 𝜎 2 𝐺2 𝐼) Δ𝑊

(14)

Since DP noise is zero-mean, it does not create directional bias. HAO’s centering addresses the orthogonal problem of A. M. Nadhir et al.: Preprint submitted to Elsevier

systematic Bellman drift. HAO’s gradient clipping bound 𝐺 directly provides the L2 sensitivity parameter, making HAO DP-ready by construction. Our experiments (Section 4) demonstrate that HAO maintains 0% boundary breaches even under strong noise (𝜎 = 1.0). In these experiments, the noise is added to the clipped mini-batch gradient; a formal (𝜖, 𝛿) guarantee additionally requires per-example clipping and a privacy accountant, which we leave to future work.

4. Results To rigorously evaluate the HAO framework, we conduct a progressive series of experiments spanning isolated mechanics to high-dimensional cryptographic deployment. The results uniformly demonstrate that the 𝑃 -matrix projection successfully bounds the Bellman drift while preserving the optimal agent policy. The environments, training parameters, and evaluated modes for these experiments are summarized in Table 2. We isolate the mechanical effect of the 𝑃 -matrix projection using the tabular MDP environment. As shown in Table 3 and Figure 3, the standard Q-learning formulation without stabilization immediately exhibits polynomial inflation, driving pre-activations up to ‖𝑧‖∞ = 1.83 (compared to 0.74 under HAO). In unconstrained environments, this super-linear growth rapidly escalates toward the Chebyshev boundary 𝐵 = 5.0, where the steep derivative of the polynomial activation function amplifies the approximation error. As illustrated in Figure 4, this uncentered baseline exhibits exploding gradient norms of 5.04 ± 0.34. This instability directly impairs learning, degrading policy decision accuracy to only 68.0% ± 8.4%. By contrast, both the Centering Only and Full HAO modes successfully isolate and remove the state-value baseline. This reduces the maximum pre-activations by 2.5× (down to 0.70 and 0.74) and compresses gradient norms by nearly two orders of magnitude (0.06±0.00). Crucially, as reported in Table 3, this numerical stabilization directly translates into superior decision-making. Both centered modes achieve 86.0%±5.5% policy accuracy, representing an 18.0% absolute improvement over standard Q-learning. To observe the theoretical Bellman drift under actual cryptographic constraints, we deployed a hybrid clientserver CartPole agent that utilizes TenSEAL CKKS operations for the forward pass. All experiments were executed using standard Kaggle CPU environments, demonstrating the feasibility of the protocol without requiring specialized hardware accelerators. As shown in Table 4, Figure 5, and Figure 6, under real CKKS arithmetic using a single hidden layer neural network, the standard Leveled HE (LHE) agent completely collapses. The pre-activations grow monotonically from an initial value of 2.44 to an explosive 28.68—a catastrophic breach of 5.7 × 𝐵. This super-linear inflation directly triggers the polynomial approximation error loop formalized in Theorem 1, pushing gradient norms to 25,003 and rendering the agent numerically unstable, capping out at a maximum reward Page 7 of 12

Homomorphic Advantage Operator

CLIENT (Private)

SERVER (Untrusted) Stored Weights 𝑊1 , 𝑏1 , 𝑊2 , 𝑏2

Logistics Environment State 𝑠 ∈ ℝ8 (private)

updates 𝑊1 , 𝑊2 𝑏1 , 𝑏2

reads 𝑊1 , 𝑏1

Encrypted Forward Pass [[𝑧]] = [[𝑠]] ⋅ 𝑊1 + 𝑏1

Reward Scaling 𝑟←𝑟×𝛼 [[𝑠]] encrypted state

Polynomial Activation [[𝑎]] = 𝑃2 ([[𝑧]]) ‖𝑧‖∞ ≤ 𝐵

CKKS Encrypt 𝑠 → [[𝑠]]

reads 𝑊2 , 𝑏2

Output Layer [[𝑄]] = [[𝑎]] ⋅ 𝑊2 + 𝑏2

̃ [[𝑄]] centered Q-values

HAO Centering ̃ = [[𝑄]] ⋅ 𝑃 (ciph × plain) [[𝑄]]

CKKS Decrypt ̃ → 𝑄(𝑠, ̃ ⋅) [[𝑄]] Weight Update (SGD) 𝑊 ← 𝑊 − 𝜂 ⋅ Δ𝑊 − 𝜆𝑊 Action + TD Target ̃ 𝑦𝑡 = 𝑟 + 𝛾 max 𝑄̃ ′ 𝑎 = arg max 𝑄,

Backprop + DP-SGD + WD Clip(𝐺),  (0, 𝜎 2 𝐺2 ), decoupled 𝜆

Δ𝑊 DP-protected

Figure 2: HAO client-server architecture for privacy-preserving RL. The client encrypts private states via CKKS, the server computes an encrypted forward pass with polynomial activations, and the client applies HAO centering after decryption. Weight updates can be perturbed with DP-SGD-style Gaussian noise. The HAO centering (gold) is the core contribution: it removes the 𝑉 (𝑠) Bellman drift without any ciphertext–ciphertext multiplication.

of only 28.0. Conversely, the pre-activations of the HAOstabilized LHE agent never exceed 3.61, remaining inside [−𝐵, 𝐵] throughout training, and settle at ‖𝑧‖∞ = 0.57 by the final episode. This stabilizes the gradients and allows the agent to learn a functional policy (Max. reward 93.0), while the encrypted centering step ([[𝑄]] ⋅ 𝑃 ) takes 1.64 seconds per episode on average. Crucially, in this experiment, the centering projection is applied server-side as a CxP operation ([[𝑄]] ⋅ 𝑃 ), demonstrating that HAO is equally effective when performed directly on encrypted data. A. M. Nadhir et al.: Preprint submitted to Elsevier

To further evaluate HAO on a realistic, high-dimensional industrial task, we deploy a 20-node, single-depot, singlevehicle capacitated routing environment inspired by the Capacitated Dynamic Vehicle Routing Problem (CDVRP) formulation in RL literature [7, 23]. Unlike the low-dimensional or sparse states of the tabular MDP and CartPole, this benchmark stresses polynomial activation bounds via an 8dimensional dense continuous state space (encoding normalized vehicle coordinates, current and remaining customer demands, vehicle capacity utilization, visitation progress, Page 8 of 12

Homomorphic Advantage Operator Table 2 Summary of Experimental Setup Methodology. Experiment

Environment

Architecture

Training Parameters

Evaluated Modes

1: Tabular MDP

10 states, 4 actions

2: Encrypted CartPole

Hybrid client-server using real TenSEAL CKKS operations 20-node routing, 8D continuous states, 6 actions, 𝛼 = 0.05

1 Hidden Layer (244 params) 1 Hidden Layer (114 params) 1 Hidden Layer (486 params)

2000 epochs, 5 random seeds 250 episodes

Standard, Full HAO, Centering Only Standard vs. HAO LHE

1000 eps, 5 seeds, shift at ep 500

Full HAO, Centering Only, Reg Only, No Stab., HAO+DP

3: Logistics Ablation

Across all experiments, we calibrate the reward scale 𝛼 = 0.05 such that Q-value magnitudes stress-test the polynomial domain boundary. This mirrors standard reward scaling practices in deep RL [21, 12]. Total trainable parameters per architecture (including input/output layers): Exp 1 (244 params), Exp 2 (114 params), Exp 3 (486 params).

Table 3 Tabular MDP (2000 epochs, 5 seeds)

Table 5 Realistic Logistics (𝛾 = 0.99, 𝛼 = 0.05, 5 seeds)

Mode

Accuracy (%)

Max ‖𝑧‖∞

Grad Norm

Mode

Standard Full HAO Centering Only

68.0± 8.4 86.0 ± 5.5 86.0 ± 5.5

1.83 ± 0.20 0.74 ± 0.07 0.70 ± 0.08

5.04 ± 0.34 0.06 ± 0.00 0.01 ± 0.00

Table 4 CKKS Encrypted CartPole (250 episodes). Q⋅P time is the mean time per episode of the encrypted centering step. Mode Standard LHE HAO LHE

Max ‖𝑧‖∞

Max ‖∇𝜃‖

Max. Reward

28.68 3.61

25, 003 2.88

28.0 93.0

Breach %

FHE-Deploy?

Full HAO Centering Only Regularization Only No Stabilization

3.89 7.79 6.98 7.82

0.0% 54.5% 23.8% 83.8%

Yes No No No

HAO + DP Moderate HAO + DP Strong

4.12 4.01

0.0% 0.0%

Yes Yes

Avg. baseline 𝑉 (𝑠) systematically overpowers the symmetric pull Q⋅P time (s) of standard weight decay.

N/A The Full HAO framework, however, achieves a 0% 1.64 boundary breach rate across all 5,000 evaluation episodes.

time pressure, and distance to the nearest unvisited node) paired with a 6-action candidate dispatch space. Because all continuous input features activate simultaneously, they accelerate pre-activation accumulation. To rigorously test the robustness of the stabilization mechanisms, we inject a severe non-stationary environment shift at Episode 500. This shift scrambles the delivery demands and perturbs all node locations, delivering a sudden shock to the TD-targets. This forces the agent to rapidly adapt to a novel state distribution, thereby stress-testing the network’s susceptibility to catastrophic gradient explosion under sudden temporal difference errors. As in the tabular experiment, this ablation trains the FHE-compatible network (degree-2 activation) in plaintext; the encrypted forward pass is evaluated in the CartPole experiment. The ablation results (Table 5, Figure 7, and Figure 8) clearly demonstrate that standard neural network stabilization techniques are fundamentally insufficient. The ’Regularization Only’ mode, which relies exclusively on L2 weight decay and gradient clipping, breaches the boundary on 3 of 5 random seeds (23.8% of all episodes). This occurs because the asymmetric force of the drifting state-value

A. M. Nadhir et al.: Preprint submitted to Elsevier

Max ‖𝑧‖∞

By utilizing the 𝑃 -matrix projection to annihilate the directional 𝑉 (𝑠) bias, the clipping and decay mechanisms are finally able to function symmetrically. This confirms that the centering operation is not merely a helpful heuristic, but a strict mathematical necessity for high-dimensional FHE deployment. Gradient inversion attacks present a significant privacy vulnerability in client-server machine learning. Because the HAO framework inherently requires a gradient clipping bound 𝐺 to stabilize the Bellman drift, it naturally provides the L2 sensitivity parameter required by the Gaussian mechanism of DP-SGD. As shown in Table 5 and Figures 9 and 10, integrating HAO with both moderate (𝜎 = 0.3) and strong (𝜎 = 1.0) DP-style noise maintains the perfect 0% boundary breach rate. Because the injected DP noise is strictly zero-mean, it introduces no directional bias and does not disrupt the symmetric stabilization achieved by the centering operation. The marginally lower maximum pre-activation observed under strong noise (4.01 vs. 4.12) reflects the regularizing perturbation of isotropic Gaussian noise, which disrupts systematic parameter drift along any single coordinate. The cost of this added noise is modest, incurring only a marginal reduction in raw reward output (from −2.2 down to −4.1), indicating that HAO tolerates DP-style gradient noise. Page 9 of 12

Homomorphic Advantage Operator

5. Discussion The empirical results across all three experimental domains confirm that standard deep RL stabilization techniques, namely, gradient clipping and L2 regularization, are fundamentally insufficient for FHE deployment. The HAO resolves this critical bottleneck not through heuristic hyperparameter tuning, but through a structural mathematical correction to the Bellman drift error. Our empirical evaluations confirm that the unanchored drift of the state-value baseline 𝑉 (𝑠) acts as a massive, asymmetric force that systematically overpowers symmetric stabilization techniques like L2 weight decay as observed in Figure 7. By deploying the 𝑃 -matrix projection, the HAO framework structurally neutralizes this drift at its source. Isolating the advantage function 𝐴(𝑠, 𝑎) effectively collapses the magnitude of the signal flowing through the polynomial activation, guaranteeing that pre-activations remain bounded within [−𝐵, 𝐵] without requiring heuristic gradient penalties (Table 5). Crucially, the Centering Only ablation still suffered boundary breaches in the high-dimensional logistics environment. While centering successfully stops the asymmetrical drift of 𝑉 (𝑠), an unregularized neural network will still naturally and symmetrically expand its weights over time as it learns (Table 5 and Figure 7). This confirms that the 𝑃 matrix is not a replacement for standard machine learning regularization. Rather, it is the mathematical prerequisite that finally allows standard regularization to function correctly in an FHE context. Our findings also provide critical insight into the computational cost structure of recent encrypted RL architectures. Notably, Nguyen et al. [24] achieved encrypted deep RL by pairing an HE-compatible Adam optimizer with degree-15 and degree-31 minimax polynomial activations. However, supporting such high polynomial degrees necessitates full FHE bootstrapping, resulting in a computational latency of ≈ 505 s per single training step on CartPole [24]. We observe that these high-degree polynomials are fundamentally required to create an artificially wide convergence window that tolerates unconstrained pre-activation growth. By contrast, HAO directly attacks the root cause of this growth by annihilating the 𝑉 (𝑠) drift at its mathematical source, and bounding pre-activations within [−5, 5] using a lightweight degree-2 polynomial. Thus, it eliminates the need for bootstrapping, and its encrypted centering step takes 1.64 s per episode in our CartPole evaluations. This figure times the [[𝑄]] ⋅ 𝑃 step alone, so it is not a like-for-like comparison with a full bootstrapped training step. The two approaches are complementary: HAO provides the activation-domain stabilization necessary for low-degree leveled HE, while optimizer reparameterization [24] resolves gradient-level bottlenecks. In standard FHE-RL, calibrating the reward scale (𝛼) is traditionally the sole mechanism available to researchers to prevent polynomial explosion [30]. By testing HAO across a massive sweep of 𝛼 ∈ [0.01, 0.25], we demonstrated that the framework maintains strict operational stability (‖𝑧‖∞ < 𝐵) across a 4× wider operational window (𝛼 ∈ [0.01, 0.08]) A. M. Nadhir et al.: Preprint submitted to Elsevier

Table 6 Maximum Pre-Activation (‖𝑧‖∞ ) Across Reward Scale (𝛼) Sweeps. Values > 5.0 trigger divergence. 𝛼→

𝟎.𝟎𝟏

𝟎.𝟎𝟐

𝟎.𝟎𝟓

𝟎.𝟎𝟖

𝟎.𝟏𝟎

𝟎.𝟐𝟓

Reg. Only Full HAO

2.14 1.25

4.91 1.87

Breach 2.95

Breach 4.88

Breach Breach

Breach Breach

compared to regularization alone (Table 6). By drastically relaxing this rigid hyperparameter tuning requirement, HAO makes FHE-RL practical to deploy across diverse environments without exhaustive trial-and-error. It is worth noting that adaptive value normalization techniques, such as PopArt [11], are widely used in plaintext deep RL to normalize targets across widely varying reward scales. However, PopArt relies on tracking online statistics and dividing targets by the running standard deviation, a non-polynomial operation that is incompatible with the arithmetic constraints of leveled homomorphic encryption. HAO achieves an analogous stabilizing effect through a purely linear, additive projection matrix 𝑃 , serving as an HEnative counterpart to return normalization that incurs zero multiplicative depth. The HAO framework is architecturally agnostic and inherently synergistic with DP. Because the centering projection is mathematically neutral and introduces no directional bias, the framework cleanly absorbs zero-mean Gaussian noise without destabilizing the polynomial bounds (Table 5 and Figure 9). In our implementation, this supports two distinct privacy tiers: (1) CxP forward evaluation for maximum throughput, where the client keeps their state private; and (2) CxP combined with Gaussian noise on the weight updates, which, with per-example clipping and privacy accounting, can provide formal (𝜖, 𝛿)-privacy guarantees against gradient inversion attacks. Furthermore, the HAO centering matrix 𝑃 is a constant linear projection; therefore, it can be applied either on the client side or directly on the server without any additional ciphertext–ciphertext multiplication, meaning the mathematical framework theoretically extends to full Ciphertext-Ciphertext (CxC) evaluation for strict model confidentiality without increasing the polynomial degree overhead. While HAO successfully anchors the Bellman drift, our current implementation is constrained to a single hiddenlayer network and a degree-2 polynomial activation. These constraints are strictly dictated by the restrictive multiplicative depth limitations inherent to the TenSEAL CKKS encryption scheme. Scaling this framework to deeper architectures or higher-degree polynomials will require the integration of FHE bootstrapping, a computationally expensive operation that remains a central challenge for cryptographic machine learning. Furthermore, while HAO expands the viable operational window for reward scaling, 𝛼 remains an environment-specific hyperparameter that requires baseline calibration.

Page 10 of 12

Homomorphic Advantage Operator

6. Conclusion and Future Work We identified the Bellman drift as a fundamental obstacle to deploying RL under FHE polynomial constraints and proposed the Homomorphic Advantage Operator (HAO) to solve it. Through centering TD targets with a zero-mean projection, HAO eliminates the persistent state-value bias that drives pre-activation growth. In empirical evaluations, HAO achieved 0% boundary breaches while regularization alone breached the bound on 3 of 5 seeds, and simultaneously expanded the viable operational window for reward scaling by 4×. Because it requires no additional non-linear depth and is mathematically neutral, HAO is inherently HEcompatible and tolerates DP-style gradient noise. Thus, this makes it a highly practical and necessary component for privacy-preserving RL. Future work must address the two primary frontiers of FHE-RL deployment: architecture depth and model confidentiality. Overcoming the strict multiplicative depth limits of shallow networks will require the integration of FHE bootstrapping, an open challenge that would enable scaling to deep, multi-layer topologies. For deployments requiring strict model confidentiality, extending the framework to Ciphertext-Ciphertext (CxC) evaluation is a natural next step. Because the HAO centering matrix 𝑃 is a constant linear projection that consumes no multiplicative depth, it can be seamlessly integrated directly onto the server’s ciphertext pipeline, making it perfectly suited for future CxC paradigms.

References [1] Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L., 2016. Deep learning with differential privacy, in: Proceedings of the ACM Conference on Computer and Communications Security (CCS), pp. 308–318. doi:10.1145/2976749.2978318. [2] Antwi-Boasiako, E., Zhou, S., Liao, Y., Dong, Y., 2023. Privacypreserving distributed deep learning via LWE-based certificateless additively homomorphic encryption (CAHE). Journal of Information Security and Applications 74, 103462. doi:10.1016/j.jisa.2023. 103462. [3] Araki, T., et al., 2025. A training framework for optimal and stable training of polynomial neural networks. arXiv preprint arXiv:2505.11589 . [4] Baruch, M., et al., 2022. A methodology for training homomorphic encryption friendly neural networks, in: Proceedings of ACNS Workshop on Security in Machine Learning and its Applications (SiMLA). [5] Bertsekas, D.P., Tsitsiklis, J.N., 1996. Neuro-Dynamic Programming. Athena Scientific. [6] Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H.B., Patel, S., Ramage, D., Segal, A., Seth, K., 2017. Practical secure aggregation for privacy-preserving machine learning, in: Proceedings of the ACM Conference on Computer and Communications Security (CCS). doi:10.1145/3133956.3133982. [7] Boute, R.N., Gijsbrechts, J., van Jaarsveld, W., et al., 2022. Reinforcement learning for logistics and supply chain management: Methodologies, state of the art, and future opportunities. Transportation Research Part E: Logistics and Transportation Review 162, 102719. [8] Cheon, J.H., Kim, A., Kim, M., Song, Y., 2017. Homomorphic encryption for arithmetic of approximate numbers, in: Proceedings of ASIACRYPT, pp. 409–437. doi:10.1007/978-3-319-70694-8_15. [9] Gilad-Bachrach, R., Dowlin, N., Laine, K., Lauter, K., Naehrig, M., Wernsing, J., 2016. CryptoNets: Applying neural networks to

A. M. Nadhir et al.: Preprint submitted to Elsevier

encrypted data with high throughput and accuracy, in: Proceedings of the International Conference on Machine Learning (ICML), pp. 201–210. [10] Gupta, M., Kumar, M., Dhir, R., 2025. Privacy-preserving federated learning in asynchronous environment using homomorphic encryption. Journal of Information Security and Applications 93, 104116. doi:10.1016/j.jisa.2025.104116. [11] van Hasselt, H., Guez, A., Hessel, M., Mnih, V., Silver, D., 2016. Learning values across many orders of magnitude, in: Advances in Neural Information Processing Systems (NeurIPS). [12] van Hasselt, H., Hessel, M., Kroiss, J., 2021. Return-based scaling: Yet another normalisation trick for deep RL. arXiv preprint arXiv:2105.05824 . [13] Javadpour, A., Ja’fari, F., Taleb, T., Turkmen, F., Benzaïd, C., 2026. Beyond reinforcement learning for network security: A comprehensive survey and tutorial. Journal of Information Security and Applications 96, 104294. doi:10.1016/j.jisa.2025.104294. [14] Jesu, A., Darvariu, V.A., Staffolani, A., Montanari, R., Musolesi, M., 2021. Reinforcement learning on encrypted data. arXiv preprint arXiv:2109.08236 . [15] Juvekar, C., Vaikuntanathan, V., Chandrakasan, A., 2018. GAZELLE: A low latency framework for secure neural network inference, in: Proceedings of USENIX Security Symposium, pp. 1651–1669. [16] Lee, E., Lee, J.W., Lee, J., Kim, Y.S., Kim, Y., No, J.S., Choi, W., 2022. Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions, in: Proceedings of the International Conference on Machine Learning (ICML). [17] Li, X., Li, H., Gao, J., Wang, R., 2023. Privacy preserving via multi-key homomorphic encryption in cloud computing. Journal of Information Security and Applications 74, 103463. doi:10.1016/j. jisa.2023.103463. [18] Lin, J., Yu, W., Zhang, N., Yang, X., Zhang, H., Zhao, W., 2017. A survey on internet of things: Architecture, enabling technologies, security and privacy, and applications. IEEE Internet of Things Journal 4, 1125–1142. [19] Liu, X., Deng, R.H., Choo, K.K.R., Yang, Y., 2021. Privacypreserving reinforcement learning design for patient-centric dynamic treatment regimes. IEEE Transactions on Emerging Topics in Computing 9, 456–470. doi:10.1109/TETC.2019.2907409. [20] Lou, Q., Jiang, L., 2020. Glyph: Fast and accurately training deep neural networks on encrypted data, in: Advances in Neural Information Processing Systems (NeurIPS). [21] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al., 2015. Human-level control through deep reinforcement learning. Nature 518, 529–533. doi:10.1038/nature14236. [22] Munos, R., Szepesvári, C., 2008. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research 9, 815–857. [23] Nazari, M., Oroojlooy, A., Snyder, L., Takáč, M., 2018. Reinforcement learning for solving the vehicle routing problem, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 9839–9849. [24] Nguyen, C.H., Hoang, D.T., Nguyen, D.N., Lauter, K., Kim, M., 2025. Empowering artificial intelligence with homomorphic encryption for secure deep reinforcement learning. Nature Machine Intelligence 7, 1913–1926. doi:10.1038/s42256-025-01135-2. [25] Park, J., Kim, D.S., Lim, H., 2020. Privacy-preserving reinforcement learning using homomorphic encryption in cloud computing infrastructures. IEEE Access 8, 203564–203579. doi:10.1109/ACCESS.2020. 3036899. [26] Pirillo, A., Colombo, L., 2026. ReBoot: Encrypted training of deep neural networks with CKKS bootstrapping, in: Proceedings of the AAAI Conference on Artificial Intelligence. [27] Pulido-Gaytan, B., Tchernykh, A., 2024. Self-learning activation functions to increase accuracy of privacy-preserving convolutional neural networks with homomorphic encryption. PLOS ONE doi:10. 1371/journal.pone.0307131.

Page 11 of 12

Homomorphic Advantage Operator

6WDQGDUG )XOOB+$2 &HQWHULQJB2QO\

  

0D[ z

     











(SRFK









Figure 3: Pre-activation growth over 2000 epochs. Centering successfully bounds polynomial inflation and stabilizes training.



*UDGLHQW1RUP

[28] Saleh, Z., Hanbali, A.A., Baubaid, A., 2026. Enhancing courier scheduling in crowdsourced last-mile delivery through dynamic shift extensions: a deep reinforcement learning approach. Computers & Industrial Engineering 212, 111693. doi:10.1016/j.cie.2025.111693. [29] Sefsaf, B., Dandani, A., Seddiki, A., Mohammed, A., Chielle, E., Maniatakos, M., Baghdadi, R., 2026. CHEHAB RL: Learning to optimize fully homomorphic encryption computations, in: Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). [30] Sun, X., Sun, Z., Wang, T., Jiang, Z.L., 2026. Privacy-preserving collective reinforcement learning using fully homomorphic encryption in usage-based insurance. Information Sciences . [31] Sutton, R.S., Barto, A.G., 2018. Reinforcement Learning: An Introduction. 2nd ed., MIT Press. [32] Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., de Freitas, N., 2016. Dueling network architectures for deep reinforcement learning, in: Proceedings of the International Conference on Machine Learning (ICML), pp. 1995–2003. [33] Xie, Q., Zhang, H., Wang, L., Wang, M., Wu, W., Liu, Y., 2025. SecureLoc: A fully homomorphic encryption-based privacy protection scheme for location-based services. Journal of Information Security and Applications 93, 104190. doi:10.1016/j.jisa.2025.104190. [34] Xiong, J., Chen, J., Lin, J., Jiao, D., Liu, H., 2024. Enhancing privacypreserving machine learning with self-learnable activation functions in fully homomorphic encryption. Journal of Information Security and Applications 86, 103887. doi:10.1016/j.jisa.2024.103887. [35] Zhu, L., Liu, Z., Han, S., 2019. Deep leakage from gradients, in: Advances in Neural Information Processing Systems (NeurIPS).



  6WDQGDUG )XOOB+$2 &HQWHULQJB2QO\ %RXQGDU\% 

  









(SRFK









Figure 4: Gradient norm dynamics on a logarithmic scale over 2000 epochs. The uncentered baseline exhibits exponential gradient explosion upon breaching the polynomial boundary, whereas HAO centering compresses gradient variance.



6WDQGDUG/+( +$2/+( %RXQGDU\% 



0D[ z

     





(SLVRGH







Figure 5: Trapping region (pre-activation trajectory) for the hybrid TenSEAL CKKS CartPole agent. The network utilizes a single hidden layer architecture.

A. M. Nadhir et al.: Preprint submitted to Elsevier

Page 12 of 12

























 6WDQGDUG/+( +$2/+( 





(SLVRGH











Figure 6: Optimization health (gradient norm) over 250 episodes (log scale). Standard LHE diverges exponentially.

0D[ z

*UDGLHQW1RUP

Homomorphic Advantage Operator

 )XOOB+$2 +$2B'3B0RGHUDWH +$2B'3B6WURQJ %RXQGDU\%  (QY6KLIW

  







(SLVRGH









Figure 9: Pre-activation robustness under DP noise. HAO successfully bounds the trajectory despite strong DP noise.

0D[ z

  )XOOB+$2 &HQWHULQJB2QO\ 5HJXODUL]DWLRQB2QO\ 1RB6WDELOL]DWLRQ %RXQGDU\%  (QY6KLIW

   





(SLVRGH







Figure 7: Pre-activation growth over 1,000 episodes. The HAO framework strictly bounds pre-activations below 𝐵 = 5.0, whereas uncentered baseline agents diverge under the environment shift.

7RWDO5HZDUG6PRRWKHG



7RWDO5HZDUG6PRRWKHG



 

   













)XOOB+$2 &HQWHULQJB2QO\ 5HJXODUL]DWLRQB2QO\ 1RB6WDELOL]DWLRQ (QY6KLIW

  



)XOOB+$2 +$2B'3B0RGHUDWH +$2B'3B6WURQJ (QY6KLIW



(SLVRGH











(SLVRGH







Figure 10: Agent performance with DP noise injected. The privacy guarantee trades off only a marginal reduction in reward output.

Figure 8: Agent performance (moving average reward) over 1,000 episodes in the 20-node Logistics environment. Full HAO maintains optimal performance post-shift.

A. M. Nadhir et al.: Preprint submitted to Elsevier

Page 13 of 12

Related documents

• Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints#1157422
• Securing Machine Learning Engines in IoT Applications with Attribute-Based Encryption#491027
• Data Protection Issues in Automated Decision-Making Systems Based on Machine Learning: Research Challenges#328645
• Teaching fairness to artificial intelligence: Existing and novel strategies against algorithmic discrimination under EU law#461910
• Machine Learning (In) Security: A Stream of Problems#328794