arXiv CSopen access

Replay-buffer engineering for noise-robust quantum circuit optimization

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Akash Kundu1,2∗ Sebastian Feld1,2 Delft University of Technology, Delft, The Netherlands 2 Quantum Computing Division, QuTech, The Netherlands {A.kundu, S.Feld}@tudelft.nl

arXiv:2604.21863v1 [quant-ph] 23 Apr 2026

Abstract Deep reinforcement learning (RL) for quantum circuit optimization faces three fundamental bottlenecks: replay buffers that ignore the reliability of temporaldifference (TD) targets, curriculum-based architecture search that triggers a full quantum-classical evaluation at every environment step, and the routine discard of noiseless trajectories when retraining under hardware noise. We address all three by treating the replay buffer as a primary algorithmic lever for quantum optimization. We introduce ReaPER+, an annealed replay rule that transitions from TD error-driven prioritization early in training to reliability-aware sampling as value estimates mature, achieving 4-32× gains in sample efficiency over fixed PER, ReaPER, and uniform replay while consistently discovering more compact circuits across quantum compilation and QAS benchmarks; validation on LunarLander-v3 confirms the principle is domain-agnostic. Furthermore we eliminate the quantum-classical evaluation bottleneck in curriculum RL by introducing OptCRLQAS which amortizes expensive evaluations over multiple architectural edits, cutting wall-clock time per episode by up to 67.5% on a 12-qubit optimization problem without degrading solution quality. Finally we introduce a lightweight replay-buffer transfer scheme that warm-starts noisy-setting learning by reusing noiseless trajectories, without network-weight transfer or ϵ-greedy pretraining. This reduces steps to chemical accuracy by up to 85-90% and final energy error by up to 90% over from-scratch baselines on 6-, 8-, and 12-qubit molecular tasks. Together, these results establish that experience storage, sampling, and transfer are decisive levers for scalable, noise-robust quantum circuit optimization.

Introduction

Optimization is one of the most important application areas of quantum computing [1, 2], both as a target problem class and as an internal algorithmic task in the design and execution of quantum algorithms. On current quantum hardware, quantum computation is realized through sequences of elementary logic gates arranged into quantum circuits [3, 4, 5], and their depth, gate count, and hardware compatibility of these circuits directly determines what can be achieved in practice [6, 7]. This makes circuit optimization a central challenge across both near-term and fault-tolerant regimes. In the near term [8, 9], limited coherence, connectivity, and high gate noise place strong pressure on circuit depth, two-qubit gate count, and hardware compatibility. In the longer term [10, 11, 12], fault-tolerant quantum computing will relax some hardware constraints but will not remove the need for efficient circuit synthesis, compilation, and task-adapted circuit design. Across both ∗ Corresponding author.

Preprint.

settings, the ability to construct compact and effective quantum circuits remains a key determinant of performance, resource cost, and practical utility. Figure 1: Overview of replay-buffer engineering for quantum optimization. (Left) Buffer engineering improves experience reuse through replay design and sampling. (Middle) Amortized learning reduces the cost of curriculum RL-based quantum architecture search by performing expensive quantum-classical updates only every m steps. (Right) Noise-aware transfer warm-starts the RL-training in noisy environment by reusing trajectories collected in noiseless training. Buffer engineering

Amortized learning

Noise-aware transfer

1-Step

Noiseless

Observable

Sampling

Experience

2-Step

Replay buffer batch

Noise

m-Step

Noisy

Reinforcement learning (RL) [13] has emerged as a promising framework for this problem because it treats circuit design as a sequential decision process, enabling agents to construct gate sequences or parameterized circuit architectures step by step [14, 15]. RL has been successfully applied to quantum compilation [16, 17] and variational quantum circuit design [14, 18]. However, its practical use remains limited by a basic inefficiency: past experience is often poorly reused. In most existing workflows [14, 19, 20, 21], each new setting is treated as a fresh training problem, so trajectories collected in a noiseless simulator are discarded when the task is revisited under hardware noise which is an especially costly choice in quantum optimization [22], where the noiseless-tonoisy gap represents a shift between fundamentally distinct computational substrates that is physically motivated, practically unavoidable, and grows in severity with system size, making experience reuse both more challenging and more valuable than in classical transfer settings. In the standard variational framework [23], a parameterized quantum circuit U (θ, A) is defined by a discrete ar⊗n chitecture A and continuous parameters θ, preparing a state |ψ(θ, A)⟩ = U (θ, A) |0⟩ , with cost C(θ, A) = ⟨ψ(θ, A)| H |ψ(θ, A)⟩ estimated on the QPU and a classical optimizer updating θ toward targets d(U (θ, A), Utar ) ≤ εcomp or C(θ, A) − E0 ≤ εchem . Each RL step may trigger a full quantum-classical evaluation, so wall-clock time grows rapidly with system size. While TensorRLQAS [24] narrows the search space via tensor-network warm-starts, it still incurs substantial per-step cost and relies on standard replay buffers without explicit experience transfer; similarly, transfer in RL-optimized error-correcting codes [25] does not treat replay-buffer design as the primary transfer vehicle. As a result, the practicality of RL-based quantum optimization depends not only on the agent architecture, but critically on how experience is stored, prioritized, and transferred across the noiseless-to-noisy scenario which is a direction that is not explored yet in the scope of quantum optimization. We address these bottlenecks by treating the replay buffer as a primary algorithmic lever for making RL practical in quantum optimization. This is motivated by evidence from classical RL that buffer composition and sampling strongly affect stability and sample efficiency [26, 27, 28, 29, 30]. Figure 1 summarizes our replay-buffer engineering framework, which combines annealed replay, amortized curriculum learning, and lightweight noiseless-to-noisy transfer. Concretely, we make three contributions: 1. Annealed replay for quantum optimization. We systematically study replay design for quantum compiling and architecture search under a fixed DQN [31]/DDQN [32] setup, 2

isolating the effect of the buffer from the agent or environment. Based on this analysis, we introduce ReaPER+, an annealed replay rule that transitions from TD-error prioritization [33] early in training to reliability-aware replay [34] later on, improving sample efficiency by 4×-32× over fixed PER, fixed ReaPER, and uniform replay while finding more compact circuits. A classical RL validation on LunarLander-v3 (Appendix E) confirms that ReaPER+ is not domain-specific. 2. Amortized curriculum learning. We introduce OptCRLQAS, an amortized variant of CRLQAS [19] that reuses expensive quantum-classical evaluations across multiple architectural edits. On 12-qubit H2 O ground-state preparation, OptCRLQAS reduces average wall-clock time per episode by 67.5% (3× faster) without degrading final energy error or gate count. 3. Replay-based transfer to noise. We propose a lightweight transfer scheme that reuses noiseless trajectories to warm-start learning in noisy settings, without network-weight transfer or long ϵ-greedy pretraining. On 6-qubit BEH2 , and 8- and 12-qubit H2 O molecular tasks, this buffer-only transfer cuts the steps needed to reach chemical accuracy by up to 85-90% and improves final energy error by up to 90% over from-scratch noisy baselines, yielding a transfer advantage that grows with system size.

Related work

Quantum circuit optimization [1] has been approached via adaptive ansatz construction [35], differentiable architecture search [36, 37], evolutionary heuristics [38, 39], Bayesian optimization [40], Monte Carlo tree search [41], neural predictors [42], and sampling-based strategies [43], demonstrating automatic circuit structure discovery but revealing persistent challenges in scaling and noise robustness. Within this landscape, reinforcement learning has emerged as a flexible framework for both quantum compiling [16] and quantum architecture search [14, 44, 45], with value-based and policy-gradient methods applied to variational ground-state preparation [14, 19, 24], entangled-state generation [45, 46], and hardware-aware circuit design [47, 48]. We refer the reader the curated list [49] for a complete list of automated quantum circuit optimization approaches. However, existing RL-based frameworks treat each new setting as a fresh training instance, when a task moves from a noiseless simulator to noisy hardware, the agent is retrained from scratch and accumulated experience is discarded, incurring substantial GPU and CPU cost per retraining cycle. Experience replay is a central component of off-policy deep RL [31], with uniform replay, hindsight experience replay [50], prioritized experience replay [33], and reliability-aware variants [34] providing increasingly structured trade-offs between coverage, informativeness, and target reliability. In classical RL, recent offline-to-online methods [27, 28, 29, 30] demonstrate that buffer composition and sampling rules are critical for stable, sample-efficient fine-tuning. In quantum optimization [19, 25], however, replay buffers have largely been treated as fixed implementation choices rather than as a primary algorithmic design lever. This is the gap our work directly addresses by treating the replay buffer as the central object of design for quantum circuit optimization.

Methods

We introduce a replay-buffer engineering algorithm for quantum circuit optimization as depicted in Figure 1 with three components: (1) annealed replay (ReaPER+, described in the next paragraph), (2) amortized curriculum learning (OptCRLQAS), and (3) lightweight noiseless-to-noisy buffer transfer. To isolate the effect of replay design from other algorithmic choices, all experiments use a common off-policy deep Q-learning (for quantum compiling) and double deep Q-learning (for quantum architecture search) agent with fixed state representation, action space, reward, and training protocol; only the replay mechanism is varied. ReaPER+. Among existing strategies, PER and ReaPER emerge as complementary baselines. PER (see Equation 11) prioritizes transitions with large TD errors, providing aggressive early exploration but potentially amplifying noisy targets. ReaPER (see Equation 13) discounts transitions whose downstream TD errors indicate unreliable targets, better matching the long-horizon structure of episodic quantum optimization but converging more slowly in the early phase. This comple3

mentary behavior motivates a hybrid that transitions from PER-like exploration to ReaPER-like refinement over the course of training. To exploit the best of both regimes, we introduce ReaPER+, a replay strategy that transitions smoothly from PER-like prioritization to ReaPER-like prioritization over the course of training. Specifically, at training step τ we define (+,τ )

(+,τ )

Ψt

(+,τ )

= Rtωτ (δt+ )α ,

µt

Ψ = P t (+,τ ) , i Ψi

(1)

where the annealing exponent ωτ ∈ [0, 1] is non-decreasing in τ ; ωτ = 0 recovers PER and ωτ = 1 recovers ReaPER exactly. In practice, we use the linear schedule τ ωτ = ωmin + (ωmax − ωmin ) min ,1 , (2) Tann which is monotone and controlled by a single interpretable timescale Tann , avoiding the rapid early transitions of cosine or exponential schedules when reliability estimates are still poorly calibrated. We set ωmin = 0.1 > 0 so that a residual TD-error signal is retained at initialization, where Rt carries little meaning under a random Q-function, and ωmax = 0.7 < 1 to prevent premature over-commitment to reliability scores before Q-function convergence (values ωmax ≥ 0.9 produced slower convergence in both settings; see Appendix I). Tann is set to half the total training budget (5 × 105 for compilation, 5 × 104 for LunarLander-v3; Table 9), so the transition completes by midtraining when value estimates are sufficiently stable. Early in training sampling is thus driven by TD error; later, the influence of Rt grows, biasing replay toward transitions that are both informative and reliable. ReaPER+ therefore preserves the sample-efficiency advantages of PER at the beginning of learning while inheriting the stability of ReaPER once value estimates mature, a design choice directly motivated by the empirical finding that PER excels early while ReaPER is preferable once estimates stabilize this is confirmed by our systematic benchmark of uniform replay, HER [50], PER [33], and ReaPER [34] under a fixed agent across all tasks. A theoretical justification is given in Appendix C. Lightweight buffer transfer. In addition to replay-buffer engineering, we study a replay buffer transfer scheme for noise-robust quantum optimization. The idea is to first collect trajectories in a source environment, typically the noiseless version of a given compiling or QAS task, and then use the resulting replay memory to initialize training in the corresponding noisy target environment. Let (0) Bsrc denote the replay buffer obtained after source training and Btgt the initial replay buffer of the target task. We initialize the target buffer directly as (0)

Btgt ← Bsrc ,

(3)

that is, all stored transitions (St , At , Rt , St+1 , dt ) are copied from the source buffer to the target buffer without modification, relabeling, or filtering. This is valid because the noiseless and noisy environments share identical state and action spaces: the RL-state and the gate action set A are unchanged by the introduction of noise, which only affects the transition dynamics and reward statistics. Buffer-only transfer of this kind, where source experience warm-starts a target replay buffer without sharing network weights, has recently proved effective in classical deep RL. Zhou et al. [27] show that seeding the online replay buffer with offline transitions suffices for sampleefficient fine-tuning without retaining the offline dataset or transferring network parameters. Lee et al. [28] study balanced mixing of offline and online replay in the offline-to-online setting, finding that the buffer composition critically determines early learning speed. Our scheme follows the same instance-transfer paradigm [51]: experience is transferred at the trajectory level rather than the parameter level, deliberately decoupling the two mechanisms so that their individual contributions can be isolated through controlled ablations that separately compare buffer transfer, network initialization transfer, and their combination. Consequently, trajectories that are informative in the noiseless setting can still provide a strong initialization for learning under noise by improving early buffer coverage and accelerating the discovery of high-quality circuits. In Appendix D we provide a formal argument for why replay-buffer transfer is natural in quantum optimization settings under the assumption that source and target tasks share approximately similar state and action spaces. 4

OptCRLQAS. Finally, to scale while operating with CRLQAS we realized the time per episode scales rapidly with qubit count, and training the RL-agent with CRLQAS beyond 10-qubit can take a substantial amount of GPU compute. To address this issue we introduce OptCRLQAS, an efficiency-oriented variant of curriculum reinforcement learning-based quantum architecture search. In standard CRLQAS, each environment step triggers a full quantum-classical evaluation of the current circuit, including variational parameter optimization and cost-function estimation, so for an episode of length T , the total evaluation cost scales as T · Ceval , where Ceval denotes the cost of a single quantum-classical call. OptCRLQAS reduces this by amortizing each evaluation over m consecutive architectural edits: rather than invoking a new variational optimization at every step, the agent accumulates m local gate modifications before triggering a single evaluation, spreading the cost Ceval across m steps. Formally, let uτ ∈ {0, 1} denote the update indicator, 1, if τ mod m = 0 or the episode terminates, uτ = (4) 0, otherwise, so that a full architecture evaluation is performed only when uτ = 1. Hence, for an episode of length T , the number of expensive quantum-classical evaluations is reduced from T in CRLQAS to approximately ⌈T /m⌉ in OptCRLQAS, yielding an expected reduction by a factor of about m when episode lengths are sufficiently large. Beyond the computational saving, accumulating m edits before evaluating also improves the learning signal: single-gate modifications are often too small to produce distinguishable rewards when variational parameters θ can compensate local changes, leading to weakly separated TD targets and slow value propagation. Judging a block of m edits jointly increases reward contrast and allows useful gate combinations to be credited as a unit. A more detailed discussion of both effects is provided in Section 4.2. Throughout the paper, OptCRLQAS refers to m = 10 unless stated otherwise.

Results

We evaluate replay-buffer design in quantum compilation, where the agent must synthesize a target unitary with encoded in an RL-state of size 2 × 2N by sequentially appending gates from a fixed action space defined in Section B.2.1. In every training episode the environment resets to the |0⟩ and a fresh Haar-random 1-qubit target unitary is sampled; the agent acts until the target fidelity tolerance is met or the maximum episode length L is reached. All agents are trained for 5 × 104 episodes and evaluated on 105 independently sampled Haar-random target unitaries. Full hyperparameter details are given in Appendix M. Figure 2: 1-qubit compiling of Haar-random target unitaries with RX, RY, RZ(±π/128) gates. (Left) Success probability and mean fidelity at tolerances 0.99, 0.999, and 0.9999, where ReaPER+ performs best overall. (Right) mean circuit length with std. dev. error bars versus tolerance; although all methods require deeper circuits at higher accuracy and exhibit a similar growth rate with tightening tolerance, ReaPER+ maintains a consistently lower circuit-length offset, giving the best accuracy-length tradeoff across all tolerance levels. Avg. fidelity ↑

ReaPER+ (Ours) PER [33] ReaPER [34] HER [50] PPO [52]

89.30 85.81 85.15 76.49 75.40

0.94 0.94 0.93 0.89 0.88

ReaPER+ (Ours) PER ReaPER HER PPO

85.30 82.35 80.97 74.44 71.60

0.95 0.94 0.93 0.90 0.87

ReaPER+ (Ours) PER ReaPER HER PPO

81.40 72.66 64.35 63.30 69.90

0.95 0.94 0.93 0.90 0.87

Method

0.99

0.999

0.9999

120

HER PER ReaPER

100 Gate count

Success (%) ↑

Tol

PPO ReaPER+

80 60 40 20 0 0.990

0.992

0.994 0.996 Fidelity

0.998

1.000

4.1

Quantum compiling

We evaluate replay-buffer design in quantum compiling, where the agent synthesizes a target unitary by sequentially appending gates from the action spaces defined in Section B.2.1. All agents are trained for 5 × 104 episodes and evaluated on 105 Haar-random target unitaries (hyperparameters in Appendix M). 1-qubit compiling. We evaluate two settings: (i) the small-rotation basis RX, RY, RZ(±π/128) and (ii) the discrete HRC gate set [53], following Ref. [16]. Figure 2 reports success probability and fidelity over tolerances 0.99-0.9999 (40 seeds). ReaPER+ achieves the highest success at every tolerance (89.30%, 85.30%, 81.40%), outperforming PER (85.81%, 82.35%, 72.66%) and fixed ReaPER (85.15%, 80.97%, 64.35%), while showing the slowest circuit-length growth, yielding the best success-fidelity-length tradeoff overall. For the HRC basis, Table 4 shows ReaPER+ reaches 100% success with mean fidelity 0.995 and shortest circuits (14.30 ± 7.89 gates) at 1.56 × 106 steps, ∼ 24% faster than fixed ReaPER, ∼ 26% faster than PER, and ∼ 72% faster than our tuned HER baseline (k=5 relabelings, 100-episode warm-up), which despite being stronger than the original [16] still plateaus at 95% success. All subsequent “HER” references denote this tuned implementation. 2-qubit compiling. Using target unitaries sampled via Algorithm 1, we test whether agents can approximate a ZZ(π) gate at fidelity threshold 0.9914, matching the benchmark of Ref. [16]. Table 1 shows ReaPER+ reaches fidelity 0.9920 in only 2.5 × 104 episodes, a 4× reduction over fixed ReaPER, PER, and HER (all at 105 episodes), and a 32× reduction over PPO [16] (8 × 105 episodes). Although fixed ReaPER attains a marginally higher best fidelity (0.9931) given its longer budget, ReaPER+ reaches a comparable fidelity regime far more efficiently, the operationally relevant metric when quantum-classical simulation time is limited. Table 1: ReaPER+ outperforms all baselines on ZZ(π) approximation using 2-qubit gates. ReaPER+ matches or exceeds fidelity at 32× fewer episodes than PPO [16]. Method ReaPER+ (Ours) ReaPER [34] PPO (Moro et al. [16]) PER [33] HER [50]

4.2

Episodes ↓ 2.5 × 10 105 8 × 105 105 105

Min. gates ↓

Best fidelity ↑

123 126 122 127 NA

0.9920 0.9931 0.9914 0.9918 < 0.9914

Quantum architecture search

We evaluate replay buffers in QAS, where the agent seeks a parameterized circuit that prepares the ground state of a target molecular Hamiltonian. Following CRLQAS [19, 14], we use the gate set in Eq. 19 and reward in Eq. 22, benchmarking molecular ground-state preparation tasks of increasing scale (molecule geometries and hyperparameters in Appendices J and K respectively). Figure 3 compares replay strategies on the smaller-scale systems. ReaPER+ achieves the lowest energy error among prioritized methods with competitive circuit compactness; fixed ReaPER (ω=0.4 for BEH2 , ω=0.6 for H2 O) produces the most compact circuits, reflecting the longer-horizon credit assignment at larger scale (full ω sensitivity in Appendix I). Uniform replay yields superficially shorter circuits but at substantially higher energy error, indicating early trapping in local minima. As shown in Table 2, OptCRLQAS with ReaPER+ (denoted as “OptCRLQAS + ReaPER+”) achieves the lowest energy error across 5-, 6-, and 8-qubit QAS problems, outperforming non-RL baselines such as DQAS [36], GQAS [54], TF-QAS [55], SA-QAS [56], and quantumDARTS [37] while using competitive or fewer gates. For the 5-qubit problem, we target the ground-state energy of the Heisenberg model (Hamiltonian given in Appendix J.2). Since this problem is well-suited to the gateset {RXX(θxx ), RYY(θyy ), RZZ(θzz ), RX(θx ), RY(θy ), RZ(θz )}, we employ an alternative RL state encoding described in Appendix L. At 12-qubit scale, standard CRLQAS is prohibitively expensive since every step triggers a full variational optimization. OptCRLQAS amortizes this cost over m steps, reducing evaluations per episode 6

Table 2:

OptCRLQAS + ReaPER+ vs. non-RL baselines. Our method achieves the lowest energy error across all systems while using competitive or fewer gates. The 5-qubit Heisenberg model is described in Appendix J.2. For 5-Heisenberg we utilize an action space consists of {XX(θxx ), YY(θyy ), ZZ(θzz ), RX(θx ), RY(θy ), RZ(θz )} gates. Problem

Method

Min error (Ha)

Total gates

CNOT

−4

5-Heisenberg

OptCRLQAS + ReaPER+ (ours) DQAS [36] GQAS [54] TF-QAS [55]

5.9 × 10 1.1 × 10−1 7.1 × 10−4 1.2 × 10−3

41 35 35 35

NA NA NA NA

6-BEH2

OptCRLQAS + ReaPER+ (ours) TF-QAS [55] SA-QAS [56]

5.8 × 10−5 1.8 × 10−3 5.6 × 10−3

54 57 73

12 NA 45

8-H2 O

OptCRLQAS + ReaPER+ (ours) quantumDARTS [37] SA-QAS [56]

1.2 × 10−4 1.7 × 10−4 2.6 × 10−3

134 219 95

52 68 69

Figure 3: Replay-buffer design controls circuit compactness in QAS. For 6-BEH2 and 8-H2 O, ReaPER+ variants yield the lowest total, CNOT, and rotation gate counts compared to PER and uniform replay (mean ± std over seeds). ω=0 recovers PER; ω=1 gives fully reliability-adjusted replay. CNOT

ROT

6-BeH2

8-H2O

150 100 50

ill a Va n

0. R

(ω

+ R

ill a

Va n

R E

0. = (ω

Total gate/CNOT/ROT count

Total gates

from T to ⌈T /m⌉ and cutting average wall-clock time by 67.5% (∼3×) without degrading solution quality (Fig. 4). Batching m edits also yields a more separable learning signal by crediting meaningful architectural blocks rather than nearly indistinguishable single-step edits. Figure 4: Efficiency and performance on 12-qubit H2 O. (Left) OptCRLQAS reduces wall-clock time per episode by 67.5% over CRLQAS [19]. (Right) ReaPER achieves the lowest minimum energy error and fastest convergence across all replay baselines. Avg. time per episode (s)

400

200

67.5% lower time

100

Min err. ↓

Method

300

CRLQAS

ReaPER ReaPER+ PER Vanilla

−2

1.7 × 10 2.3 × 10−2 2.5 × 10−2 2.5 × 10−2

Gates ↓

CNOT ↓

Steps ↓

196 241 121 151

109 94 66 91

1.6 × 104 3.7 × 104 9.2 × 104 1.7 × 104

OptCRLQAS

With OptCRLQAS enabling 12-qubit training, ReaPER attains the lowest minimum energy error (1.7 × 10−2 Ha) in the fewest steps (1.6 × 104 ), while PER reaches a similar floor at 5.7× the cost 7

and vanilla replay converges rapidly to a worse solution. Appendix H further shows OptCRLQAS can reduce quantum simulation time by up to 89% and classical optimization time by up to 85% in matched comparisons. 4.3

Noise-robust learning through buffer transfer

We evaluate the transfer scheme of Section 3, in which a buffer Bsrc collected in a noiseless source environment initializes training in a related noisy target, to our knowledge, the first demonstration that a lightweight, weight-free replay buffer alone suffices to transfer noiseless experience to realistic depolarizing-noise settings, with advantages that grow with system size up to 12-qubit. All experiments use OptCRLQAS with uniform replay to isolate the effect of transfer. We study (i) noiseless-to-noiseless and (ii) noiseless-to-noisy transfer (depolarizing noise with single-qubit strength p1 , two-qubit strength p2 ) on molecular ground-state preparation benchmarks of increasing scale. The source buffer is obtained by training a vanilla agent for a fixed 12 GPU-hour budget; to exploit the warm start, we reduce initial exploration from ϵ=1.0 to 0.55 and tighten the curriculum (Appendix K). We quantify transfer via a multi-objective score S = w1 ∆steps + w2 ∆ROT + w3 ∆CNOT + w4 ∆err (weights 0.4, 0.1, 0.2, 0.3), measuring relative improvement over the no-transfer baseline in steps to chemical accuracy, rotation and CNOT gate counts, and best energy error [57]. Figure 5: Weighted transfer matrix for BEH2 under noiseless and noisy transfer. Buffer transfer reduces steps to chemical accuracy by 47-58% and improves final energy by up to 90.2% across all noise settings, yielding composite scores of 19.2-35.8%. The strongest score (35.8%) is driven by the largest energy improvement at p2 =0.001. Transfer matrix (6 − BeH2 ): w1 =0.4, w2 =0.1, w3 =0.2, w4 =0.3 53.0%

-28.6%

26.3%

0.5%

23.8%

50 p2 =0.001

57.7%

-38.1%

-52.6%

90.2%

35.8%

0 p2 =0.005

57.2%

-14.3%

31.6%

6.5%

29.7%

p1 =0.001, p2 =0.005

47.1%

-19.0%

10.5%

0.3%

19.2%

∆steps

∆ROT

∆CNOT

∆err

Score

−50

Relative improvement (%)

Noiseless

Figure 6: Weighted transfer matrix for H2 O under noiseless and noisy transfer. Step reductions range from 49.8% to 84.8%, and energy improvements reach 46.7% under combined noise (p1 =0.001, p2 =0.005), yielding the highest score of 28.7%.

Noiseless

61.1%

-25.0%

-20.0%

9.8%

20.9%

p2 =0.001

49.8%

-19.3%

-19.5%

27.0%

22.2%

p1 =0.001, p2 =0.005

84.8%

-40.7%

-75.7%

46.7%

28.7%

−50

∆steps

∆ROT

∆CNOT

∆err

Score

Relative improvement (%)

Transfer matrix (8 − H2 O): w1 =0.4, w2 =0.1, w3 =0.2, w4 =0.3

Across all systems and noise settings, buffer transfer consistently reduces steps to chemical accuracy and improves final energy error. At the smallest scale (Figure 5), step reductions of 47-58% 8

Figure 7: Weighted transfer matrix for 12-qubit H2 O under noiseless and noisy transfer. Transfer reduces steps to achieve a similar accuracy as non-transfer by 88.2% and CNOT count by 57.6%, under combined depolarizing noise (p1 =0.001, p2 =0.005).

Noiseless

33.8%

13.6%

1.2%

14.9%

19.6%

50 0

p1 =0.001, p2 =0.005

88.2%

41.9%

57.6%

0.0%

51.0%

∆steps

∆ROT

∆CNOT

∆err

Score

−50

Relative improvement (%)

Transfer matrix (12 − H2 O): w1 = 0.4, w2 = 0.1, w3 = 0.2, w4 = 0.3

and energy improvements of up to 90.2% yield composite scores of 19.2-35.8%. At 8-qubit scale (Figure 6), step reductions reach 84.8% under combined noise. Crucially, as shown in Figure 7, the transfer advantage strengthens with system size: at 12-qubit under combined depolarizing noise, buffer transfer reduces steps by 88.2% and achieves the highest composite score of 51.0% across all benchmarks, demonstrating that lightweight replay-buffer transfer is not only effective but becomes more beneficial as the quantum system grows. All gains require no architectural changes, no reward relabeling, and no network weight sharing. The noiseless and noisy tasks share identical state and action spaces, so their optimal value functions differ only by a bounded perturbation. Classical transfer analyses [58] show that high-value source trajectories remain informative under modest dynamics shifts; recent Bellman-alignment theory [59] formalizes this, proving that small one-step Bellman mismatch suffices for provable sample-complexity reduction via source replay, corrected online. 4.4

ReaPER+ generalizes beyond quantum domain

To confirm that the annealing mechanism of ReaPER+ is not specific to quantum reward structure, we benchmark all three replay strategies on LunarLander-v3 [60] using identical DQN agents and the same ω schedule (ωmin =0.1 → ωmax =0.7, Tann =5×104 steps; full details in Appendix E). ReaPER+ achieves a 9% gain in normalized cumulative-return AUC over both baselines and reaches first solve in 9.3% fewer environment steps than fixed ReaPER (3.8 × 105 vs. 42 × 105 ), while sustaining a higher asymptotic success rate (60% vs. 50-55% at episode 4500), confirming that the PER → ReaPER annealing principle generalizes across reward regimes.

Conclusion

We introduced a replay-buffer engineering framework for quantum circuit optimization built on three complementary components: ReaPER+, which improves sample efficiency and circuit compactness through annealed PER → ReaPER prioritization, OptCRLQAS, which cuts wall-clock time per episode by up to 67.5% by amortizing expensive quantum-classical evaluations and lightweight buffer transfer, which reduces steps to chemical accuracy by up to 88% and improves final energy error by up to 90% over from-scratch noisy baselines, without network-weight sharing or reward relabeling. Across quantum compiling and QAS benchmarks, these results show that how experience is stored, sampled, and transferred is a primary algorithmic lever for scalable, noise-robust quantum circuit optimization. A classical validation on LunarLander-v3 further confirms that ReaPER+’s annealing principle is domain-agnostic, yielding a +9% AUC improvement over PER and fixed ReaPER with identical agents and schedules as shown in Appendix E.

References [1] Amira Abbas, Andris Ambainis, Brandon Augustino, Andreas Bärtschi, Harry Buhrman, Carleton Coffrin, Giorgio Cortiana, Vedran Dunjko, Daniel J Egger, Bruce G Elmegreen, et al. 9

Challenges and opportunities in quantum optimization. Nature Reviews Physics, 6(12):718– 735, 2024. [2] Edward Farhi, Jeffrey Goldstone, and Sam Gutmann. A quantum approximate optimization algorithm. arXiv preprint arXiv:1411.4028, 2014. [3] Michel H Devoret and Robert J Schoelkopf. Superconducting circuits for quantum information: an outlook. Science, 339(6124):1169–1174, 2013. [4] Xanthe Croot, Kasra Nowrouzi, Christopher Spitzer, Carmen G Almudever, Alexandre Blais, Malcolm Carroll, Jerry Chow, Daniel Friedman, Masao Tokunari, Edoardo Charbon, et al. Enabling technologies for scalable superconducting quantum computing. arXiv preprint arXiv:2512.15001, 2025. [5] Morten Kjaergaard, Mollie E Schwartz, Jochen Braumüller, Philip Krantz, Joel I-J Wang, Simon Gustavsson, and William D Oliver. Superconducting qubits: Current state of play. Annual Review of Condensed Matter Physics, 11(1):369–395, 2020. [6] Abhinav Kandala, Antonio Mezzacapo, Kristan Temme, Maika Takita, Markus Brink, Jerry M Chow, and Jay M Gambetta. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. nature, 549(7671):242–246, 2017. [7] Youngseok Kim, Andrew Eddins, Sajant Anand, Ken Xuan Wei, Ewout Van Den Berg, Sami Rosenblatt, Hasan Nayfeh, Yantao Wu, Michael Zaletel, Kristan Temme, et al. Evidence for the utility of quantum computing before fault tolerance. Nature, 618(7965):500–505, 2023. [8] John Preskill. Quantum computing in the nisq era and beyond. Quantum, 2:79, 2018. [9] Kishor Bharti, Alba Cervera-Lierta, Thi Ha Kyaw, Tobias Haug, Sumner Alperin-Lea, Abhinav Anand, Matthias Degroote, Hermanni Heimonen, Jakob S Kottmann, Tim Menke, et al. Noisy intermediate-scale quantum algorithms. Reviews of Modern Physics, 94(1):015004, 2022. [10] Austin G Fowler, Matteo Mariantoni, John M Martinis, and Andrew N Cleland. Surface codes: Towards practical large-scale quantum computation. Physical Review A—Atomic, Molecular, and Optical Physics, 86(3):032324, 2012. [11] Tobias V Forster, Nils Quetschlich, and Robert Wille. Quantum circuit optimization for the fault-tolerance era: Do we have to start from scratch? In 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), volume 1, pages 584–590. IEEE, 2025. [12] Zoltán Zimborás, Bálint Koczor, Zoë Holmes, Elsi-Mari Borrelli, András Gilyén, Hsin-Yuan Huang, Zhenyu Cai, Antonio Acı́n, Leandro Aolita, Leonardo Banchi, et al. Myths around quantum computation before full fault tolerance: What no-go theorems rule out and what they don’t. arXiv preprint arXiv:2501.05694, 2025. [13] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. [14] Mateusz Ostaszewski, Lea M Trenkwalder, Wojciech Masarczyk, Eleanor Scerri, and Vedran Dunjko. Reinforcement learning for optimization of variational quantum circuit architectures. Advances in neural information processing systems, 34:18182–18194, 2021. [15] Marin Bukov and Florian Marquardt. Reinforcement learning for quantum technology. arXiv preprint arXiv:2601.18953, 2026. [16] Lorenzo Moro, Matteo GA Paris, Marcello Restelli, and Enrico Prati. Quantum compiling by deep reinforcement learning. Communications Physics, 4(1):178, 2021. [17] ZT Wang, Qiuhao Chen, Yuxuan Du, ZH Yang, Xiaoxia Cai, Kaixuan Huang, Jingning Zhang, Kai Xu, Jun Du, Yinan Li, et al. Quantum compiling with reinforcement learning on a superconducting processor. arXiv preprint arXiv:2406.12195, 2024. [18] Akash Kundu. Reinforcement learning-assisted quantum architecture search for variational quantum algorithms. arXiv preprint arXiv:2402.13754, 2024. 10

[19] Yash J. Patel, Akash Kundu, Mateusz Ostaszewski, Xavier Bonet-Monroig, Vedran Dunjko, and Onur Danaci. Curriculum reinforcement learning for quantum architecture search under hardware errors. In The Twelfth International Conference on Learning Representations, 2024. [20] Ryan Sweke, Markus S Kesselring, Evert PL van Nieuwenburg, and Jens Eisert. Reinforcement learning decoders for fault-tolerant quantum computation. Machine Learning: Science and Technology, 2(2):025005, 2021. [21] Kevin Reuer, Jonas Landgraf, Thomas Fösel, James O’Sullivan, Liberto Beltrán, Abdulkadir Akin, Graham J Norris, Ants Remm, Michael Kerschbaum, Jean-Claude Besse, et al. Realizing a deep reinforcement learning agent for real-time quantum feedback. Nature Communications, 14(1):7138, 2023. [22] Tyler LeBlond, Christopher Dean, George Watkins, and Ryan Bennink. Realistic cost to execute practical quantum circuits using direct clifford+ t lattice surgery compilation. ACM Transactions on Quantum Computing, 5(4):1–28, 2024. [23] Marco Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, et al. Variational quantum algorithms. Nature Reviews Physics, 3(9):625–644, 2021. [24] Akash Kundu and Stefano Mangini. Tensorrl-qas: Reinforcement learning with tensor networks for improved quantum architecture search. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [25] Remmy Zen, Jan Olle, Luis Colmenarez, Matteo Puviani, Markus Müller, and Florian Marquardt. Quantum circuit discovery for fault-tolerant logical state preparation with reinforcement learning. Physical Review X, 15(4):041012, 2025. [26] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [27] Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations, 2025. [28] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In 5th Annual Conference on Robot Learning, 2021. [29] Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, Philip S. Yu, and Yi Chang. Sample efficient offline-to-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 36(3):1299–1310, 2024. [30] Chihyeon Song, Jaewoo Lee, and Jinkyoo Park. Adaptive replay buffer for offline-to-online reinforcement learning, 2025. [31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [32] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016. [33] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. [34] Leonard S. Pleiss, Tobias Sutter, and Maximilian Schiffer. Reliability-adjusted prioritized experience replay. In The Fourteenth International Conference on Learning Representations, 2026. [35] Harper R Grimsley, Sophia E Economou, Edwin Barnes, and Nicholas J Mayhall. An adaptive variational algorithm for exact molecular simulations on a quantum computer. Nature communications, 10(1):3007, 2019. 11

[36] Shi-Xin Zhang, Chang-Yu Hsieh, Shengyu Zhang, and Hong Yao. Differentiable quantum architecture search. Quantum Science & Technology, 7(4):045023, 2022. [37] Wenjie Wu, Ge Yan, Xudong Lu, Kaisen Pan, and Junchi Yan. Quantumdarts: differentiable quantum architecture search for variational quantum algorithms. In International conference on machine learning, pages 37745–37764. PMLR, 2023. [38] Riccardo Rasconi and Angelo Oddi. An innovative genetic algorithm for the quantum circuit compilation problem. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7707–7714, 2019. [39] Leo Sünkel, Darya Martyniuk, Denny Mattern, Johannes Jung, and Adrian Paschke. Ga4qco: genetic algorithm for quantum circuit optimization. arXiv preprint arXiv:2302.01303, 2023. [40] Kim Nicoli, Christopher J Anders, Lena Funcke, Tobias Hartung, Karl Jansen, Stefan Kühn, Klaus-Robert Müller, Paolo Stornati, Pan Kessel, and Shinichi Nakajima. Physics-informed bayesian optimization of variational quantum circuits. Advances in Neural Information Processing Systems, 36:18341–18376, 2023. [41] Peiyong Wang, Muhammad Usman, Udaya Parampalli, Lloyd CL Hollenberg, and Casey R Myers. Automated quantum circuit design with nested monte carlo tree search. IEEE Transactions on Quantum Engineering, 4:1–20, 2023. [42] Shi-Xin Zhang, Chang-Yu Hsieh, Shengyu Zhang, and Hong Yao. Neural predictor based quantum architecture search. Machine Learning: Science and Technology, 2(4):045027, 2021. [43] Yuxuan Du, Tao Huang, Shan You, Min-Hsiu Hsieh, and Dacheng Tao. Quantum circuit architecture search for variational quantum algorithms. npj Quantum Information, 8(1):62, 2022. [44] Thomas Fösel, Murphy Yuezhen Niu, Florian Marquardt, and Li Li. Quantum circuit optimization with deep reinforcement learning. arXiv preprint arXiv:2103.07585, 2021. [45] En-Jui Kuo, Yao-Lung L Fang, and Samuel Yen-Chi Chen. Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.07715, 2021. [46] Akash Kundu, Aritra Sarkar, and Abhishek Sadhu. Kanqas: Kolmogorov-arnold network for quantum architecture search. EPJ Quantum Technology, 11(1):76, 2024. [47] David Kremer, Victor Villar, Hanhee Paik, Ivan Duran, Ismael Faro, and Juan Cruz-Benito. Practical and efficient quantum circuit synthesis and transpiling with reinforcement learning. arXiv preprint arXiv:2405.13196, 2024. [48] Akash Kundu and Leopoldo Sarra. Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware. Communications Physics, 2026. [49] Akash Kundu. awesome-QAS: A curated list of resources for quantum architecture search, June 2025. [50] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017. [51] Heng You, Tianpei Yang, Yan Zheng, Jianye Hao, E Taylor, et al. Cross-domain adaptive transfer reinforcement learning based on state-action correspondence. In Uncertainty in Artificial Intelligence, pages 2299–2309. PMLR, 2022. [52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [53] Aram W Harrow, Benjamin Recht, and Isaac L Chuang. Efficient discrete approximations of quantum gates. Journal of Mathematical Physics, 43(9):4445–4451, 2002. 12

[54] Zhimin He, Jiachun Wei, Chuangtao Chen, Zhiming Huang, Haozhen Situ, and Lvzhou Li. Gradient-based optimization for quantum architecture search. Neural Networks, 179:106508, 2024. [55] Zhimin He, Maijie Deng, Shenggen Zheng, Lvzhou Li, and Haozhen Situ. Training-free quantum architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12430–12438, 2024. [56] Xudong Lu, Kaisen Pan, Ge Yan, Jiaming Shan, Wenjie Wu, and Junchi Yan. Qas-bench: rethinking quantum architecture search and a benchmark. In International conference on machine learning, pages 22880–22898. PMLR, 2023. [57] Azhar Ikhtiarudin, Aditi Das, Param Thakkar, and Akash Kundu. Benchrl-qas: Benchmarking reinforcement learning algorithms for quantum architecture search. In Proceedings of the AAAI Symposium Series, volume 7, pages 358–367, 2025. [58] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009. [59] Jinhang Chai, Enpei Zhang, Elynn Chen, and Yujun Yan. Optimistic transfer under task shift via bellman alignment. arXiv preprint arXiv:2601.21924, 2026. [60] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024. [61] Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. In The Thirteenth International Conference on Learning Representations, 2025. [62] Francisco JR Ruiz, Tuomas Laakkonen, Johannes Bausch, Matej Balog, Mohammadamin Barekatain, Francisco JH Heras, Alexander Novikov, Nathan Fitzpatrick, Bernardino RomeraParedes, John Van De Wetering, et al. Quantum circuit optimization with alphatensor. Nature Machine Intelligence, 7(3):374–385, 2025. [63] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016. [64] Michael JD Powell. A direct search optimization method that models the objective and constraint functions by linear interpolation. In Advances in optimization and numerical analysis, pages 51–67. Springer, 1994.

Limitations and future work

Limitations. All quantum experiments use a fixed DQN/DDQN backbone; whether ReaPER+’s annealing advantage persists under policy-gradient or actor-critic agents remains an open question. The replay-buffer transfer scheme assumes shared state and action spaces between source and target tasks, which may not hold when porting across substantially different hardware topologies or gate sets. Future work. Immediate extensions include: (i) combining OptCRLQAS with tensor-network warm-starts [24] to further reduce search cost at 20-qubit scale; (ii) hardware-aware replay, where buffered circuits are ranked using limited QPU evaluations before transfer to realistic noise models; and (iii) prioritized generative replay [61] to synthesize high-value transitions for richer buffer initialization under depolarizing noise.

Preliminaries

Here we discuss the preliminaries based on which the methods (in Section 3) are based on. This discussion includes a through description of the state-of-the-art replay buffer. Also t includes the encoding methods through which we represent the action space, reward functions and the RL-state. B.1

Replay buffers in deep reinforcement learning

Experience replay is a central component of off-policy deep RL because it determines which past transitions are revisited during optimization. Uniform replay [31], hindsight experience replay [50], prioritized experience replay [33], and more recent reliability-aware variants [34] provide increasingly structured ways to trade off coverage, informativeness, and target reliability. In quantum optimization, however, replay buffers have largely been treated as fixed implementation choices rather than as a primary object of algorithmic design. To formalize the replay mechanisms studied in this work, we consider an episodic reinforcementlearning setting. An episode is a sequence D = {Ct }nt=1 of transitions, where each transition Ct = (St , At , Rt , St+1 , dt ) records the state St ∈ S, action At ∈ A, scalar reward Rt , next state St+1 , and terminal flag dt ∈ {0, 1}, which equals 1 if the episode ends at step t and 0 otherwise. The agent maintains an online Q-network Qθ with parameters θ and a periodically updated target network Qθ̄ with parameters θ̄, and γ ∈ (0, 1] denotes the discount factor. For each stored transition, we define four quantities that replay strategies use to assign sampling priorities. The TD target Yt is the bootstrapped return estimate used to update Qθ : Yt = Rt + γ(1 − dt ) max Qθ̄ (St+1 , a′ ). ′ a

(5)

The TD error δt measures how far the current value estimate deviates from this target: δt = Yt − Qθ (St , At ),

(6)

and its magnitude δt+ = |δt | serves as a proxy for how much a transition still has to teach the agent. The true value error et measures the deviation of the current estimate from the optimal Q-function Q⋆ : et = Qθ (St , At ) − Q⋆ (St , At ). (7)

Finally, the target bias εt captures the error introduced by the target network itself: εt = Qθ̄ (St , At ) − Q⋆ (St , At ).

(8)

Replay strategies differ in how they assign sampling preference to stored transitions. For each tranPN sition Ci in the replay buffer, let Ψi denote its unnormalized replay priority and µi = Ψi / j=1 Ψj the corresponding sampling probability. Different replay rules are then specified by different choices of Ψi . Uniform replay (Vanilla). A uniform replay buffer samples all stored transitions with equal probability. If the current buffer contains {Ci }N i=1 , then 1 (9) Ψuni = 1, µuni = . i i N 14

Hindsight experience replay (HER). HER augments the replay buffer by relabeling goals from achieved future states within the same trajectory [50]. The key idea is that even a failed trajectory, one that did not reach the intended goal g, can be treated as a successful one with respect to a different goal g ′ , typically the state actually reached. Concretely, given a goal-conditioned transition (St , At , Rt , St+1 , dt , g), HER adds an additional relabeled copy to the buffer, St , At , Rt′ (g ′ ), St+1 , d′t (g ′ ), g ′ , (10) where g ′ is the relabeled goal and Rt′ (g ′ ), d′t (g ′ ) are the reward and terminal flag recomputed with respect to g ′ . Unless stated otherwise, HER samples uniformly from the augmented buffer. HER is utilized in the quantum compilation task in Ref. [16]. Prioritized experience replay (PER). error [33]:

PER samples transitions according to their absolute TD

ΨPER = (δi+ )α , i

ΨPER i , µPER = P i N PER j=1 Ψj

(11)

where α > 0 controls the strength of prioritization. Reliability-adjusted prioritized experience replay (ReaPER). ReaPER was recently introduced in ref. [34]. It discounts transitions whose targets are unreliable due to large downstream TD errors. For an episode D = {Ct }nt=1 , the reliability score is Pn δi+ Rt = 1 − Pi=t+1 (12) n + , i=1 δi and the corresponding priorities are ΨReaPER = Rtω (δt+ )α , t

ΨReaPER µReaPER = Pn t ReaPER , t i=1 Ψi

(13)

where ω ∈ [0, 1] controls the strength of reliability weighting (ω = 0 recovers PER and ω = 1 corresponds to fully reliability-adjusted replay). Thus, relative to PER, high-error and low-reliability transitions are down-weighted. B.2

Reinforcement learning for quantum optimization

Reinforcement learning has been applied to both quantum compilation and parameterized quantum circuit design. Existing work includes RL-based gate-sequence synthesis [62, 44, 47], architecture search for molecular ground-state preparation [14], and hardware-aware variational circuit construction [19], demonstrating that RL can discover nontrivial circuit structures across a range of quantum tasks. At the same time, prior methods typically operate under substantial sample and compute demands, especially when each environment step requires an expensive variational optimization or when the target setting includes hardware noise. To tackle this, we formulate both quantum compilation and quantum architecture search as Markov decision processes in which an agent sequentially constructs a quantum circuit. At time step t, the agent observes St ∈ S, selects At ∈ A, receives reward Rt , and transitions to St+1 . Episodes terminate when a target threshold is reached or when a maximum circuit length is exceeded. B.2.1

RL-state and action for quantum compilation

In quantum compilation, the agent aims to approximate a target unitary Utar by appending gates ⊗n from a finite base G. Following Ref. [16], the system is initialized in the all-zero state |0⟩ , and Qt ⊗n the circuit unitary after t steps is Ut = j=1 Aj , so that the prepared state at step t is Ut |0⟩ . The observation is based on the residual unitary Ot = Ut† Utar , whose real and imaginary entries are provided to the agent. 15

(14)

1-qubit compiling

For 1-qubit compiling, the chosen gateset base is n π n π oo (1) , (15) Grot = Ri (θj ) i ∈ {x, y, z}, θj ∈ + ,− 128 128 where Ri (θj ) denotes a single-qubit rotation by angle θj about axis i. The rotation angle is thus restricted to a binary set of small fixed increments ±π/128, making each action a minimal discrete rotation rather than a continuous parameter choice. In an alternate setting, we (HRC)[53] efficient universal also consider the Harrow-Recht-Chuang 1 2i 1 2 1 + 2i 0 1 1 1 basis, V1 = √5 , V 2 = √5 , V 3 = √5 , with action space 2i 1 −2 1 0 1 − 2i (1)

AHRC = {V1 , V2 , V3 }.

(16)

2-qubit compiling. (2)

A B.2.2

For the 2-qubit setting, we use the action space: π π ,Y Y = Y Y ± }, = {Rz (θj ) ⊗ I, I ⊗ Rz (θj ), XX = XX ± 128 128

(17)

RL-state and action for quantum architecture search

For quantum architecture search (QAS), the agent searches over parameterized quantum circuits to minimize a task-dependent cost, typically the expectation value [23]. Following CRLQAS [19], the state is represented by a tensor-based binary encoding of the circuit: SQAS = (Et , ct ) Et ∈ {0, 1}Ndepth ×Nwires ×Ngate type , ct ∈ R , (18) where Et encodes the circuit structure and ct summarizes its current performance by calculating the current cost function i.e. the expectation value of Hamiltonian. Using the gate set GQAS = {RX(θx ), RY(θy ), RZ(θz ), CNOT}, the action space consists of placing a gate at a particular circuit location. A compact description is AQAS = a = (ℓ, i, g) ℓ ∈ {1, . . . , Nlayers }, i ∈ {1, . . . , Nwires }kg , g ∈ GQAS , (19) where ℓ is the layer index, i specifies the qubit(s) the gate acts on, and kg is the number of qubits required by gate g which isequal to 1 for single-qubit gates {RX, RY, RZ} and 2 for the two-qubit gate CNOT. Illegal actions are masked out during training. B.2.3

Reward structure

In both tasks, an episode ends either when a target threshold is met or when the maximum number of steps is reached. quantum compilation. Following Ref. [16], we use two reward designs depending on the gate base. For bases small-rotation (i.e ±π/128), we use the dense reward ( (L − t) + 1, if d(Ut , Utar ) < ε, comp rt = (20) −d(Ut , Utar )/L, otherwise, where L is the maximum episode length and ε is the target accuracy threshold. For discrete bases such as HRC, we use the sparse reward ( 0, if d(Ut , Utar ) < ε, comp rt = (21) −1/L, otherwise. Quantum architecture search. Here we use the reward used as in [14, 19]. Let Ct be the optimized variational cost at step t, ξ the target threshold, Cmin the desired minimum cost, and Tse the maximum number of steps in episode e. Then   5, if Ct < ξ,     −5, if t ≥ Tse and Ct ≥ ξ, (22) RtQAS =   C − C  t−1 t  , −1 , otherwise. max Ct−1 − Cmin This reward provides a positive terminal signal for success, a negative terminal signal for failure, and otherwise a normalized improvement-based intermediate reward. 16

Justification for ReaPER+

Here we provide an argument on why the proposed ReaPER+ replay rule can be expected to outperform a fixed replay strategy over the full course of training. Recall that ReaPER+ defines the replay priority of transition t at training step τ as (+,τ )

(+,τ )

Ψt

(+,τ )

= Rtωτ (δt+ )α ,

µt

Ψ = P t (+,τ ) , i Ψi

(23)

where δt+ = |δt | is the absolute temporal-difference error, Rt ∈ [0, 1] is the reliability score, α > 0 is the prioritization exponent, and ωτ ∈ [0, 1] is a non-decreasing annealing parameter. When ωτ = 0, the scheme reduces to PER, whereas ωτ = 1 recovers ReaPER. Proposition 1. Assume that: (i) larger reliability Rt corresponds to smaller bias in the TD target of transition t; (ii) early in training, reliability estimates are noisy and only weakly correlated with true target quality; and (iii) later in training, reliability estimates become more informative. Then a schedule ωτ that starts near 0 and increases toward 1 induces a replay distribution that is better aligned with the needs of the learning process than either fixed PER [33] or ReaPER [34] used throughout training. Justification. ReaPER+ is

Consider two transitions i and j.

Their relative sampling probability under !α ω (ω) Riω (δi+ )α Ri µi δi+ = ω + α = . (24) (ω) Rj Rj (δj ) δj+ µj

If Ri > Rj , then for any ω2 > ω1 we have (ω )

(ω )

µi 2

µi 1 > . (ω ) (ω ) µj 2 µj 1

(25)

Hence increasing ω monotonically shifts replay mass toward transitions with higher reliability while preserving the PER-style dependence on TD error. Figure 8: ReaPER+ progressively concentrates buffer mass toward higher-fidelity transitions (fidelity ≥ 0.95) while retaining broader early-training coverage, consistent with its annealed transition from PER-like exploration to ReaPER-like reliability-aware sampling. PER maintains broader low-fidelity coverage throughout training, while ReaPER shows intermediate concentration behavior. Episode 100

ReaPER+

Episode 2000

Episode 4000

Episode 6000

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

6 4 2 0

PER

6 4 2 0

ReaPER

8 6 4 2

0 0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

Average gate fidelity

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

This monotonicity explains the role of the annealing schedule. At the beginning of training, the value function is inaccurate and the resulting reliability estimates can be unstable. In this regime, setting ωτ ≈ 0 prevents the learner from overcommitting to a noisy reliability signal and recovers 17

the PER [33] behavior that emphasizes large TD-error transitions. Later in training, once the critic becomes more accurate, the reliability score carries more information about whether a large TD error corresponds to a genuinely useful correction or to a misleading target. Increasing ωτ then down-weights high-error but low-reliability transitions and shifts the replay distribution toward more trustworthy updates. Equivalently, one may examine the log-priority (+,τ )

log Ψt

= ωτ log Rt + α log δt+ .

(26)

The annealing coefficient ωτ controls the contribution of log Rt relative to log δt+ . Early in training, the replay rule is dominated by the TD-error term; later, the reliability term gradually regularizes prioritization by suppressing transitions whose downstream targets are less trustworthy. Therefore, ReaPER+ inherits the strong early-learning behavior of PER and the late-stage stability of ReaPER. The argument above does not claim universal dominance for every environment or every annealing schedule. Rather, it shows that under mild assumptions on the quality of reliability estimates across training, ReaPER+ provides a natural continuation method between two useful replay regimes. As shown in Figure 8, during the 1-qubit compiling task, ReaPER+ progressively shifts the fidelity distribution toward higher-fidelity, more reliable replay samples significantly faster than both PER and ReaPER.

Replay buffer transfer under source-target task similarity

We next justify why replay-buffer transfer from a noiseless source environment to a corresponding noisy target environment can accelerate learning in quantum optimization. Let Msrc and Mtgt denote two Markov decision processes with the same state and action spaces, where the target task differs from the source task by noise in the transition dynamics, reward evaluations, or both. Let Bsrc be a replay buffer collected in the source task, and let the target replay buffer be initialized as (0)

Btgt ← T (Bsrc ),

(27)

where T is a transfer operator that copies compatible transitions from source to target. Proposition 2. Assume that: (i) the source and target tasks share the same state and action spaces; (ii) the target task is a bounded perturbation of the source task in the sense that rewards and transition kernels do not differ arbitrarily; and (iii) high-value trajectories in the source task remain informative, though not necessarily optimal, in the target task. Then initializing the target replay buffer with transferred source transitions can improve early target learning relative to training from an empty buffer. Justification.

Let the source and target Bellman targets for a transition (s, a) be h i ′ ′ Q(s , a ) , Y src (s, a) = rsrc (s, a) + γEs′ ∼P src (·|s,a) max ′

(28)

h i ′ ′ Y tgt (s, a) = rtgt (s, a) + γEs′ ∼P tgt (·|s,a) max Q(s , a ) . ′

(29)

and

If the source and target tasks are sufficiently similar in rewards and transition dynamics, then the induced Bellman targets differ by a bounded amount, as suggested by classical simulation-lemma arguments and recent Bellman-alignment analyses for transfer reinforcement learning [59]. For example, if |rtgt (s, a) − rsrc (s, a)| ≤ εr (30) and the transition mismatch is bounded so that h i h i ′ ′ ′ ′ src max Q(s , a ) EP tgt max Q(s , a ) − E ≤ εp , (31) P ′ ′ a

then

|Y tgt (s, a) − Y src (s, a)| ≤ εr + γεp . (32) Thus, a transition that is informative for the source task remains approximately informative for the target task whenever the source-target shift is moderate. 18

The second in the proposition 2 is improved coverage. At the start of training in the noisy environment, an empty replay buffer contains little information about which gate sequences, partial circuits, or architectural motifs are promising. In contrast, a transferred buffer already contains trajectories concentrated in regions of state-action space that were useful in the source task. Since the noisy and noiseless tasks share the same circuit-building structure, these transitions provide a more informative initial replay distribution than random target experience alone. Consequently, the learner can begin updating from semantically meaningful trajectories before it has independently rediscovered them under noise. Whereas the third mechanism in proposition 2 is self-correction during continued training. As the agent interacts with the target environment, new target-domain transitions are added to the replay buffer and progressively adjust the replay distribution toward the true noisy task. Hence transfer does not lock the agent into the source domain; rather, it supplies a warm start that is gradually refined by on-target experience. In prioritized replay schemes, this correction can be even more effective because target transitions that carry larger TD errors or higher reliability automatically gain greater replay probability. The above argument explains why replay-buffer transfer is especially natural in quantum optimization settings where the noisy and noiseless problems share the same circuit representation, action space, and objective structure, but differ in the stochastic distortion introduced by hardware noise. Under such source-target similarity, transferred source trajectories provide a useful initial memory for the target learner, improving early sample efficiency while still allowing subsequent adaptation to the noisy environment.

Classical RL validation: LunarLander-v3

The three contributions presented in the main text ReaPER+, OptCRLQAS, and replay-buffer transfer, are motivated by and evaluated on quantum circuit optimization tasks. To verify that the annealing schedule underlying Reaper+ is not specific to the sparse, long-horizon reward structure of quantum environments, we provide a supplementary validation on LunarLander-v3 [60], a wellestablished continuous-control benchmark from classical deep RL with dense rewards and a standard solved threshold of +200 cumulative return. We stress that this experiment is not intended as a primary result; its sole purpose is to confirm that the PER→ReaPER annealing idea generalizes beyond the quantum domain. Figure 9: LunarLander-v3 validation of ReaPER+. (Left) rolling success rate (300-episode window). (Middle) ReaPER+ (blue) reaches a higher success rate faster and maintains a higher asymptotic level than fixed ReaPER (red) and PER (green). (Right) normalized cumulative-return AUC. ReaPER+ accumulates +9% more return over the full training run, confirming improved sample efficiency on a dense-reward classical benchmark. All methods use identical DQN agents, only the replay mechanism differs.

1.0 0.8

ReaPER PER ReaPER+

0.6 0.4 ReaPER+ reaches 0.2 0.0 1,000

higher faster

2 × 103

Episode

3 × 103 4 × 103

Normalized return (AUC)

Rolling success (300-ep)

LunarLander-v3: ReaPER+ vs ReaPER vs PER 1.0 0.6 0.4

+9% AUC advantage

0.2 0.0

419k

ReaPER

0.8

380k

ReaPER+

428k

PER 1

100

Episode

1,000

200k

400k

Steps to first solve

600k

We benchmark three replay strategies using identical DQN agents (two hidden layers of 128 units, SELU activations, Table 3): • PER [33]: prioritization by absolute TD error, α=0.6, β0 =0.4 annealed to 1.0 over 5×104 frames. • ReaPER [34]: reliability-adjusted prioritization with fixed ω=0.4, α=0.4. 19

• ReaPER+ (ours): annealed ω schedule ωmin =0.1 → ωmax =0.7 over Tann =5×104 steps (identical schedule to the quantum compiling experiments), α=0.4. All agents are trained for 5,000 episodes on LunarLander-v3 with γ=0.99, ε-greedy exploration (ε0 =1.0 → εmin =0.05, decay 0.9995 per episode), learning rate 10−3 , batch size 64, replay capacity 105 , and target-network synchronization every 10 episodes. Four gradient updates are performed per episode after a 20-episode warm-up. The same agent architecture and training code used for the quantum experiments are employed without modification; only the environment is swapped. Results are reported for 3 different initializations of neural network. Figure 9 summarizes two complementary metrics. Left: the 300-episode rolling success rate (fraction of episodes exceeding the +200 solved threshold). ReaPER+ reaches a higher success rate earlier in training and sustains a higher asymptotic level (≈60% at episode 4500) compared with fixed ReaPER (≈50%) and PER (≈55%). Right: the normalized area under the reward curve (AUC), computed as the per-episode running mean of the cumulative return, shifted and normalized to [0, 1]. ReaPER+ achieves a +9% AUC advantage over both baselines by episode 5000, indicating better sample efficiency throughout training. These results are consistent with the mechanism described in Section 3 and Appendix C: early in training, ω ≈ 0 recovers PER-style exploration; as value estimates stabilize, rising ω down-weights unreliable high-error transitions and concentrates replay on trustworthy updates. This two-phase behavior is beneficial regardless of whether rewards are sparse and episodic (quantum compilation) or dense and continuous (LunarLander), supporting the view that ReaPER+ is a general-purpose annealing strategy rather than a domain-specific heuristic. Table 3: Hyperparameters for the LunarLander-v3 validation experiment. Shared parameters apply to all three replay strategies; method-specific parameters are listed in the lower block. Parameter Shared (all methods) Network architecture Discount γ Learning rate Batch size Replay capacity ε schedule Gradient updates/ep. Training episodes Solved threshold

Value 2 × 128 hidden units, SELU 0.99 10−3 64 105 1.0 → 0.05, decay 0.9995/episode 4 (after 20-episode warm-up) 5000 +200 cumulative return

PER α β0

0.6 0.4, annealed to 1.0 over 5×104 frames

ReaPER α ω β0

0.4 0.4 (fixed) 0.4, annealed to 1.0 over 5×104 frames

ReaPER+ α ωmin ωmax Tann β0

0.4 0.1 0.7 5×104 steps 0.4, annealed to 1.0 over 5×104 frames

1-qubit compiling with HRC gateset

For a direct comparison with Ref. [16], we evaluate the discrete HRC basis [53] using the action space and sparse reward defined in Eq. 16 and Eq. 21. Table 4 shows that ReaPER+ reaches 100% success with a mean fidelity of 0.995 and the shortest average circuit length (14.30 ± 7.89 gates), achieving this at 1.56 × 106 steps. This is slightly earlier than fixed ReaPER (1.60 × 106 steps), 20

clearly earlier than PER (2.10 × 106 steps), and substantially earlier than our tuned HER baseline (5.50 × 106 steps), while also producing shorter circuits than all three. In contrast, the original unopTable 4: 1-qubit compiling of Haar-random targets using the HRC basis. Success rate, mean fidelity, and circuit length for different replay strategies. All methods except the original HER baseline [16] reach 100% success. ReaPER+ yields the shortest circuits (14.30 ± 7.89 gates) and reaches peak performance ∼24%, ∼26%, and ∼72% faster than fixed ReaPER, PER, and tuned HER, respectively. Method ReaPER+ (Ours) ReaPER [34] PER [33] HER (tuned) HER (Moro et al. [16])

Success (%) ↑

Fidelity ↑

Mean length ↓

At step ↓

100.0 100.0 100.0 100.0 95.0

0.995 0.995 0.995 0.992 0.990

14.30 ± 7.89 20.34 ± 16.16 19.54 ± 14.61 28.60 ± 21.58 35

1.56 × 106 1.60 × 106 2.10 × 106 5.50 × 106 –

timized HER implementation from Moro et al. [16] plateaus at 95% accuracy. Note that throughout the remainder of this paper, any reference to the HER baseline indicates our tuned implementation rather than the original formulation of Ref. [16], ensuring all comparisons are made against the strongest possible version of the baseline.

Target dataset for 2-qubit compiling task

For the compiling experiments with the 1-qubit small-rotation basis and the 2-qubit gate set, we generate target unitaries synthetically by sampling random circuits from the corresponding elementary gate library. Concretely, a target unitary is constructed by first sampling a circuit length uniformly at random and then composing gates drawn uniformly from the same basis used by the agent during training. This yields a diverse dataset of reachable targets with varying circuit complexity while ensuring that the target distribution is consistent with the underlying compilation task. For the 2-qubit setting, Algorithm 1 details this procedure for the gate set {XX(±π/128), Y Y (±π/128), Rz (±π/128) ⊗ I, I ⊗ Rz (±π/128)}; the 1-qubit small-rotation targets are generated analogously from their corresponding 1-qubit rotation basis. Algorithm 1: 2-qubit target unitary generation. π π π π Input: Gate set B = XX(± 128 ), Y Y (± 128 ), Rz (± 128 ) ⊗ I, I ⊗ Rz (± 128 ) Output: Target unitary Utar ∈ SU(4) 4 1 Sample circuit length N ∼ Uniform {6, 7, . . . , 10 − 1} ; 2 Initialize Utar ← I4 ; 3 for k = 1 to N do 4 Sample gate Gk ∼ Uniform(B); 5 Utar ← Gk · Utar ; 6 end 7 return Utar

Benchmarking OptCRLQAS against CRLQAS [19]

To empirically validate the computational efficiency gained by amortizing the architecture update overhead, we benchmarked OptCRLQAS against the baseline CRLQAS framework across 6-, 8, and 10-qubit configurations. We tracked both the expectation value evaluation and the classical optimization time over multiple episodes. Figure 10 illustrates the substantial reduction in computational cost when utilizing OptCRLQAS across varying step intervals (m ∈ {3, 5, 7, 10}). Because CRLQAS triggers a full quantum-classical evaluation at every environment step, its runtime scales poorly as circuit complexity increases. By accumulating local architecture edits and restricting full evaluations to every m steps, OptCRLQAS closely matches the expected runtime reduction factor of ≈ 1/m. Our empirical results demonstrate an aggressive decline in runtime as 21

Figure 10: Comparative runtime analysis of the baseline CRLQAS versus OptCRLQAS across 6-, 8-,

Classical opt. time (s)

Quantum sim. time (s)

and 10-qubit tasks. The left panel shows the quantum energy evaluation time (in seconds), and the right panel displays the classical optimization time (in seconds). Results for OptCRLQAS are reported for different replay buffer sizes (m ∈ {3, 5, 7, 10}). Error bars denote the standard deviation measured over 3 to 5 independent episodes. The utilization of OptCRLQAS yields an average runtime reduction of up to 89% for quantum evaluations and 85% for classical optimizations at m = 10.

6-qubit 8-qubit 10-qubit

500 400 300 200 100 0

) ) 0) =5 =7 =1 L ( (m (m R m S S S ( C QA QA QA AS RL RL RL LQ C C C t t t R p p p O O O tC Op S QA

)

) ) ) ) S 10 =3 =5 =7 QA (m (m (m m= S S S ( QA QA QA AS RL RL RL LQ C C C t t t R p p p O O O tC Op

3 m=

L CR

m increases. For quantum energy evaluations, OptCRLQAS achieves an average time reduction of 69.2% at m = 3, scaling up to an impressive 89.3% reduction at m = 10 across all qubit scales. Similarly, the classical optimization time is heavily mitigated, yielding an average decrease of 48.2% at m = 3 and up to 84.9% at m = 10. This benchmarking clearly establishes that accumulating structural modifications before parameter optimization successfully breaks the runtime bottleneck, enabling RL-based quantum architecture search to scale to larger qubit regimes without prohibitive GPU compute requirements. Table 5: Comparison between vanilla CRLQAS and OptCRLQAS (m = 10) on 8-qubit H2 O over three random seeds. For OptCRLQAS, the total gate count is computed as ROT + CNOT. Method

Seed

Error (Ha)

Total gates

CNOT

ROT

−3

CRLQAS [19]

1 42 786

1.167 × 10 1.171 × 10−3 1.167 × 10−3

153 80 100

129 59 66

24 21 34

OptCRLQAS (Ours)

1 42 786

1.167 × 10−3 1.171 × 10−3 1.168 × 10−3

81 97 87

45 23 31

36 74 56

To isolate the effect of amortized evaluation beyond wall-clock speedup, we additionally compare vanilla CRLQAS and OptCRLQAS on 8-H2 O under matched training conditions, varying only the update schedule. Table 5 shows that the two methods reach essentially identical final energies: the mean error is 1.168636 × 10−3 Ha for CRLQAS and 1.168738 × 10−3 Ha for OptCRLQAS, a difference of only ∼ 10−7 Ha. Despite this negligible difference in energy, OptCRLQAS produces substantially more hardware-efficient circuits, reducing the mean total gate count from 111.0 to 88.3 (20.4% fewer gates) and the mean CNOT count from 84.7 to 33.0 (61.0% fewer CNOTs), at the cost of more single-qubit rotations. This is consistent with the intuition behind OptCRLQAS: by evaluating the circuit only after multiple architectural edits, the agent receives a less myopic and more informative reward signal, which can preserve final accuracy while steering learning toward circuits with substantially lower 2-qubit cost. 22

ReaPER ω sensitivity and variant selection

In the main text, Figure 3 reports gate counts for a single fixed-ω ReaPER variant per molecular system: ω = 0.4 for 6-qubit BEH2 and ω = 0.6 for 8-qubit H2 O. This appendix documents the full sweep over ω that motivates these choices. For each system we train agents with ω ∈ {0.2, 0.4, 0.6} (6-qubit) and ω ∈ {0.2, 0.4, 0.6, 0.8} (8-qubit), using three independent random seeds {1, 42, 786} per setting. All other hyperparameters are identical to the main-text QAS experiments (Appendix K). We report mean and standard deviation over seeds for CNOT count and single-qubit rotation (ROT) count. Results are given in Table 6. Since two-qubit gates dominate both circuit noise and compilation cost on near-term hardware, we select the ω value that minimizes mean CNOT count subject to the agent achieving chemical accuracy (∆E ≤ εchem ) across all seeds, using ROT count as a tiebreaker. Table 6: CNOT and ROT gate counts (mean ± std. dev. over 3 seeds) for ReaPER ω variants. All entries achieve chemical accuracy (∆E ≤ εchem = 1.6 × 10−3 Ha). Bold rows indicate the variant selected for the main text. PER, ReaPER+, and Vanilla are included for reference.

6-qubit BEH2

8-qubit H2 O

ReaPER

CNOT

ROT

CNOT

ROT

ω = 0.2 ω = 0.4 ω = 0.6 ω = 0.8

30.7 ± 18.8 23.7 ± 9.0 30.7 ± 7.4 –

20.7 ± 9.9 29.3 ± 21.5 30.3 ± 2.1 –

84.3 ± 75.4 71.7 ± 46.6 51.0 ± 25.1 78.7 ± 46.2

39.3 ± 32.0 38.3 ± 14.6 24.3 ± 20.6 35.0 ± 5.0

For 6-qubit BEH2 , ω = 0.4 achieves the lowest mean CNOT count of 23.7 ± 9.0 gates among all ReaPER variants, compared with 30.7 ± 18.8 for ω = 0.2 and 30.7 ± 7.4 for ω = 0.6. For 8-qubit H2 O, ω = 0.6 achieves the lowest mean CNOT count of 51.0 ± 25.1 gates–a 41% reduction relative to PER (86.3 ± 37.5) and 29% relative to ω = 0.4 (71.7 ± 46.6).

Across both systems, increasing ω shifts the replay distribution toward reliability-aware sampling more aggressively, which tends to reduce CNOT count at the cost of slightly higher energy variance. The optimal ω is therefore task-dependent: shorter-horizon problems with lower qubit counts benefit from moderate reliability weighting (ω = 0.4), while longer-horizon, larger systems favour stronger weighting (ω = 0.6) to suppress noisy TD targets in a deeper search space. This task-dependence is precisely what motivates the annealed ReaPER+ construction, which avoids committing to a fixed ω by scheduling the transition adaptively during training (Section 3). where Xi , Yi , Zi denote the Pauli operators acting on site i and the first sum runs over nearest-neighbour pairs i ∈ {1, . . . , n−1}. The isotropic exchange interaction (equal XX, YY, and ZZ couplings) places the ground state in a highly entangled singlet sector, making it a non-trivial testbed for variational ansatz construction: a circuit that correctly captures the ground state must generate multi-qubit entanglement across the full chain, which rewards replay strategies that retain long-horizon, high-fidelity trajectories. The target energy E0 is obtained by exact diagonalization and the sum of Pauli coefficients are utilized as the fake minium energy.

Hamiltonians

J.1

Chemical Hamiltonian

In Table 7 we provide the detailed configuration of molecules utilized throughout the quantum architecture search. J.2

Heisenberg model Hamiltonian formulation

In addition to molecular chemistry benchmarks, we evaluate OptCRLQAS and replay-buffer design on the n=5 qubit one-dimensional isotropic Heisenberg model with a uniform longitudinal field. 23

Table 7: The geometry and basis of molecules used in this research. The coordinates are in Angstrom units. Molecule

Geometry

Basis

6-BEH2

H (0,0,-1.33); Be (0,0,0); H (0,0,1.33)

STO-3G

8-H2 O

H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0)

STO-3G

10-H2 O

H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0)

6-31G

12-H2 O

H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0)

6-31G

The Hamiltonian is defined on a chain of n spin- 12 particles with open boundary conditions as HHeis =

n n X X Xi Xi+1 + Yi Yi+1 + Zi Zi+1 + Zi , i=1

(33)

i=1

Table 8: Environment, agent, and optimizer configuration for quantum architecture search benchmarks. Parameter

6-qubit

8-qubit

12-qubit

5000 6 70 5.5

5000 8 250 5.0 0.001 2000 5.0 50 10 10

1000 12 300 5.0

Environment Episodes Qubits Max layers Accept error (Ha) Shift threshold ball Shift threshold time Success switch (Ha) Success threshold opt step interval (m) energy interval (m)

5.5 10 10

5.0 15 15

Agent (DQN with n-step returns [63]) Network layers [1000]3 [1000]4 [1000]4 Batch size 1000 Replay buffer size 20 000 Learning rate 3 × 10−4 n-step return 5 6 6 Target net update Every 500 steps γ (discount) 0.005 Dropout 0.0

Exploration (ϵ-greedy) ϵstart ϵmin ϵdecay

1.0 0.05 0.99995

Classical optimizer (COBYLA [64]) Iterations

1000

Hyperparameters for quantum architecture search

Tables 8 summarize the environment and agent configurations used across the three QAS benchmarks. All experiments share the same agent architecture (multi-step DQN with n-step returns), optimizer, and curriculum type, differing only where the problem scale demands it. Environment. The maximum circuit depth (num layers) grows with qubit count, from 70 (6-qubit) to 250 (8-qubit) to 300 (12-qubit), to accommodate the deeper circuits required 24

for larger Hamiltonians. The curriculum uses a MovingThreshold introduced in ref. [19] schedule in all cases, with the acceptance error initialized at 5.0-5.5 Ha and tightened by 0.001 Ha every 2000 episodes. OptCRLQAS evaluates the circuit every m architectural edits (opt step interval = energy interval): m=10 for the 6- and 8-qubit problems and m=15 for the 12-qubit problem, reflecting the longer episodes at larger scale. Agent. The DQN agent uses a 3-layer MLP ([1000, 1000, 1000]) for the 6-qubit task and a 4-layer MLP ([1000, 1000, 1000, 1000]) for the 8- and 12-qubit tasks. The n-step return horizon is n=5 for 6-qubit and n=6 for 8- and 12-qubit. Exploration follows an ϵ-greedy schedule decaying from ϵ=1.0 at rate 0.99995 per step to a floor of ϵmin =0.05. The replay buffer holds 20 000 transitions in all cases, with a batch size of 1000. Classical optimizer. All three tasks use COBYLA [64] with a local search size of 8. The 6- and 8qubit tasks run 1000 global iterations per evaluation, as does the 12-qubit task; the reduced number of evaluations per episode at 12-qubit due to the larger amortization interval m=15 keeps the total classical optimization cost manageable.

State encoding for quantum architecture search

In both gateset configurations, the partially constructed ansatz at each step is represented as a threedimensional state tensor S ∈ RL×R×n , where L is the maximum number of circuit layers (moments), n is the number of qubits, and R is the number of row channels encoding gate placement and, optionally, variational angle parameters. A moment-tracking vector µ ∈ Zn records the next available layer index per qubit, ensuring causal gate ordering. At each time step the agent places at most one 2-qubit gate and one 1-qubit rotation, updates µ accordingly, and returns a flattened view of S as the observation. Encoding I (for gateset {RX, RY, RZ, CX}). This a a complete binary encoding. The action is a four-tuple a = [a0 , a1 , a2 , a3 ], where a0 is the CNOT control qubit (a0 = n signals no CNOT), the target is t = (a0 + a1 ) mod n, a2 is the rotation qubit (a2 = n signals no rotation), and a3 ∈ {1, 2, 3} encodes the axis (X/Y/Z). Gate placement writes are strictly binary: S[ℓ][t][a0 ] ← 1 S[ℓ][n + a3 − 1][a2 ] ← 1

(CNOT), (rotation),

(34) (35)

where ℓ is the resolved moment from µ. Because the gate set contains only one two-qubit gate type (CX), all non-zero connectivity entries are uniformly 1 and gate identity is unambiguous. Encoding II (for gateset {RXX, RYY, RZZ, RX, RY, RZ}). This encoding is integer-valued type. The xx yy yy zz zz action is an eight-tuple a = [axx 0 , a1 , a0 , a1 , a0 , a1 , arot , aaxis ], providing a control-qubit and an offset for each of the three two-qubit gate types plus a single-qubit rotation. Targets are tg = (ag0 + ag1 ) mod n for g ∈ {xx, yy, zz}. Rather than introducing separate binary planes per gate type, distinct integer labels are assigned within a single shared connectivity plane: S[ℓ][txx ][axx 0 ] ← 1,

S[ℓ][tyy ][ayy 0 ] ← 2, S[ℓ][tzz ][azz 0 ] ← 3, S[ℓ][n + aaxis − 1][arot ] ← 1 (rotation).

(36) (37) (38) (39)

A fully binary encoding for K distinct two-qubit gate types requires K separate n × n connectivity planes, giving a connectivity footprint of O(Kn2 ). Encoding II consolidates all K types into a single plane via integer labels, reducing this to O(n2 ) regardless of K, which is particularly advantageous as both the gate vocabulary and qubit count grow. The tradeoff is that integer labels introduce an implicit ordinal relationship among gate types that carries no physical meaning; a fully binary scheme provides a strictly one-hot signal per gate type that may be easier for the network to disentangle. A systematic comparison of the two encoding strategies across a controlled set of benchmarks is beyond the scope of the present work and is left as a direction for future investigation. 25

Table 9: Hyperparameters used for 1 and 2-qubit quantum compilation experiments. All agents share a common environment and training configuration; buffer-specific parameters are listed separately in the lower block.

Parameter

Value

Network (all agents) Architecture Hidden layer Activation Input size Output size Discount factor γ Learning rate Batch size Replay capacity Target update freq. Gradient clipping ε-start ε-min ε-decay

2 (each 128 units) ReLU 8 6 / 3 (small rotations / HRC) 0.99 3 × 10−4 200 5 × 105 100 episodes 1.0 1.0 0.01 0.99931

HER k-future relabeling Strategy

5 future

PER α β0 β anneal frames

0.6 0.4 105

ReaPER α ω β0 β anneal frames

0.4 0.2 0.4 105

ReaPER+ α ωmin ωmax Tann β0 β anneal frames

0.4 0.1 0.7 5 × 105 0.4 105

Hyperparameters for quantum compilation

To isolate the effect of replay design, all quantum compilation experiments use a common off-policy deep Q-learning setup. The network architecture, optimization procedure, exploration schedule, and replay capacity are held fixed across agents, so that performance differences can be attributed to the replay mechanism rather than to changes in model capacity or training protocol. Table 9 lists the hyperparameters used in the 1- and 2-qubit compiling benchmarks. The upper block gives the shared network and training configuration, while the lower blocks specify the additional method-dependent parameters for HER, PER, ReaPER, and ReaPER+. In this way, the comparison remains controlled, and the gains of ReaPER and ReaPER+ can be interpreted as arising from replay design itself.

Record · ID 126503 · SHA-256 5a5c8c7f59c25891
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.