Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points Mustafa Emre Gürsoy1,2 , Stefan Uhlich1 , Ryoga Matsuo1,2 , Yağız Gençer1,2 , Arun Venkitaraman1 , Chia-Yu Hsieh1 , Andrea Bonetti1 , Eisaku Ohbuchi3 , Lorenzo Servadei1,4 1 Sony Group Corporation, Switzerland
2 EPFL, Switzerland
3 Sony Semiconductor Solutions, Japan
4 TU Munich, Germany
arXiv:2607.14008v1 [cs.LG] 15 Jul 2026
Abstract In this paper, we introduce Lighthouse RL, a sample-efficient reinforcement learning (RL) approach for analog circuit sizing. Traditional methods lack generalization across different performance targets, while standard RL approaches waste resources exploring unpromising regions. Our method addresses these inefficiencies through a strategic reset strategy that initializes episodes from high-performing configurations discovered during training, called “lighthouses”. These states, which are closer to the target objectives, guide exploration toward promising regions. When compared to RL and Bayesian optimization methods from the literature, we demonstrate the effectiveness of our approach on a 2D benchmark problem and on two analog circuits, showing significant improvements in sample efficiency (up to 1.72× faster), optimization performance (100% vs. 0-87% success rate), generalization (75% vs. 0-50% extrapolation success), and objective maximization. This efficiency is particularly valuable for computationally expensive black-box optimization problems, and our reset strategy can be used as a plugand-play enhancement for any RL-based optimization approach.
Keywords Reinforcement Learning, Analog Sizing, Optimization, Black-Box. Figure 1: Concept of Lighthouse RL: strategic reset points guide exploration toward feasible solutions with fewer steps.
1
Introduction
Analog circuit sizing presents optimization challenges due to complex parameter-performance relationships. Traditional optimization approaches such as Bayesian Optimization (BO) [25] or Evolutionary Strategies (ES) [3] require re-optimization from scratch whenever target objectives change. This limitation becomes increasingly problematic as the number of potential target combinations grows, making these methods computationally inefficient for practical circuit design workflows, where designers frequently need to explore multiple objectives. Although reinforcement learning (RL) has shown promise for this task by learning adaptable policies, standard RL approaches still suffer from sample inefficiency, wasting computational resources exploring unpromising regions of the parameter space. In this paper, we introduce Lighthouse RL, a novel approach that addresses these limitations through a strategic reset strategy that adaptively initializes episodes from high-performing parameter configurations called “lighthouses” discovered during training. These states guide exploration toward promising regions by providing advantageous starting points closer to the target objectives. This approach significantly improves sample efficiency and generalization capabilities compared to conventional methods, making it particularly valuable for black-box circuit optimization problems where simulation could be computationally expensive.
In this work, we present a reinforcement learning formulation specifically designed for multi-objective circuit optimization problems with black-box evaluation functions. In particular, we present a novel reset strategy for episodic reinforcement learning that maintains a set of high-performing solutions discovered during training. These states serve as strategic reset points for subsequent episodes, guiding exploration towards the promising regions of the state-space during training. The proposed reset strategy can be integrated with any RL-based black-box optimization approach without modifying the underlying algorithm, making our approach a plug-and-play enhancement to existing algorithms.
2 Background and Previous Work 2.1 Previous Work Circuit sizing optimization has evolved from traditional methods to machine learning approaches. Evolutionary algorithms [10, 14] and Bayesian Optimization [15–17, 30, 31] offer effective solutions but require re-optimization for each new target objective. Recent RL approaches by Settaluri et al. [24] and Shi et al. [26] have shown promise in adaptability, while Wang et al. [34] combined graph neural networks with RL for transferable transistor sizing. Uhlmann et al. [32] addressed the challenge of sparse rewards in circuit sizing
Gürsoy, Uhlich, Matsuo, Gençer, Venkitaraman, Hsieh, Bonetti, Ohbuchi, Servadei
by implementing Hindsight Experience Replay [1]. Budak et al. [4] explored changing reset points on a per-time step basis, which works well to maximize a scalar but struggles with generalization across objective ranges. Cao et al. [5] improved sample efficiency by using Bayesian optimization to find starting points, but it is highly dependent on the quality of these initial solutions. Despite these advances, existing approaches still struggle with sample efficiency when exploring the parameter space. Our work addresses this limitation through a strategic reset strategy that leverages high-performing configurations discovered during training, without requiring additional optimization methods.
2.2
Reinforcement Learning
Reinforcement learning [27] enables agents to learn optimal decision making through environmental interaction, with successes in continuous control [29], games [18], and autonomous driving [11]. The RL framework is typically expressed as a Markov Decision Process (MDP) defined by (S, A, P, R, 𝛾), where S is the state space, A is the action space, P represents the transition probabilities, R is the reward function, and 𝛾 ∈ [0, 1] is the discount factor. The agent’s goal is to learn a policy 𝜋 : S → A that maximizes the expected cumulative discounted reward 𝐽 (𝜋) by mapping states to actions.
2.3
Soft Actor-Critic
Soft Actor-Critic (SAC) [8] is an off-policy algorithm that optimizes both expected return and policy entropy for continuous control tasks. The algorithm employs three key neural networks: a policy network 𝜋𝜙 (𝑎|𝑠) parameterized by 𝜙, and two Q-networks 𝑄𝜃 1 (𝑠, 𝑎) and 𝑄𝜃 2 (𝑠, 𝑎) parameterized by 𝜃 1 and 𝜃 2 . The Q networks estimate the expected return and are trained to minimize the Bellman error with the loss function given as: h 2i 𝐿𝑄 (𝜃 𝑖 ) = E (𝑠,𝑎,𝑟,𝑠 ′ )∼D 𝑄𝜃𝑖 (𝑠, 𝑎) − 𝑦 𝑦 = 𝑟 + 𝛾 (𝑄 target (𝑠 ′, 𝑎 ′ ) − 𝛼 log 𝜋𝜙 (𝑎 ′ |𝑠 ′ ))
(1) (2)
where 𝑎 ′ ∼ 𝜋𝜙 (·|𝑠 ′ ), D is the replay buffer, and 𝑄 target (𝑠 ′, 𝑎 ′ ) = min 𝑗=1,2 𝑄𝜃¯𝑗 (𝑠 ′, 𝑎 ′ ) uses target networks to reduce the overestimation bias [9]. The policy network outputs a Gaussian distribution and is trained to maximize both Q-values and entropy with the objective of: h i 𝐽SAC (𝜙) = E𝑠∼D E𝑎∼𝜋𝜙 min𝑖=1,2 𝑄𝜃𝑖 (𝑠, 𝑎) − 𝛼 log 𝜋𝜙 (𝑎|𝑠) (3)
This dual objective allows SAC to balance exploitation (maximizing Q-values) with exploration (maximizing entropy), making it effective for complex continuous control problems. We chose SAC because, compared to on-policy methods like [22, 23, 28], it offers better sample efficiency by reusing past experiences and eliminates the need for complex trust region constraints. Against other off-policy algorithms such as [6, 13], SAC’s entropy maximization provides more robust exploration, reduces sensitivity to hyperparameters, and helps prevent policy collapse to deterministic solutions, resulting in better stability and performance across diverse environments [9].
3
Problem Formulation
We formulate the multi-objective circuit sizing task as a feasibility problem. Let x ∈ R𝑛 represent the vector of decision variables constrained within the bounds xmin ≤ x ≤ xmax . Let I denote the set of objective indices. For a given set of parameters x, we define a set of objective functions o(x) = [𝑜𝑖 (x)] 𝑖 ∈ I that evaluate the quality of the solution. Each objective function 𝑜𝑖 (x) can be evaluated via a black-box process (e.g., simulation), making gradient information unavailable. Our goal is to find a feasible solution x∗ , i.e., Find x∗ ∈ X
such that 𝑜𝑖 (x∗ ) ≥ 𝑜𝑖,target
∀𝑖 ∈ I
(4)
where X = {x ∈ R𝑛 : xmin ≤ x ≤ xmax } represents the design space. Formulation of the RL problem: We reformulate this feasibility problem as a reinforcement learning task. The RL agent operates in the continuous action space A ⊂ R𝑛 where each action a ∈ A is bounded by predefined limits [amin, amax ]. At each time step 𝑡, the agent performs an incremental change to the parameters, followed by clamping to ensure the values remain within the designed space, that is: x𝑡 +1 = clip(x𝑡 + a𝑡 , xmin, xmax )
(5)
where clip(v, vmin, vmax ) constrains each element of vector v to lie within the corresponding bounds. An episode consists of 30 time steps and is terminated early if the target objectives are met. Reward function: The total reward at each time step 𝑡 encourages the agent to meet the target objectives. The objective penalty 𝑟 obj (x) penalizes undershooting targets: ∑︁ 𝑟 obj (x) = 𝛼𝑖 · 𝑟𝑖 (x) (6) 𝑖∈I
where for each objective, the reward component is defined as 𝑟𝑖 (x) = min(0,
𝑜𝑖 (x) − 𝑜𝑖,target ) 𝑜𝑖𝑛𝑜𝑟𝑚
(7)
with 𝛼𝑖 being a weighting factor, and 𝑜𝑖norm as a normalization constant. The final reward at time step 𝑡 is ( 𝑅 if 𝑜𝑖 (x) ≥ 𝑜𝑖,target ∀𝑖 𝑟𝑡 (xt ) = (8) max(𝑟 obj(xt ) , 𝑟 min ) otherwise where 𝑅 is a substantial reward for the agent for finding a feasible solution, and 𝑟 min is the lower bound for stable learning. Network structure: The network employs an actor-critic architecture that processes circuit structures as graphs, where each component (e.g., NMOS, PMOS, resistor, capacitor) is represented as a node with features encoding component type and normalized parameters similar to [34]. The actor transforms the circuit graph through four Graph Attention Network (GAT) [33] layers with residual connections and layer normalization [2], while processing the design objectives separately via an MLP. These representations are concatenated and fed through another MLP to generate action distribution parameters. Critic functions similarly, but to obtain the Q-value, we concatenate the action with the objective and parameter embeddings after processing it via a different MLP.
Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points
Algorithm 1 Lighthouse RL Initialize replay buffer D , priority queue P, successful set S, lighthouse L , parameters
𝑛, 𝑚, 𝑘, 𝑁 update // Phase 1: Exploration - Finding Lighthouse States while True do if episode mod 𝑁 update = 1 then min , omax ) Sample new target onew target ∼ U (o
Form R as top 𝑛 params closest to onew target from P end if Reset env with randomly sampled x0 from R , set target onew target for each time step do Sample action a𝑡 ∼ 𝜋𝜃 (· |s𝑡 ) , execute, store transition, update model following SAC end for Extract best parameters x∗ , specs o∗ if o∗ ≥ omin then Add (x∗ , o∗ ) to S else Add (𝑑 (o∗ , omax ), x∗ , o∗ ) to P end if if |S| ≥ 𝑚 then Choose top 𝑘 from P as lighthouse L Break end if end while // Phase 2: Exploitation - Using Lighthouse States while True do if episode mod 𝑁 update = 1 then min , omax ) Sample new target onew target ∼ U (o end if Reset env with randomly sampled x0 from L , set target onew target for each time step do Sample action a𝑡 ∼ 𝜋𝜃 (· |s𝑡 ) , execute, store transition, update model following SAC end for if current time step > maximum then break end if end while
4
Proposed method: Lighthouse RL
Lighthouse RL addresses a fundamental challenge in reinforcement learning for optimization: balancing exploration of the parameter space with efficient exploitation of promising regions. As illustrated in Fig. 1, traditional RL approaches typically start each episode from a fixed point (red circle in top panel), requiring the agent to repeatedly explore similar trajectories (blue/orange paths) to reach feasible solutions. However, learning in green regions is ineffective for optimization, adding only computational cost without benefit. Our key insight is to strategically identify and leverage highperformance parameter configurations discovered during training as lighthouse states (bottom panel). These lighthouses serve as intelligent starting points for subsequent episodes, allowing the agent to begin exploration from advantageous positions closer to feasible regions. The complete algorithm is presented in Alg. 1, which consists of two distinct phases: an exploration phase to discover these lighthouse states, followed by an exploitation phase that leverages them to accelerate convergence. We now describe each phase in detail.
4.1
Phase 1: Exploration - Finding Lighthouse States
For each objective 𝑖 ∈ I, we define a target range [𝑜𝑖min, 𝑜𝑖max ] representing acceptable performance bounds. During the exploration phase, the algorithm systematically searches for parameter configurations that approach but do not quite satisfy all minimum objectives. These near-optimal configurations, which we call lighthouse states, will later serve as strategic starting points for the exploitation
phase. To use later, we define a distance function 𝑑 (o𝑎 , o𝑏 ) between the objective vectors, v t 2 1 ∑︁ min(0, 𝑜𝑏,𝑖 − 𝑜 𝑎,𝑖 ) (9) 𝑑 (o𝑎 , o𝑏 ) = |I| 𝑜𝑖norm 𝑖∈I
where 𝑜 𝑎,𝑖 is the value of the objective 𝑖 in the vector o𝑎 , and 𝑜𝑖norm is a normalization factor for each objective. At the beginning of each episode, we sample a new objective min , omax ) to use as the next target. To efficiently vector onew target ∼ U (o track our progress, we maintain two distinct sets of solutions. First, the priority queue P is defined as P = {(𝑑 (o𝑝 , omax ), x𝑝 , o𝑝 ) | 𝑝 ∈ {1, 2, ..., |P|}, ∃𝑖 ∈ I : 𝑜 𝑝,𝑖 < 𝑜𝑖min }
(10)
stores promising but incomplete solutions, where 𝑑 (o𝑝 , omax ) measures how close each solution is to the upper performance bounds, and sorts the queue. Solutions with smaller distance values are prioritized as they are closer to satisfying all objectives at their maximum levels. Second, the successful set S is defined as S = {(x𝑠 , o𝑠 ) | 𝑠 ∈ {1, 2, . . . |S|}, ∀𝑖 ∈ I : 𝑜𝑠,𝑖 ≥ 𝑜𝑖min }
(11)
contains solutions that fully satisfy all minimum objectives. This separation into P and S offers three key advantages: (1) It prevents the waste of computational resources in initializing episodes with parameters that already satisfy all requirements. (2) By tracking successful solutions in S, we establish a clear criterion for when to transition from exploration to exploitation. (3) The discovered solutions have practical value as readyto-use parameter sets for analog designers with similar objectives. In every 𝑁 update episodes, we sample new target objectives and select the 𝑛 parameters from P with minimal 𝑑 (o𝑝 , onew target ) to form the reset point set R. For each subsequent episode, we randomly sample a starting point from R, focusing exploration on promising regions of the parameter space. When the successful set S reaches a predefined size 𝑚, we select the first 𝑘 solutions of P with minimal 𝑑 (o𝑠 , omax ) as our lighthouse states L = {x𝐿1 , x𝐿2 , ..., x𝑘𝐿 }. This selection marks the transition from the exploration phase to the exploitation phase, where we consolidate on a set of lighthouses, and fine-tune the policy starting only from them.
4.2
Phase 2: Exploitation - Using Lighthouse States
In the exploitation phase, we stop the process of appending solutions to the sets P and S. Instead, reset states are only randomly selected from the lighthouse states, i.e.: xreset ∼ Uniform(L)
(12)
This randomization is important, as it gives the model the ability to generalize, rather than overfitting to a certain trajectory. Phase 2 serves an important purpose in our approach. Once we have collected enough successful solutions in Phase 1, we will have sufficient lighthouse states to guide future exploitation. At this
Gürsoy, Uhlich, Matsuo, Gençer, Venkitaraman, Hsieh, Bonetti, Ohbuchi, Servadei
LH RL
RL Backbone - Fixed
RL Backbone - Random
Parameter Space - Training
Lighthouse State
Target
Parameter Space - Inference
5
Target Sampling Space
Objective Space - Inference
5
0
0
0
f₂
x₂
x₂
−20 −40 −60 −5
−5 −5
0
5
−80
−5
0
x₁
5
−40
−20
x₁
(a)
0
f₁
(b)
Figure 2: (a) Total number of successful episodes over 12,000 training time steps. (b) Multi-objective optimization comparison between Lighthouse RL and RL backbone approaches with fixed and random reset points. (Left) Parameter space trajectories during training showing how each method navigates the search space. Lighthouse states provide strategic starting points that accelerate convergence to optimal solutions. (Middle) Parameter space trajectories during inference for two distinct targets, demonstrating Lighthouse RL’s superior convergence properties. For the random reset RL backbone, several trajectories are presented starting from random reset points. (Right) Objective space visualization during inference showing how all three methods perform against target objectives. While all approaches successfully reach the inference target, Lighthouse RL takes the fewest steps. point, continuing to discover new regions would force the agent to adapt to foreign territories, which, while favorable for the diversity of solutions, would slow the training progress. By discontinuing the exploration of new parameter regions and focusing solely on the lighthouse states, we transition from exploration to exploitation, strengthening the model’s understanding of the regions near these strategic starting points. Although our formulation focuses on circuit optimization, the Lighthouse RL approach applies to general black-box optimization problems. In 5.1, we demonstrate this adaptability by studying a simple 2D multi-objective feasibility problem before addressing two more complex circuit sizing tasks.
Table 1: Parameter Number in RL-Based Sizing Algorithms [5]
[26]
[4]
[24]
[34]
This work
# Parameters
10
20
20
29
25
29
Table 2: Ranges of Design Parameters and Objectives Parameters
Two-Stage
Multistage [21]
𝑊 𝐿 𝐶 𝑅
1–64 𝜇 m 1–10 𝜇 m 0.1–5 nF 1–100k Ω – – 11
0.36/0.42–100 𝜇 m (NMOS/PMOS) 0.15–10 𝜇 m 0.1–5 nF 0.1–1M Ω 0–1.8 V 1–100 𝜇 A 29
VCM I # params
5 Experiments 5.1 2D Benchmark Function Analysis To evaluate our reset strategy, we apply Lighthouse RL to a multiobjective feasibility problem featuring two sphere functions. Each sphere function is defined as 𝑓𝑖 (x) = −∥x−x𝑖0 ∥ 2 with global maxima at different locations: x01 = −x02 = [2, 2]. This creates a landscape with two distinct regions of interest. The objective vector is defined as [𝑓1 (x), 𝑓2 (x)], where we aim to maximize both function values. Following our methodology, we sample the target objectives and assign the agent to find parameters [𝑥 1, 𝑥 2 ] that meet these objectives. After training our Lighthouse RL method and RL backbones with fixed and random reset points for 12,000 time steps, we evaluated them targeting [1.5, 2] with function value (−0.25, −28.25), a challenging point near the maxima of one of the sphere functions. As shown in Fig. 2b, Lighthouse RL converges to target objectives in fewer inference steps than both alternatives. By initializing from lighthouse states rather than exploring suboptimal regions, Lighthouse RL reduces computational cost, which is particularly valuable when function evaluations are expensive.
Reference
Objectives Gain BW PM GM Sat. V
Train [60,65] dB [2,3] MHz [45,60]° [10,15] dB >0V
Extrap. [65,70] dB [3,4] MHz [60,70]° [15,17.5] dB >0V
Train [55,65] dB [4,5] MHz [50,70]° [10,15] dB -
Extrap. [75,80] dB [5,6] MHz [65,70]° [10,15] dB -
Max [130,135] dB [15,20] MHz [60,65]° [45,50] dB -
As shown in Fig. 2a, the graph indicates that Lighthouse RL achieves more successful episodes during training by leveraging previously discovered parameter sets. This benchmark demonstrates how Lighthouse RL navigates optimization landscapes with multiple regions of interest. This behavior is desirable, as many black-box optimization problems have multiple viable solutions that exist in different regions of the parameter space.
5.2
Analog Circuit Sizing: Experimental Setup
We evaluated Lighthouse RL on two analog circuits. The first is a two-stage operational amplifier (Fig. 3) consisting of eight transistors, one compensation capacitor, and one resistor. Due to matching requirements, certain transistors share parameters, resulting in 11 independent variables that need to be optimized. The second circuit
Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points
Table 3: Comparison of circuit sizing algorithms for two circuits. Success rate and average inference steps are presented with means and one standard deviation for 5 different seeds. The best results are in bold. Two-Stage OpAmp Within Distribution (C)
Method BO RoSeOpt RoSeOpt (f-g) RL Backbone Lighthouse RL
Multistage Amplifier[21] Extrapolation (D)
Within Distribution (C)
Extrapolation (D)
T.S.E.†
SR (%)‡
Inf. Steps§
SR (%)
Steps
T.S.E.†
SR (%)‡
Inf. Steps§
SR (%)
Steps
N/A 1.00× 1.35× 1.10× 1.55×
37.6 ± 4.8 58.2 ± 15.4 87.2 ± 14.7 100.0 ± 0.0 100.0 ± 0.0
123.1 ± 6.2 18.0 ± 4.2 11.1 ± 1.2 15.7 ± 2.1 3.8 ± 1.4
8.0 ± 4.4 5.6 ± 4.0 10.0 ± 14.4 50.4 ± 41.6 75.2 ± 26.8
144.7 ± 2.8 28.7 ± 1.0 27.8 ± 3.1 24.7 ± 5.6 15.6 ± 6.5
N/A 1.02× 1.00× 1.00× 1.72×
26.4 ± 6.2 13.2 ± 26.4 2.0 ± 4.0 0.0 ± 0.0 87.2 ± 18.0
560.8 ± 19.7 28.0 ± 4.0 29.8 ± 0.3 30.0 ± 0.0 7.9 ± 4.8
10.0 ± 1.8 7.6 ± 15.2 0.4 ± 0.3 0.0 ± 0.0 29.2 ± 37.0
612.5 ± 2.0 28.8 ± 2.3 29.9 ± 0.02 30.0 ± 0.0 22.8 ± 10.2
† Training Sample Efficiency - relative training efficiency (number of simulations to reach 50 successful episodes), normalized to the least efficient method. ‡ Success rate - percentage of successful episodes during inference out of 50 randomly sampled target objectives. § Inference Steps - average number of simulations for inference episodes. For failed episodes, the maximum episode length was used.
Figure 3: Two-stage operational amplifier schematic
Skywater SKY130 PDK [7] with Ngspice [20] for simulation and with a typical process corner (TT). For fair comparison, all learning-based methods were allocated 6,000 SPICE [19] simulations for the two-stage OpAmp and 30,000 simulations for the multistage amplifier during training. For the first two experiments, all methods were evaluated on 50 different target objectives sampled from the ranges in Table 2, with a maximum of 30 inference steps allowed per target. Since BO does not require training, we allocate it a total budget equivalent to the combined training and inference budgets of other methods (e.g., 7,500 simulations, that are 150 simulations per target) for the two-stage OpAmp: 6,000 training + 50 targets × 30 steps maximum, similarly 630 simulations/target for multistage amplifier). Each method was evaluated in five different random seeds for reproducibility. All methods’ training progressions for the first two experiments are shown in Fig. 5a and Fig. 5b, where Lighthouse RL shows faster convergence and higher rewards for both circuits.
5.3 Figure 4: Multi-stage operational amplifier schematic is a more complex multistage amplifier from [21] (Fig. 4) that is provided as benchmark circuit in [12] with 29 independent parameters. This represents one of the largest parameter spaces addressed in the RL-based circuit sizing literature (Table 1). The decision parameters and the ranges of training objectives are detailed in Table 2. For our comparative analysis, we incorporate several baseline methods: (1) Bayesian Optimization [25], a training-free black-box optimization approach; (2) RL Backbone, our foundational reinforcement learning system, which starts episodes from a fixed point; (3) RoSeOpt [5], a state-of-the-art circuit optimization method that combines BO for initialization with RL for optimization, which has previously demonstrated superior performance compared to various optimization methods including genetic algorithms [14], Bayesian optimization [17], and other RL-based circuit optimization methods [24, 26, 34]; and (4) RoSeOpt with fine-grained action space, our modification that uses smaller discretization steps (0.25× the original for two-stage OpAmp and 0.5× for multistage amplifier) for more precise parameter adjustments, as we observed the original discretization in the paper caused abrupt objective changes for our target circuits. We evaluate each method with open-source
Experiment 1: Generalization Within Training Distribution
Our first experiment tests whether each method can meet diverse target objectives within the training ranges (Table 2), i.e., finding parameters that satisfy varying gain, bandwidth, phase margin, and gain margin requirements without retraining. As shown in Table 3, Lighthouse RL outperforms all other methods across every evaluation metric: For the two-stage OpAmp, Lighthouse RL achieves a 1.55× improvement in sample efficiency compared to RoSeOpt (1.00×), requiring fewer circuit simulations for convergence. This advantage grows to 1.72× for the multistage amplifier, compared to RoSeOpt’s 1.02×, demonstrating that Lighthouse RL’s efficiency scales favorably with problem complexity. As BO does not have a training phase, this metric is not applicable. For the two-stage OpAmp, Lighthouse RL achieves 100.0 ± 0.0% success rate across all targets and seeds, matching the RL backbone but significantly outperforming both the RoSeOpt variants (58.2 ± 15.4% and 87.2 ± 14.7% respectively) and BO (37.6 ± 4.8%). The difference becomes even more pronounced for the multistage amplifier, where Lighthouse RL maintains an 87.2 ± 18.0% success rate versus the BO, which has 26.4 ± 6.2. For the two-stage OpAmp, Lighthouse RL requires only 3.8 ± 1.4 steps on average during inference, representing a 4.1× reduction
Gürsoy, Uhlich, Matsuo, Gençer, Venkitaraman, Hsieh, Bonetti, Ohbuchi, Servadei
Table 4: Objective maximization results for multistage amplifier. Best results are shown in bold.
(a)
Gain (dB)
BW (MHz)
PM (°)
GM (dB)
Human Design RL Backbone LH RL
91 52.5 118.1
1.2 13.1 24.5
86.3 59.0 89.7
57.8 15.9 59.6
(b)
Figure 5: Reward performance comparison across methods for (a) two-stage OpAmp (6,000 SPICE simulations) and (b) multistage amplifier (30,000 SPICE simulations): RL backbone (red), RoSeOpt (dark blue), RoSeOpt fine-grained (light blue), and Lighthouse RL (green). The X-axis shows normalized training progress for fair comparison, as methods complete different numbers of episodes within the same simulation budget. Lines show median rewards (moving average, window=10) with IQR shading across five random seeds.
compared to the RL backbone and a 2.9-4.7× reduction compared to the RoSeOpt variants. For the multistage amplifier, Lighthouse RL requires only 7.9 ± 4.85 steps, compared to nearly 30 steps for all learning-based methods, resulting in a 3.5-3.8× improvement. This reduction is relevant for circuit sizing, where simulations are computationally expensive. Lighthouse RL outperforms benchmarks by strategically using lighthouse states as starting points, enabling efficient navigation to target objectives. This advantage becomes more pronounced as parameter dimensions increase from 11 to 29, where Lighthouse RL maintains high success rates while other methods struggle. Although one RoSeOpt seed performed exceptionally well, its high variance (13.2 ± 26. 4%) reveals a dependence on the initial parameters discovered by BO. Lighthouse RL shows better robustness by continually exploiting multiple high-performing states during training.
5.4
Method
Experiment 2: Extrapolation Beyond Training Distribution
Although there is good performance within the training distribution, the true value of Lighthouse RL lies in its ability to extrapolate to objectives beyond those seen during training. This extrapolation capability is especially significant, as otherwise we could simply provide designers with the best parameter sets found during training. The extrapolation task reveals even more pronounced differences between the methods. For the two-stage OpAmp, Lighthouse RL achieves 75.2 ± 26.8% success rate when extrapolating to more demanding objectives, significantly outperforming BO (8.0 ± 4.38%), both RoSeOpt variants (5.6 ± 4.0% and 10.0 ± 14.4%), and the RL backbone (50.4 ± 41.6%). Lighthouse RL requires only 15.6 ± 6.5 inference steps on average, that is 9.3× and 1.6-1.8× improvement over BO and other RL methods, respectively.
For the multistage amplifier, Lighthouse RL outperforms benchmarks with a 29.2 ± 37.0% success rate versus BO’s 10.0 ± 1.8%. This highlights how Lighthouse RL’s strategic reset is increasingly valuable as problem complexity grows.
5.5
Experiment 3: Objective Maximization
The last experiment focuses on maximizing individual circuit objectives rather than meeting specific targets in a range, a common design scenario where engineers seek to push performance limits. We conduct this experiment on the multistage amplifier. We modify our approach by providing unattainable target objectives (Table 2) during training. For this experiment, we only implement Phase 1 of Algorithm 1, continuously updating the lighthouse states rather than fixing them. This approach ensures that we always maintain and improve the best-performing configurations discovered during training. Each method is trained for 30,000 time steps for 3 different seeds and the solution with highest reward is reported. We compare Lighthouse RL against a human designer and the RL backbone. As shown in Table 4, Lighthouse RL outperforms all baselines in all objectives. It achieves a gain of 118.1 dB, a bandwidth of 24.5 MHz, a phase margin of 89.7°, and a gain margin of 59.6 dB. Note that no constraints on current consumption were imposed, enabling Lighthouse RL to achieve the reported results; additional constraints may be required in some practical settings.
6
Conclusion
In this paper, we introduced Lighthouse RL, a novel reinforcement learning approach for efficient black-box optimization that leverages strategic reset points to guide exploration. By maintaining a collection of high-performing parameter configurations discovered during training, our method significantly improves sample efficiency and generalization capabilities compared to conventional approaches. We demonstrated the effectiveness of our approach on a benchmark optimization problem and practical circuit sizing tasks, showing substantial improvements in success rates, inference efficiency, and objective maximization. The proposed reset strategy can be integrated with any RL-based optimization approach, making it valuable for addressing complex black-box optimization problems. Limitations remain in lighthouse state selection, as their diversity currently depends on training dynamics rather than explicit diversity enforcement. Future work could incorporate explicit diversity metrics and explore parallelization to accelerate the discovery of high-performance configurations.
Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points
References [1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. Advances in neural information processing systems 30 (2017). [2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). [3] Hans-Georg Beyer and Hans-Paul Schwefel. 2002. Evolution strategies–a comprehensive introduction. Natural computing 1, 1 (2002), 3–52. [4] Ahmet F Budak, Prateek Bhansali, Bo Liu, Nan Sun, David Z Pan, and Chandramouli V Kashyap. 2021. Dnn-opt: An rl inspired optimization for analog circuit sizing using deep neural networks. In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 1219–1224. [5] Weidong Cao, Jian Gao, Tianrui Ma, Rui Ma, Mouhacine Benosman, and Xuan Zhang. 2024. Rose-opt: Robust and efficient analog circuit parameter optimization with knowledge-infused reinforcement learning. IEEE Transactions on ComputerAided Design of Integrated Circuits and Systems (2024). [6] Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596. [7] Google and SkyWater Technology Foundry. 2020. SkyWater 130nm PDK. https: //github.com/google/skywater-pdk. [Online]. [8] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. Pmlr, 1861–1870. [9] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018). [10] Kourosh Hakhamaneshi, Nick Werblun, Pieter Abbeel, and Vladimir Stojanović. 2019. BagNet: Berkeley analog generator with layout optimizer boosted with deep neural networks. In 2019 IEEE/ACM International Conference on ComputerAided Design (ICCAD). IEEE, 1–8. [11] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021. Deep reinforcement learning for autonomous driving: A survey. IEEE transactions on intelligent transportation systems 23, 6 (2021), 4909–4926. [12] Jintao Li, Haochang Zhi, Ruiyu Lyu, Wangzhen Li, Zhaori Bi, Keren Zhu, Yanhan Zeng, Weiwei Shan, Changhao Yan, Fan Yang, Yun Li, and Xuan Zeng. 2025. AnalogGym: An Open and Practical Testing Suite for Analog Circuit Synthesis. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design (Newark Liberty International Airport Marriott, New York, NY, USA) (ICCAD ’24). Association for Computing Machinery, New York, NY, USA, Article 59, 9 pages. https://doi.org/10.1145/3676536.3697117 [13] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015). [14] Bo Liu, Yan Wang, Zhiping Yu, Leibo Liu, Miao Li, Zheng Wang, Jing Lu, and Francisco V Fernández. 2009. Analog circuit optimization system based on hybrid evolutionary algorithms. Integration 42, 2 (2009), 137–148. [15] Mingjie Liu, Walker J Turner, George F Kokai, Brucek Khailany, David Z Pan, and Haoxing Ren. 2021. Parasitic-aware analog circuit sizing with graph neural networks and Bayesian optimization. In 2021 Design, automation & test in Europe conference & exhibition (DATE). IEEE, 1372–1377. [16] Wenlong Lyu, Pan Xue, Fan Yang, Changhao Yan, Zhiliang Hong, Xuan Zeng, and Dian Zhou. 2017. An efficient Bayesian optimization approach for automated optimization of analog circuits. IEEE Transactions on Circuits and Systems I: Regular Papers 65, 6 (2017), 1954–1967. [17] Wenlong Lyu, Fan Yang, Changhao Yan, Dian Zhou, and Xuan Zeng. 2018. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In International conference on machine learning. PMLR, 3306–3314. [18] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602 [cs.LG] https://arxiv.org/abs/ 1312.5602 [19] Laurence Nagel and Donald O Pederson. 1973. SPICE (simulation program with integrated circuit emphasis). (1973). [20] Ngspice Contributors. 2025. Ngspice, An Open-Source SPICE Simulator. http: //ngspice.sourceforge.net/. [Online]. [21] Wanyuan Qu, Shashank Singh, Yongjin Lee, Young-Suk Son, and Gyu-Hyeong Cho. 2016. Design-oriented analysis for Miller compensation and its application to multistage amplifier design. IEEE Journal of Solid-State Circuits 52, 2 (2016), 517–527. [22] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International conference on machine learning. PMLR, 1889–1897.
[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [24] Keertana Settaluri, Zhaokai Liu, Rishubh Khurana, Arash Mirhaj, Rajeev Jain, and Borivoje Nikolic. 2021. Automated design of analog circuits using reinforcement learning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 41, 9 (2021), 2794–2807. [25] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. 2015. Taking the human out of the loop: A review of Bayesian optimization. Proc. IEEE 104, 1 (2015), 148–175. [26] Wei Shi, Hanrui Wang, Jiaqi Gu, Mingjie Liu, David Z Pan, Song Han, and Nan Sun. 2022. RobustAnalog: Fast variation-aware analog circuit design via multitask RL. In Proceedings of the 2022 ACM/IEEE Workshop on Machine Learning for CAD. 35–41. [27] Richard S Sutton, Andrew G Barto, et al. 1998. Reinforcement learning: An introduction. Vol. 1. MIT press Cambridge. [28] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12 (1999). [29] Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto MartínMartín, and Peter Stone. 2025. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 28694–28698. [30] Konstantinos Touloupas, Nikos Chouridis, and Paul P Sotiriadis. 2021. Local Bayesian optimization for analog circuit sizing. In 2021 58th ACM/IEEE design automation conference (DAC). IEEE, 1237–1242. [31] Konstantinos Touloupas and Paul P Sotiriadis. 2021. LoCoMOBO: A local constrained multiobjective Bayesian optimization for analog circuit sizing. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 41, 9 (2021), 2780–2793. [32] Yannick Uhlmann, Michael Essich, Lennart Bramlage, Jürgen Scheible, and Cristóbal Curio. 2022. Deep reinforcement learning for analog circuit sizing with an electrical design space and sparse rewards. In Proceedings of the 2022 ACM/IEEE Workshop on Machine Learning for CAD. 21–26. [33] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017). [34] Hanrui Wang, Kuan Wang, Jiacheng Yang, Linxiao Shen, Nan Sun, Hae-Seung Lee, and Song Han. 2020. GCN-RL circuit designer: Transferable transistor sizing with graph neural networks and reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC). IEEE, 1–6.