arXiv CSopen access

Robustness of Reinforcement Learning-Based Congestion Management in Low-Voltage Grids

Robustness of Reinforcement Learning-Based Congestion Management in Low-Voltage Grids Josef Hoppe1∗ , Sarra Bouchkati1† , Farah Nasr1† , Jonathan Krapp∗ , Alexander Och† , Maximilian Wirth‡ , Jan Schiefelbein-Lach§ , Oliver Pohl¶ , Andreas Ulbig† , and Michael T. Schaub∗ ∗ Computational Network Science, RWTH Aachen University

arXiv:2607.16004v1 [eess.SY] 17 Jul 2026

{hoppe, schaub}@netsci.rwth-aachen.de † IAEW, RWTH Aachen University {s.bouchkati, f.nasr, a.och, a.ulbig}@iaew.rwth-aachen.de ‡ E.ON impulse GmbH § E.ON Group Innovation GmbH ¶ Schleswig-Holstein Netz GmbH

Abstract—Increases in photovoltaic generation, charging of electric vehicles and heat-pump demand challenge operating limits in low-voltage distribution grids. This requires curative curtailment methods that can operate under sparse observability, noisy measurements, and imperfect grid models. Unlike prior end-to-end reinforcement-learning approaches for partially observable curtailment, this work decouples congestion detection and control by combining a random-forest violation pre-classifier with an actor-critic controller, and evaluates its robustness to measurement noise and grid-parameter mismatch. The framework is tested on a real low-voltage grid using synthetic future operating scenarios with low observability and controllability. With accurate grid parameters, the controller reduces total violation magnitude by 98.9%, and this performance remains nearly unchanged under the tested measurement-noise settings. Grid-model mismatch proves to be more challenging, but the controller still mitigates most violations under the tested mismatch assumptions. Index Terms—Curative Curtailment, Deep Reinforcement Learning, Optimal Power Flow, Robustness, Uncertainty.

I. I NTRODUCTION The decarbonization of energy systems is increasing the operational challenges faced by Distribution System Operators (DSOs). In low-voltage grids, the growing integration of photovoltaic (PV) systems, electric vehicles (EVs), and heat pumps is transforming grids originally designed for unidirectional power flow into systems with bidirectional and highly time-varying injections. Thus, operational limits such as thermal line loadings, voltage bounds, and transformer capacities are increasingly at risk of violation, making active congestion management essential for safe and reliable operation [1]. A common corrective action is the curtailment of controllable generation or demand to relieve asset overloading or voltage violations. The main technical challenge is to determine which asset should be adjusted, and by how much, so that grid constraints are satisfied while intervention costs are minimized. Due to volatile supply-demand conditions, such decisions must often be made in real time, within only a few 1 These authors contributed equally to this work

minutes [2]. Although this can be formulated through Optimal Power Flow (OPF) methods [3], [4], centralized OPF requires extensive measurement and communication infrastructure that is often unavailable or unreliable in practice. Edge computing has therefore been investigated as a way to reduce communication overhead and avoid single points of failure by processing data closer to its source. However, practical deployment remains challenging because only a subset of grid buses is typically observable and controllable. Classical optimization-based approaches generally assume full state information, while state estimation relies on structural assumptions and pseudo-measurements that introduce additional uncertainty. Moreover, measurement errors and inaccurate grid parameters, such as uncertain line impedances caused by incomplete documentation and aging infrastructure, create a mismatch between the simulated model and the physical system [5]. These limitations reduce the practical applicability of traditional optimization methods. In recent years, Machine Learning approaches have emerged as robust and computationally efficient alternatives to address these challenges [6]. A. Related Work Several studies have been proposed for approximating the AC-OPF problem for congestion management. In particular, Reinforcement Learning (RL) has been explored as a promising alternative for smart grid control [7]. For this, it is important to integrate constraints into the control model. Some approaches work by modifying reward functions to integrate further constraints directly [8], [9], especially by introducing system-specific Lagrangian penalties of constraint violations [10], [11]. To provide more rigorous physical feasibility guarantees, [12] instead introduces a safe convex layer, while [13] utilizes a holomorphic embedding layer within the neural architecture. Further advancing physics-informed RL, [14] proposed a framework based on power flow equations that augments the policy gradient with constraints to correct infeasible generation operations. Other approaches focus on post-processing the agent’s actions to

ensure validity: [15] enforces generation limits using a power flow solver, while [16] employs linear programming and action adjustment techniques to correct infeasible setpoints. Despite these advances, a common limitation is the assumption of a fully observable grid state. In practical settings, full observability is rarely achieved due to slow smart meter rollout or restricted access to consumer data. While our previous work [17] presented an end-to-end RL approach tailored for partial observability, the impact of real-world uncertainties remains a critical concern. Inaccuracies in grid models and measurement errors can jeopardize the performance and reliability of RL agents. Furthermore, scalability remains a significant challenge, as end-to-end frameworks must simultaneously navigate two complex tasks: identifying constraint violations under limited observability and determining optimal setpoints. The introduction of constraints already shows different approaches for handling this complexity, including separating the tasks into multiple steps, which can also be applied to the divide between detecting violations and solving them precisely.

demand Sd,i = Pd,i + jQd,i . Bold symbols denote a vector over the entire grid that consists of the individual (non-bold) measurements attached to buses or branches, respectively. The objective of active congestion management is to minimize the cost of intervention (curtailment) while satisfying the nonlinear AC power flow equations and operational limits. The generation (consumption) at bus i ∈ V is curtailed by C C Pg,i (Pd,i ). In this paper, we set the curtailment costs αi for generation and βi for consumption αi = βi = 1 for all buses, assuming the total curtailment as cost. This is formulated as an AC-OPF problem with: X C C min f (u) = αi Pg,i + βi Pd,i (1a)

B. Contributions

where u = [PgC , PdC ]⊤ is the vector of control actions, and γ represents the grid parameters (e.g., line impedances). Equation (1b) represents the physical power balance, while (1c) and (1d) ensure minimum (Vmmin ) and maximum (Vmmax ) voltage magnitude and thermal constraints (as maximum apparent max branch power Sij ). (1e) defines the maximum curtailment per bus; to define a bus as not controllable, we set maximum C,max C,max curtailment to zero (Pg,i = Pd,i = 0).

In this work, we propose a framework for the uncertaintyaware evaluation of RL-based voltage and congestion control in low-voltage distribution grids. Building upon the actor–critic architecture introduced in [17], we decouple the control logic by integrating a random forest classifier for congestion prediction, reducing the complexity of the learning task and enhancing scalability. Furthermore, we assess the robustness of the framework against model mismatch and measurement noise. The main contributions of this work are: • a modular two-step control pipeline that combines a random-forest violation pre-classifier with an actor-critic controller under partial observability, • an empirical evaluation framework for assessing sensitivity to parametric model mismatch and measurement noise, • a metric set to jointly evaluate violation mitigation and curtailment side effects, and • a case study on a real low-voltage grid topology with synthetic operating situations. II. P ROBLEM F ORMULATION This section defines the power curtailment problem. The goal is to determine the optimal operating setpoints of controllable units, while accounting for the grid constraints and the requirements of edge-based operation: partial observability and system uncertainties. A. Grid Model and AC-OPF We consider a low-voltage distribution grid represented as a graph G = (V, E), where V and E denote the sets of buses and branches, respectively. For each bus i ∈ V, the complex voltage is Vi = Vm,i ejθi , where Vm,i denotes the voltage magnitude and θi denotes the voltage angle at bus i. The apparent power Si is determined partially by controllable DER generation Sg,i = Pg,i + jQg,i and partially by flexible load

u

s.t.

i∈V

Sbus (V, γ) = P(u) + jQ(u),

(1b)

Vmmin ≤ Vm,i ≤ Vmmax , ∀i ∈ V max |Sij (V, γ)| ≤ Sij , ∀(i, j) ∈ E C,max C,max C C , 0 ≤ Pd,i 0 ≤ Pg,i ≤ Pg,i ≤ Pd,i

(1d)

(1c) (1e)

B. Partial Observability and Uncertainty In practical edge deployments, the controller lacks access to the full system state x = [P, Q, Vm , θ]. Instead, the agent operates under partial observability. Let Vobs ⊂ V be the set of observable buses. The observation vector z is defined as: z = h(xobs ) + η

(2)

where h(·) is the observation function for the monitored subset and η represents measurement noise (e.g., sensor inaccuracies or synchronization errors). Furthermore, the physical laws in (1b) and (1d) depend on the vector γ of grid parameters (e.g. impedances), which is subject to model uncertainty. We represent this uncertainty as a relative parametric mismatch, γmis = γnom ⊙ (1 + ∆γ),

(3)

where γnom denotes the nominal grid model and ∆γ represents the relative parameter mismatch. The challenge addressed in this work is to derive a control policy π : z → u that minimizes (1a) and satisfies constraints (1c)–(1d) with noisy, incomplete observations z and an inaccurate model γ. III. M ETHODOLOGY This section outlines the methodology used to develop and train a model for grid curtailment with partial measurement availability. The model is capable of handling various operational scenarios ranging from non-critical to critical grid states.

controllable observable sgen

Evaluation

Full Dataset

Result

OPF

Actor

Power Flow

Apply

No Violation

Observable

Apply

Violation Observe

Rand. Forest

PF Obs.

Fig. 2. Flow of the Evaluation. For cases where the pre-classifier predicts a violation, the actor generates multiple control steps (with updated observations) to solve the violation. Otherwise, we collect the original grid state as the result. We calculate the Optimal Power Flow (OPF) from the full data. Fig. 1. Low Voltage grid provided by Schleswig-Holstein Netz GmbH; with randomly assigned controllable and observable buses.

A. Dataset Generation The dataset is derived from the topology of a real-world low-voltage distribution grid provided by Schleswig-Holstein Netz AG. It represents a future operating scenario with increased penetration of PV systems, EVs, and heat pumps. Flexible units serve as controllable grid connection points for congestion management: their active power can be adjusted through external control signals within predefined technical limits. PV systems provide flexibility through downward curtailment of power generation, whereas EV charging and heat pumps provide demand-side flexibility by adjusting their power consumption. Following the regulatory framework in Germany, 10% of the available buses are defined as both observable and controllable (see Fig. 1). Time-series profiles are assigned to all observable and unobservable connection points using future scenarios from SimBench [18]. A quasi-static time-series simulation with 15-minute resolution over one year yields 35 040 operating points. For each operating point, the full grid state is calculated using an AC power-flow (PF) calculation. Results exceeding predefined physical limits, specifically voltage magnitudes outside Vm ∈ / [0.8, 1.2] p.u., are excluded, as such extreme conditions would typically be addressed by protection systems rather than operational congestion management. The PF results are used as the baseline, while OPF solutions provide a reference for comparison. During training, however, the model receives only PF-based information and no OPF solution data. In contrast to [17], dataset augmentation is not required because the improved control model is trained almost exclusively on operating points with violations. B. Uncertainty Modeling In this work, we consider two types of uncertainties: a) Grid Model Mismatch: As discussed above, grid parameter uncertainty may arise from inaccurate documentation, aging infrastructure, and imperfect knowledge of line parameters. In contrast to works that use uniformly distributed perturbations to assess bounded worst-case mismatch [19], we

model inaccuracies as relative Gaussian perturbations, which are usually small but may occasionally be larger. For each line and each parameter e.g. resistance, reactance, and capacitance per unit length, we define γmis = max (γnom (1 + ϵγ ), 0) ,

2 ϵγ ∼ N (0, ηmis ), (4)

where ηmis = 0.05. Perturbations are sampled independently for each line and parameter, and clipping ensures non-negative physical values. The resulting mismatched grid represents the imperfect model available to the controller, while the original grid serves as the reference physical system. b) Measurement Uncertainty: Similar to previous work [20], [21], we model measurement noise by adding zero-mean Gaussian noise to the measured quantities before feeding them to the observations. The resulting action is applied to the true (noise-free) system state to isolate the effect of observation errors on decision-making. The standard deviation of the noise is equal to half of the uncertainty range as quantified by the manufacturer η ∼ N (0, σ 2 ). To analyze the robustness more clearly, we use different multipliers for the noise, increasing or decreasing the standard deviation proportionally. C. Control Model For each operating point, the control model is executed for a fixed number of control steps. The control model consists of a random forest classifier that classifies observations into violations or non-violations, and a reinforcement-learning actor that learns to minimally curtail power to resolve violations. 1) Random Forest pre-classifier: In order to detect if a given observation contains a violation, we fit a random forest classifier to classify load situations into violations or nonviolations. This reduces the complexity of the target learned by the RL agent and speeds up the evaluation by reducing the number of time steps that need to be simulated. 2) RL Agent: The control policy is parameterized by a multilayer perceptron (MLP), called the actor, that maps observable measurements to curtailment actions. The input consists of measurements from observable buses; the output is a vector of curtailment factors for the controllable buses.

Training Setup

D. Training Setup

λ

i∈V Pg,i

+Pd,i

Verr = max max{Vmmax − Vm,i , Vm,i − Vmmin , 0} i∈V ( ) |Si,j | Serr = max max max − 1, 0 Si,j (i,j)∈E

(6) (7)

The voltage error Verr (maximum absolute violation) and branch load error Serr (maximum relative overload) are scaled to realistic maximum violations of 0.2 p.u. and +100 %, respectively. The hyperparameters ν and λ tune the relation between positive and negative rewards and the gap between small violations and maximum curtailment. It should be noted that the full grid state is only used during offline training and evaluation for reward calculation. In this stage, the simulator provides access to all bus voltages and branch loadings, which allows violation labels and rewards to be computed from

Observable Full Only Information

For training and evaluation, we split the cases in the dataset into 60% training data, 20% validation data, and 20% test data. The RL agent and the random forest are trained on the training data; and we select an agent based on the performance on the validation data. We report the results obtained on the test set to avoid erroneous good performance from the selection of the agent. The split is stratified by the type of violation to ensure each of the three sets is representative of the overall dataset. For this, the dataset is first split into the three occurring types of violation (no violation, lower voltage band violation, combination voltage band and branch load violation). Then, each set is split separately according to the above proportions, and the resulting subsets are combined to form the final training, validation, and test sets. We first train the random forest classifier on the training set. Then, we train the RL agent on all cases classified as violations by the random forest classifier (see Fig. 3). The policy is trained using an actor–critic reinforcement learning algorithm with deep deterministic policy gradient [22], [23]. Training consists of two parts: Collecting observations and training the actor and critic networks. The reinforcement learning algorithm obtains observations by first applying the actor to the observable measurements, obtaining an action. To improve variety in early observations, actions are modified with decreasing amounts of noise before being applied. The action is then applied and the algorithm calculates the power flow and calculates the reward from the full grid state. One observation is the triple of input, action, and reward. The critic is an MLP that is trained to learn the reward from the input and the action. For the training of the actor, the observations and output of the actor are connected to the critic network. Then, without updating the critic weights, the actor is trained with gradient descent, maximizing the output of the critic. We refer the reader to [22] for further details. The reward function penalizes violations of operating parameters, or, if none exist, rewards smaller curtailment:  p √ Serr ) if violation −ν( V√ err /0.2 + P C C (5) r= i∈V Pg,i +Pd,i else 1 − q P C,max C,max

Preprocessing (Full Dataset)

RL Training (Filtered Dataset)

Predict Violation and Filter Cases

Rand. Forest

e Rep

Power Flow

at

Actor

Action

Calculate Reward

Critic

Observation

Fig. 3. Training setup. A Random Forest filters non-violations; the RL loop takes cases, evaluates actions and rewards, and updates the actor-critic weights.

the complete system state. This information is not required during deployment: once trained, the random-forest classifier and actor receive only the partial, noisy observation vector from the observable buses. To allow iterative correction, the resulting state can be reevaluated by the actor for multiple control steps. Both during and after training, we apply the model as follows: 1) A new observation is taken from observable buses. 2) The random forest classifier classifies the observation as violation or no violation. • If the observation is classified as no violation, take no action. Wait for next observation and go to 1). 3) The actor generates an action from the observation. 4) Apply the action to the controllable elements of the grid. Wait for the next observation and go to 3). E. Evaluation Metrics We quantify the overall performance using a set of metrics. Specifically, we quantify the reduction in violation magnitude (compared to the baseline with no action), the number of remaining cases with violations, and the amount of curtailment (compared to the OPF) needed to achieve it. For the preclassifier, we also report the number of false negatives and false positives in detecting violations. IV. E XPERIMENTS All results reported here stem from experiments on the dataset based on the low-voltage grid provided by SchleswigHolstein Netz GmbH as described in Section III-A. The hyperparameters of the selected runs are λ = 1.07, ν = 0.98, with 5 layers of width 64 and 200k control steps for the training without pre-classification; λ = 1.14, ν = 0.77, with 8 layers of width 32 and 500k control steps for the normal dataset; and λ = 1.18, ν = 0.33, with 5 layers of width 128 and 500k control steps for the mismatch dataset. A. Implementation We use the scikit-learn implementation of a random forest classifier for the pre-classification. The actor-critic model and training is implemented using PyTorch and TorchRL [24]. The training data is stored in pypower [25] case format.

Increase in curtailment [%]

+93.8%

Relative Increase in Curtailment vs OPF +86.6%

TABLE I N UMBER OF FALSE POSITIVE AND NEGATIVE PREDICTIONS OF VIOLATIONS BY THE RANDOM FOREST CLASSIFIER .

+87.0%

+82.5%

75 50

+38.5%

Noise Factor False Pos False Pos (%) False Neg False Neg (%)

25 0

no pre-classifier

no noise

half noise Scenario

full noise

mismatch

Fig. 4. Excess curtailment of actors, relative to OPF curtailment. Different scenarios, each shows the total over cases with and without violations.

)

min. Voltage min Vm, i [p.u.] i

50

75

100

125 100

)

50

50

75

100

25

0

25

50

75

Curtailment [%]

Mismatch 4 0.1 % 119 9.2 %

Base No Noise Half Noise Noise Mismatch No Pre-Classifier

Mean Violation [p.u.] 3.0E-3 3.2E-5 3.3E-5 3.5E-5 4.6E-4 4.6E-4

Violation Magnitude Reduction 0% 98.9% 98.9% 98.8% 79.6% 90.9%

Number of Violations Remaining 726 60 65 70 301 176

(

Flexibility

75

(

25

Curtailment [%]

1.0 49 1.4 % 15 1.2 %

i

Curtailment [%] mismatch min. Voltage min Vm, i [p.u.] i

0

)

Pd,C i + Pg,C i [kW]

(

) ( )

25

Curtailment [%]

150

Curtailment [%] full noise

(

) (

0

175

0.5 20 0.6 % 10 0.7 %

TABLE II N UMBER AND MAGNITUDE OF VIOLATION IN DIFFERENT SCENARIOS .

no noise

min. Voltage min Vm, i [p.u.] i

)

min. Voltage min Vm, i [p.u.] i

Curtailment [%] half noise

1.000 0.975 0.950 0.925 0.900 0.875 0.850 0.825

min. Voltage min Vm, i [p.u.] i

(

opf

Lower Voltage Threshold Vmmin

min. Voltage min Vm, i [p.u.] i

baseline

1.000 0.975 0.950 0.925 0.900 0.875 0.850 0.825

0.0 7 0.2 % 4 0.3 %

100

Fig. 5. Lowest Voltage and curtailment, different solutions and scenarios. On the x axis, the plot shows the amount of curtailment (relative to maximum flexibility). The y axis shows the smallest voltage in the overall grid. Colored by total amount of flexibility for visual comparison.

We train on the training subsets of both the original dataset and the mismatch dataset without measurement noise. We perform a hyperparameter search with random exploration for a total of 20 runs and evaluate the agents on the validation subset of the dataset they were trained on. Then, we select the best run based on the magnitude of violations after applying the action and the total amount of power curtailed. For the evaluation, we run both original and mismatch agents on the evaluation subset of the original dataset. In the presence of measurement noise, we modify the observations, applying the action to the noise-free ground truth. This results in directly comparable metrics across evaluations. B. Results We first look at the effect of the pre-classifier, then different levels of noise and finally consider grid model mismatch. 1) Two-step Control Model: With the same number of training steps, the slightly-modified end-to-end approach [17] curtails more total power than the two-step approach (cf. Fig. 4) while reducing the total violation by less and resolving fewer cases with violations completely (cf. Table II). There are both unnecessary control actions without violations and insufficient curtailment in operating points with large violations. 2) Results with Accurate Measurements and Model: First, we look at the best-case results obtained with partial observability. Figure 4 shows that with accurate partial information,

the actor curtails less than twice the amount that the OPF does. With this, the actor is able to reduce the amount of violation by almost 99%, solving most cases (see Table II). Figure 5 gives a more visual overview of this result, showing a large group of cases that are solved with a small margin to the defined minimum voltage, and a bigger group with varying margins. Other runs with other random seeds that we considered provided alternative trade-offs, either increasing excess curtailment and decreasing remaining violations, or vice versa. 3) Measurement Noise: Second, we look at the accuracy of the approach in the presence of noise. Table I shows the performance of the random forest classifier on the test set for different noise levels. While its accuracy drops in the presence of faulty measurements, the overall number of misclassifications is very small. Table II shows that the combined model reduces the violations by 98.9 %, and is robust to noise. 4) Grid Model Mismatch: In the presence of model mismatch, the pre-classifier has substantially more false negatives (see Table I) and Fig. 4 shows a smaller total curtailment for the control model. In Table II, we see a spike in the number of cases with violations after the control action is applied, but the overall magnitude of violations is still reduced by almost 80 %. This indicates that the mismatch is skewed towards producing higher minimum voltages in the grid, biasing the control model toward smaller curtailment. Figure 5 confirms this, with the bulk of control actions having slightly less curtailment. We also see two more extreme clusters, one with a bigger overcurtailment and one with a number of cases that have no curtailment and relatively strong constraint violations. Overall, the vast majority of misclassifications is close to the threshold, as is the vast majority of remaining violations. V. C ONCLUSION This work presents an improved two-step control model and evaluates its robustness to measurement noise and grid model mismatch, with the overall goal of improving the real-world viability of ML-based congestion management.

In the experiments, the two-step model clearly outperforms an end-to-end model for the same number of training steps. Although additional training or a more extensive hyperparameter search could likely improve the end-to-end baseline, the results show a clear computational advantage for the two-step approach. Since such models must be deployed at scale and periodically retrained, reduced training cost is an important factor for practical applicability. The results further show that reinforcement-learning-based curative curtailment can remain effective in low-voltage distribution grids even when only a small share of buses is observable and controllable. By separating violation detection through a random-forest pre-classifier from curtailment control through an actor-critic controller, the proposed framework reduces the complexity of the learning task while maintaining strong control performance. On a real low-voltage grid, the controller reduces total violation magnitude by 98.9% under accurate grid parameters and achieves almost identical performance under measurement noise. This indicates that realistic measurement uncertainty is not the main limiting factor for edge-based congestion management. By contrast, grid-model mismatch has a stronger impact. Under mismatched grid parameters, the violation-magnitude reduction decreases to 79.6%, and the number of remaining violation cases increases substantially. Nevertheless, the controller still reduces violation severity compared to the uncontrolled baseline. Thus, imperfect model knowledge does not invalidate learning-based curtailment, but shifts the practical objective from exact optimality toward robust mitigation. For DSOs, this distinction is important: a controller that substantially reduces unseen violations may still be valuable in operation if model validation and safety safeguards are included. Overall, the case study shows that some likely real-world errors do not lead to catastrophic performance degradation, while the reduced computational cost improves the practical applicability of the proposed control framework. Future work should extend the robustness analysis to additional sources of mismatch, including topology uncertainty, changing load and generation profiles, and errors in the assumed availability of flexible units. Further important directions include online model calibration or adaptive retraining, as well as evaluation on multiple grid topologies and, where possible, with field data or hardware-in-the-loop experiments to assess reliability under operational constraints, communication delays, and regulatory requirements for curtailment. R EFERENCES [1] K. K. Mehmood et al., “A review of congestion management methods for power distribution networks: Current practices and future challenges,” Applied Energy, vol. 407, p. 127342, 2026. [2] F. Capitanescu et al., “State-of-the-art, challenges, and future trends in security constrained optimal power flow,” Electric Power Systems Research, vol. 81, no. 8, pp. 1731–1741, 2011. [3] M. B. Cain, R. P. O’neill, A. Castillo et al., “History of optimal power flow and formulations,” Federal Energy Regulatory Commission, vol. 1, pp. 1–36, 2012. [4] D. Bienstock and A. Verma, “Strong NP-hardness of AC power flows feasibility,” Operations Research Letters, vol. 47, no. 6, pp. 494–501, 2019.

[5] S. Zhan, J. Morren, W. van den Akker, A. van der Molen, N. G. Paterakis, and J. Slootweg, “Robustness assessment of primal–dual gradient projection-based online feedback optimization for real-time distribution grid management,” Electric Power Systems Research, vol. 242, p. 111468, 2025. [6] C. Li et al., “Optimal power flow in a highly renewable power system based on attention neural networks,” Applied Energy, vol. 359, p. 122779, 2024. [7] E. Marchesini et al., “Rl2grid: Benchmarking reinforcement learning in power grid operations,” 2025. [Online]. Available: https://arxiv.org/abs/2503.23101 [8] J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in International Conference on Machine Learning. Pmlr, 2017, pp. 22–31. [9] D. Cao et al., “Deep reinforcement learning based approach for optimal power flow of distribution networks embedded with renewable energy and storage devices,” Journal of Modern Power Systems and Clean Energy, vol. 9, no. 5, pp. 1101–1110, 2021. [10] Z. Yan and Y. Xu, “Real-time optimal power flow: A lagrangian based deep reinforcement learning approach,” IEEE Trans. Power Syst., vol. PP, pp. 1–1, 04 2020. [11] T. Wu, A. Scaglione, and D. Arnold, “Constrained reinforcement learning for stochastic dynamic optimal power flow control,” in 2023 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 2023, pp. 1–5. [12] A. R. Sayed, C. Wang, H. I. Anis, and T. Bi, “Feasibility constrained online calculation for real-time optimal power flow: A convex constrained deep reinforcement learning approach,” IEEE Trans. Power Syst., vol. 38, no. 6, 2023. [13] A. R. Sayed, X. Zhang, G. Wang, C. Wang, and J. Qiu, “Optimal operable power flow: Sample-efficient holomorphic embedding-based reinforcement learning,” IEEE Trans. Power Syst., vol. 39, no. 1, pp. 1739–1751, 2024. [14] Z. Wu, M. Zhang, S. Gao, Z.-G. Wu, and X. Guan, “Physics-informed reinforcement learning for real-time optimal power flow with renewable energy resources,” IEEE Trans. Sustain. Energy, vol. 16, no. 1, pp. 216– 226, 2025. [15] Y. Zhou et al., “A data-driven method for fast ac optimal power flow solutions via deep reinforcement learning,” Journal of Modern Power Systems and Clean Energy, vol. 8, no. 6, pp. 1128–1139, 2020. [16] Z. Yi, X. Wang, C. Yang, C. Yang, M. Niu, and W. Yin, “Realtime sequential security-constrained optimal power flow: A hybrid knowledge-data-driven reinforcement learning approach,” IEEE Trans. Power Syst., pp. 1–16, 04 2023. [17] H. Wolf et al., “End-to-end reinforcement learning of curative curtailment with partial measurement availability,” in 2024 IEEE PES Innovative Smart Grid Technologies Europe (ISGT EUROPE), 2024, pp. 1–6. [18] S. Meinecke et al., “Simbench—a benchmark dataset of electric power systems to compare innovative solutions based on power flow analysis,” Energies, vol. 13, no. 12, p. 3290, 2020. [19] T. Jouini and Z. Sun, “Distributed learning for optimal allocation of synchronous and converter-based generation,” 06 2021, pp. 386–391. [20] A. Wen, B. Wen, J. Li, and J. Xu, “Heterogeneous graph neural network with local and global message passing for ac-optimal power flow solutions,” Applied System Innovation, vol. 9, no. 1, p. 18, 2026. [21] L. Dabush, A. Kroizer, and T. Routtenberg, “State estimation in partially observable power systems via graph signal processing tools,” Sensors, vol. 23, no. 3, p. 1387, 2023. [22] H. Tan, “Reinforcement Learning with Deep Deterministic Policy Gradient,” in 2021 International conference on artificial intelligence, big data and algorithms (CAIBDA). IEEE, 2021, pp. 82–85. [23] E. H. Sumiea et al., “Deep deterministic policy gradient algorithm: A systematic review,” Heliyon, vol. 10, no. 9, 2024. [24] A. Bou et al., “TorchRL: A data-driven decision-making library for PyTorch,” in The Twelfth International Conference on Learning Representations, 2024. [25] R. e. a. Lincoln, “PYPOWER: Power flow and optimal power flow (OPF) solver.” [Online]. Available: https://pypi.org/project/PYPOWER/