Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs Jose E. Aguilar Escamilla1 , Lingdong Zhou1 , Xiangqi Zhu1 , and Huazheng Wang1
arXiv:2605.12462v1 [cs.AI] 12 May 2026
1
School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR, USA May 13, 2026
Abstract Extreme weather and volatile wholesale electricity markets expose residential consumers to catastrophic financial risks, yet demand response at the distribution level remains an underutilized tool for grid flexibility and energy affordability. While a demand-response program can shield consumers by issuing financial credits during high-price periods, optimizing this sequential decision-making process presents a unique challenge for reinforcement learning despite the plentiful offline historical smart meter and wholesale pricing data available publicly. Offline historical data fails to capture the dynamic, interactive feedback loop between an electric utility’s pricing signals and customer acceptance and adaptation to a demand-response program. To address this, we introduce DR-Gym, an open-source, online Gymnasium-compatible environment designed to train and evaluate demand-response from the electric utility’s perspective. Unlike existing device-level energy simulators, our environment focuses on the market-level electric utility setting and provides a rich observational space relevant to the electric utility. The simulator additionally features a regime-switching wholesale price model calibrated to real-world extreme events, alongside physics-based building demand profiles. For our learning signal, we use a configurable, multi-objective reward function for specifying diverse learning objectives. We demonstrate through baseline strategies and data snapshots the capability of our simulator to create realistic and learnable environments.
Keywords: demand response, electricity markets, reinforcement learning, risk-aware learning, building energy simulation, Gymnasium
1
Introduction
Wholesale electricity markets are increasingly vulnerable to extreme price volatility. California Independent System Operator (CAISO) day-ahead locational marginal prices (LMPs) regularly spike during heat events, while ERCOT prices reached the system-wide cap of $9,000/MWh for over 72 consecutive hours during Winter Storm Uri in February 2021, leaving some residential customers facing bills exceeding $10,000 [11]. Although Texas has banned the residential electricity plan which is directly related to LMPs in wholesale market after this winter storm, residential customers across the U.S. still face risks of high electricity bill when wholesale LMPs are pushed high [1]. This creates an urgent need to develop a risk-resilient market mechanism that protects customers from extreme financial outcomes.
1
A demand response program can potentially address this by pooling demand-response (DR) [28] capacity across households when the grid is in need, and then issuing financial credits to participating customers during high-price periods [28]. In exchange for curtailing consumption in demand response events, customers receive a bill credit; the electric utility improves electricity affordability while simultaneously smoothing the load curve. This market-design concept is well-established in the DR literature [4, 17], but optimizing the credit-issuance policy across heterogeneous customers and time-varying prices is still challenging and is a sequential decision-making problem well-suited for reinforcement learning (RL) [26]. External Data
DemandResponseEnv
Reward Grid Relief & CVaR Penalty
CityLearn Customer Response
Building Simulator
Action Financial Credit Weather Data
Market Data
GridStress Calculator
Reward Computer
Market Model
OpSavings Generator
RL Agent
Observation Market & Grid State
Figure 1: Overview of simulator architecture. From left to right, we outline the external data we ingest, the various modules generating data, and the RL interaction interface. Despite this need, existing RL environments fail to capture the market-level dynamics required for electric utility operations [26, 28]. Open-source environments such as CityLearn [27] focus on device-level scheduling, such as HVAC set-points and battery dispatch for individual buildings. Theoretical models of electric utility level pricing [20, 4] typically rely on risk-neutral objectives and assume perfect or static compliance, failing to model the stochastic behavioral fatigue that occurs when consumers face repeated DR interventions. To bridge this gap, we present DR-Gym (Demand-Response Gymnasium), an online, Gymnasiumcompatible [9] environment for training and evaluating electric utility demand-response policies. DR-Gym is designed as a general-purpose testbed : it supports a configurable reward formulation out of the box to specify a diverse range of learning objectives. The environment features a Markov regime-switching wholesale price model calibrated to real-world extreme weather events, physics-based building demand profiles from CityLearn’s EnergyPlus/ResStock dataset, and a heterogeneous customer response model with dynamic behavioral fatigue [20]. Our contributions are two-fold : 1. We present the first standardized Gymnasium environment for electrical utility-level demandresponse (DR) via credit issuance, providing a novel reinforcement learning testbed. We include experiments and data snapshots to further validate the realism and learnability of DR-Gym. 2. Secondly, we implement a configurable reward function to allow a diverse specification of 2
learning objectives. This allows our simulator to entice a learning agent to balance revenue optimization, grid stability, and consumer protection through a myriad of “plug-and-play” risk-aware metrics [12].
2
Background and Related Work
2.1
Markov Decision Process Formulation
We model the electric utility’s decision problem as an episodic Markov Decision Process (MDP) M = ⟨S, A, P, R, γ⟩ [24]. At each timestep t, the agent observes state st ∈ S ⊂ R32 (hourly mode) and selects action at = ct ∈ A = [0, cmax ]. The environment transitions to st+1 according to the simulator dynamics P , and the agent receives hP rewardirt = R(st , at , st+1 ). The goal is to maximize T t expected cumulative discounted reward, E t=0 γ rt .
2.2
Related Work
RL for electricity markets. RL has been applied to supply-side generator bidding since the early 2000s [21], but demand-side electric utility RL remains comparatively understudied. Antonopoulos et al. [2] survey 160+ papers on AI/ML for demand-side response and identify RL as the dominant approach for electric utility-level incentive dispatch, while noting that few environments support the multi-building setting with explicit tail-risk objectives. Similarly, Vázquez-Canteli and Nagy conducted an extensive overview of reinforcement learning for demand-response applications in smart grids. Despite the good suitability of reinforcement learning for demand-response, the authors report an extensive lack of dynamic environments for learning and benchmarking agents that do not rely on demand-independent variables, which diminishes realism. RL simulation environments. While other works have presented simulators for demandresponse, these works often focus on specific aspects of the electrical grid rather than the market distribution level. van Tilburg et al.’s work is closest to ours, which introduces an incentive-based DR program for multi-agent RL (MARL-iDR). Our work is distinct from MARL-iDR by being an open-source testbed for single-agent RL with validated building loads, price model, user fatigue, and configurability, rather than just a specific use-case as in MARL-iDR. Other works related to ours include: CityLearn1 [27] provides a Gym interface for multi-building Demand response with HVAC and battery scheduling, targeting device-level control. Sinergym [6] is a simulator for Building Energy Optimization (BEO) which provides training and running controllers for BEO with ML and RL. Grid2Op [19] is a power network simulation package that allows realistic sequential network operation optimization with RL. Finally, Pymgrid [15] is a microgrid simulator for self-contained electrical grids, allowing RL to control a multitude of these systems. Electricity price modeling. Weron [29] establishes the three-component structure of realistic electricity price models: a seasonal/periodic base, an AR(p) persistence component, and a separate spike process. Huisman and Mahieu [16] showed that Markov regime-switching models capture spike clustering far better than i.i.d. jump processes, which tend to produce unrealistically isolated single-hour spikes. Both findings directly inform the DR-Gym price model. Demand-response customer modeling. Faruqui and Sergici [10] meta-analyze 15 largescale residential DR pilots and find acceptance rates of 20–72% and peak demand reductions of 3–20%, with substantial heterogeneity across household types. Moghaddam et al. [20] formalize 1
DR-Gym is complementary to CityLearn: we use CityLearn’s building demand profiles as a read-only data source while adding a market-level electric utility layer with explicit consumer-protection objectives and a heterogeneous behavioral customer model absent from prior environments.
3
customer participation as a logistic function of incentive level, the functional form adopted in our acceptance model. We use these works to calibrate and validate our simulator.
3
DR-Gym Simulator Design
3.1
Problem Formulation
At each hour t, the electric utility observes the current wholesale price pt (determined before the credit decision), issues a per-kWh credit ct , and observes which of N buildings accept the credit and how much load they curtail. The electric utility earns a margin on the load it serves while paying credits to accepting customers. Formally: PN • Revenue: Rt = (pretail − ct − pt ) · Dt , where Dt = i=1 di,t is post-reduction aggregate demand (kWh) and pretail = $0.15/kWh is the fixed retail rate. • Consumer cost: Ct = (pretail − ct ) · Dt ; i.e., consumers pay the retail rate net of any credit. • Budget constraint: The electric utility has a daily operational budget B (drawn from seasonal savings). Total credits paid cannot exceed B per day; unspent budget rolls over at rate 0.95. The action space is A = [0, 0.10] $/kWh (continuous), and the episode length is configurable (default: 24 hours = 1 day).
3.2
Observation Space
Table 1: Observation space for hourly mode (32 dimensions). normalized to [0, 1]. For more details, see Appendix A.
Stress indicators are sigmoid-
Index
Feature
Range / Notes
0 1 2 3 4–7 8 9 10 11 12 13 14 15–24 25–29 30 31
Hour of day Day of week Aggregate demand (kWh) Wholesale price ($/kWh) Price forecast (4 steps) Temperature (°C) Demand stress Price stress Thermal stress Overall stress Budget remaining ($) Last credit ($/kWh) Building loads (kWh) Demand history (kWh) Cumulative credits ($) Day within episode
{0, . . . , 23} {0, . . . , 6} Post-reduction [0.02, 9.50] TOU + AR(1) projection From building simulator Sigmoid [0, 1] Sigmoid [0, 1] Sigmoid [0, 1] Weighted average [0, 1] Daily operational budget Previous action 10 buildings (padded/truncated) 5-step aggregate demand window Episode total Multi-day episodes
4
The 32-dimensional hourly observation vector is detailed in Table 1. It encodes time context, energy market signals, grid stress indicators, budget state, per-building loads, and a demand history window.
3.3
Component Architecture DemandResponseEnv (Gymnasium Environment)
External Data Sources CityLearn Building Energy Data
BuildingSimulator Building Energy Data
CustomerResponse Acceptance & Reduction
Weather Data Temperature, Solar
GridStressCalculator Stress Indicators
Reward Computer Multi-objective + CVaR
Market Data CAISO/ERCOT/PJM
MarketModel Price Generation
OpSavingsGenerator Budget Allocation
Action: Credit $/kWh
RL Agent Policy (e.g., PPO, SAC)
Observation 32-dim
Figure 2: Detailed simulator architecture. We outline each part of our simulator as well as the information flow among them. DR-Gym is built from six loosely coupled components that can be individually configured or replaced (Figure 2). Each component exposes a clean interface, enabling researchers to swap in alternative models—e.g., a real-data price feed, a different customer behavioral model, or a physics-based grid stress calculator—without modifying the agent interface.
3.4
Building Demand Simulator
Building demand profiles are sourced from CityLearn [27] via the citylearn challenge 2022 phase 1 dataset2 , which contains EnergyPlus-simulated [7] hourly load profiles for residential buildings derived from NREL ResStock archetypes [30]. The simulator replays these pre-computed profiles sequentially, extracting non shiftable load per building and outdoor temperature from the weather module. ResStock’s whole-building simulations capture appliance-level loads, HVAC cycling, and seasonal variation that generic sinusoidal demand models cannot reproduce. Demand-Persistence Feedback. A key closed-loop feature is demand-persistence feedback: credits issued at step t reduce building loads for subsequent hours via per-building exponential decay multipliers mi,t ∈ (0, 1]: mi,t+1 = mi,t · γ + (1 − δi,t )(1 − γ),
(1)
where γ ∈ [0, 1] is the decay rate (demand feedback decay = 0.9 gives a ≈ 7-hour half-life, consistent with LBNL residential DR elasticity data [13]) and δi,t = ri,t /dbase i,t ∈ [0, 0.5] is the fractional reduction. Customer acceptance is always computed against the raw baseline demand dbase i,t , not the multiplier-adjusted demand, preventing compounding. 2
We emphasize that our simulator allows using any other building demand profile dataset outside of CityLearn’s.
5
3.5
Wholesale Market Model
The price model decomposes hourly electricity prices into three components [29]: pt = pTOU (ht ) + ξt + st ,
(2)
where ht ∈ {0, . . . , 23} is the hour of day. Time-of-use base pTOU (ht ): A step function with off-peak, shoulder, and peak tiers reflecting typical retail tariff structure. AR(1) persistence ξt : Temporal correlations are modeled as ξt = ρ · ξt−1 + εt , where εt ∼ N (0, (σε · hht )2 ), ρ = 0.9, and σε = 0.02 $/kWh. Following Weron [29], the noise is heteroscedastic: the multiplier hht scales variance by hour of day, ranging from 1.0 during overnight hours to 1.8 during morning ramp (7–9 AM) and evening peak (18–20 PM), reproducing the 40–80% higher peak-hour volatility documented in real markets. The coefficient ρ = 0.9 reflects the typical AR(1) persistence range of 0.7–0.95 found in mature day-ahead markets [29]. Regime-switching spikes st : A two-state Markov chain {Normal, Spike-storm} captures the temporal clustering of electricity price spikes [16]. Transition probabilities are: Pr(enter spike) = λ0 · 1[peak hour] · (1 + δT ),
(3)
with base entry rate λ0 = 0.005/h, a 2× multiplier during on-peak hours, and a temperature boost δT = 0.03 at extreme temperatures (above 35°C or below 0°C). The exit probability is λexit = 0.15/h (expected storm duration ≈ 7 hours), consistent with empirical spike-storm durations in European markets [16]. During a spike storm, the magnitude multiplier is log-normal: mt ∼ exp(N (0.4, 0.8)), and prices are capped at pmax = $9.50/kWh, matching the ERCOT system-wide offer cap [11]. A four-step ahead price forecast (indices 4–7 in the observation) is computed as: p̂t+h = pTOU (ht+h ) + ρh+1 ξt ,
h = 0, 1, 2, 3.
(4)
Price-demand elasticity. To close the demand–price feedback loop, aggregate demand reductions lower the next-step clearing price via a configurable elasticity term: red padj = pt − λ · ξt−1 , t
(5)
red is an EWMA of recent aggregate reductions (kWh) with decay α = 0.8, and λ is the where ξt−1 e price-elasticity coefficient ($/kWh per kWh; default λ = 0, disabled). A small nonzero λ (≈ 0.001) reflects the empirical finding that large portfolio-scale DR events in ERCOT can measurably shift clearing prices in subsequent hours.
3.6
Heterogeneous Customer Response
Following Moghaddam et al. [20], customer acceptance of a credit offer is modeled as a logistic function of the credit level. We extend this to a heterogeneous population with four empiricallymotivated archetypes [10], detailed in Table 2. The effective acceptance probability for building i of type k at credit ct is: paccept = p̄k · fi,t · σ(−κk (ct − 0.05)) , i,t
(6)
where p̄k is the base acceptance rate, κk is the credit sensitivity, σ(·) is the logistic function, and fi,t ∈ [0.3, 1] is a fatigue factor that decays at rate 0.1 per consecutive activation and recovers on non-activation steps [2]. If customer i accepts, their load is reduced by ri,t ∼ N (µk , σk ) (demand reduction mean/std per type). 6
Table 2: Customer archetype parameters. Base acceptance, reduction mean, and credit sensitivity are calibrated to the ranges reported in Faruqui and Sergici [10]. Proportions sum to 1.
Type
Proportion
Base accept
Reduction
Sensitivity
30% 20% 35% 15%
0.80 0.85 0.65 0.40
20% 18% 12% 8%
3.0 1.5 2.0 1.0
Price-sensitive Eco-conscious Neutral Reluctant
3.7
Grid Stress Calculator
Three sigmoid-based indicators normalize grid conditions to [0, 1]: σdemand = σ(Dt − D∗ ) ,
D∗ = 100 kWh,
σprice = σ(20(pt − 0.25)) , 0 − Tt Tt − 35 + max 0, , σthermal = max 0, 10 10
(7) (8) (9)
where Tt is the outdoor temperature. Overall stress is a weighted average with weights (wD , wP , wT ) = (0.3, 0.5, 0.2) set for our experiments.
3.8
Operational Budget
The electric utility’s daily credit budget is drawn from a stochastic seasonal model: Bday ∼ 2 ) with µ N (µB , σB B = $100, σB = $20, modulated by a cosine-squared seasonal factor (higher in summer and winter, lower in spring and fall). The default µB is calibrated to the simulator’s internal parameters: for N = 50 buildings with mean hourly load ≈ 2 kWh/h and weightedaverage acceptance rate ≈ 0.65, the daily credit expenditure under moderate stress (6 peak hours at c = 0.05 $/kWh) is ≈$20, while a sustained spike storm (10 hours at c = 0.08 $/kWh) requires ≈$52. Setting µB = $100 provides meaningful headroom above typical-day needs while constraining the agent well below the theoretical daily maximum of $240, creating a non-trivial budget-allocation problem. This range is broadly consistent with residential DR program incentive levels reported by LBNL, where direct load control capacity payments of $0.3–4.6/kW-month and event-based performance payments of 2–40 ¢/kWh imply per-portfolio daily costs on the order of $20–80 for a 50-building cohort [3]. Unspent budget rolls over at r = 0.95 per day across multi-day episodes. Users can select a custom regime for the operational budget.
3.9
Reward Function
As mentioned before, we implement a multi-objective configurable reward function that allows the user to balance revenue, consumer welfare, and grid stress. The per-step reward balances these objectives with an optional risk-aware penalty: Rt Ct overall rt = λs wR · − wC · − wσ · σt − wrisk · ∆riskt , (10) N N where N is the number of buildings, λs = 0.01 is a scale factor, and (wR , wC , wσ , wrisk ) are weights corresponding to the utility’s revenue, consumer cost, grid stress, and an optional risk-aware term, 7
respectively. The incremental risk-aware term ∆riskt = risk(b1:t )−risk(b1:t−1 ) measures the change in risk of the running consumer bill vector b1:t , providing a per-step risk signal to the agent. We set (wR , wC , wσ , wrisk ) = (0.3, 0.5, 0.2, 0.3), with Conditional Value-at-Risk as our risk-aware measure for our experiments (see Section 5). We emphasize that the user can use any risk-aware measure as “plug-and-play.”
4
Simulator Validation
We assess the realism of each simulator component by comparing data outputs against real-world benchmarks. We also present representative data snapshots to illustrate the signals available to an RL agent. For wholesale prices, we compare against CAISO day-ahead LMPs for the NP15 North hub, accessed via the gridstatus library [5]. For building demand, we compare against the underlying CityLearn/ResStock statistics.
4.1
Wholesale Price Dynamics
Figure 3: Wholesale price dynamics produced by the DR-Gym market model. (a) A representative 7-day trajectory: red shading marks regime-switching spike storms; orange shading marks on-peak hours (16–21 h). Multi-hour storm clustering is clearly visible and would be absent from an i.i.d. jump process. (b) Log-scale price distribution, showing the heavy right tail characteristic of real electricity markets. Figure 3 shows the key price statistics generated by the simulator. The AR(1) model produces a first-order autocorrelation ρ̂1 ≈ 0.85, consistent with the range of 0.78–0.93 reported for CAISO, PJM, and Nord Pool in Weron [29]. The daily periodicity is captured by the TOU base, and the regime-switching spike model produces multi-hour price storms (mean duration ≈ 7 hours) rather than isolated single-hour spikes, qualitatively consistent with real spike clustering in ERCOT realtime data. The simulator’s market model is parameterized to match ERCOT day-ahead market statistics; see Section 4.1.1 for validation. The AR(1) model implements hour-of-day heteroscedastic noise multipliers following Weron [29], addressing the variance modulation observed during morning ramp and evening peak periods.
8
4.1.1
ERCOT Day-Ahead Calibration
Figure 4: Market model validation against ERCOT day-ahead market statistics. (a) Simulator 24-h TOU median profile versus ERCOT literature benchmark ranges [11]. (b) Price-minus-TOU residual distribution (heavy-tailed due to regime-switching spikes); Gaussian overlay shows the configured AR(1) noise component (σ = 0.020 $/kWh), within the literature range [0.015, 0.025] $/kWh [29]; green bands mark the literature range. (c) Hourly spike rate (fraction of hours with price >$1/kWh); note that price crossings overcount storm events since one storm may cause multiple level-crossings. (d) Seven-day Uri-analog trajectory generated with elevated spike entry probability and temperature boost; red shading marks spike-state hours (≥$1/kWh). Figure 4 validates the simulator’s market model against ERCOT day-ahead market statistics reported in the literature. We generate a 4,380-step (≈6-month) trajectory at N = 50 buildings and extract price statistics offline, requiring no live API access. TOU base (Panel a). The simulator’s three-tier TOU profile falls within the off-peak ($0.05– 0.09/kWh), shoulder ($0.10–0.14/kWh), and peak ($0.14–0.22/kWh) benchmark ranges reported for ERCOT day-ahead LMPs [29, 11]. AR(1) noise (Panel b). The AR(1) noise component is configured with σ = 0.020 $/kWh and ρ = 0.90, within the σ ∈ [0.015, 0.025] $/kWh and ρ ∈ [0.78, 0.93] ranges documented across U.S. wholesale markets [29]. The price-minus-TOU residual distribution is heavy-tailed (Panel b) because it combines both the Gaussian AR(1) component and the regime-switching spike component; the Gaussian overlay shows the configured AR noise contribution. Spike frequency (Panel c). The regime-switching model generates sustained spike storms in which prices frequently exceed $1/kWh during the storm duration (Panel c). Because one storm 9
causes multiple price-level crossings, the raw crossing count exceeds the literature estimate of 2– 5 discrete spike events per month [16]; the default spike exit prob= 0.15 implies mean storm duration ≈ 7 hours, consistent with real ERCOT spike clustering. Uri-analog stress test (Panel d). Running a seven-day episode with elevated spike-entry probability (spike entry base=0.08) and temperature boost (temp spike boost=0.15) generates multi-hour price storms qualitatively consistent with Winter Storm Uri dynamics, confirming the model’s capacity for extreme-event training scenarios.
4.2
Building Demand
Figure 5: Aggregate building demand profile: mean and standard deviation across 50 buildings and 40 episodes. Shaded regions mark the morning ramp (6–9 h) and evening peak (17–21 h). The bimodal daily pattern and inter-episode variability are well-reproduced by the EnergyPlus/ResStock source data. Figure 5 shows the aggregate demand profile generated by the simulator. The CityLearn ResStock profiles exhibit the expected residential demand pattern: morning ramp (6–9 AM), midday dip, and evening peak (5–9 PM), with substantial inter-building variability. We note that the building demand is sourced from physics-based EnergyPlus simulations, which have been validated against AMI smart meter measurements [30].
4.3
Customer Response
The acceptance probability function (Equation 6) is calibrated so that at credit c = $0.05/kWh, the weighted average acceptance rate across the four archetypes is approximately 0.65, consistent with the 20–72% range reported across 15 DR pilot programs [10]. The demand reduction magnitudes (8– 20% by archetype) fall within the empirically-reported 3–20% range for critical peak pricing pilots. The fatigue mechanic (declining acceptance under repeated activations, Figure 6b) is supported qualitatively by Antonopoulos et al. [2].
10
Figure 6: Customer response model validation. (a) Acceptance probability curves for each of the four archetypes as a function of credit level (Equation 6); weighted average acceptance at c = $0.05/kWh is ≈ 0.65, consistent with empirical DR pilot ranges [10]. (b) Fatigue decay: acceptance factor over consecutive daily activations for each archetype, illustrating the decliningparticipation dynamic captured by the fatigue mechanic.
5
Learning Experiments
The primary goal of these experiments is to demonstrate that DR-Gym provides a learnable environment: an RL agent can discover policies that improve on hand-designed heuristics, establishing the environment as a productive testbed for further algorithm development. We do not claim state-of-the-art performance; rather, we use these results to validate that the environment design— multi-objective reward, regime-switching prices, heterogeneous customers— produces a challenging but tractable optimization problem.
5.1
Experimental Setup
All experiments use hourly mode with N = 50 buildings, one-day episodes, and a fixed random seed for reproducibility. The PPO agent is trained using Stable-Baselines3 [22], a popular RL algorithm library, with the following hyperparameters: clip ratio ϵ = 0.2, entropy coefficient 0.01, 2048 timesteps per rollout, 10 optimization epochs per update, and learning rate 3× 10−4 . Training runs for 2×106 environment steps (≈ 80,000 episodes). We select Conditional Value-at-Risk (CVaR) to demonstrate the risk-aware feature of our multi-objective reward. We refer the reader to Garcia and Fernandez [12] for more on risk-aware measures. We include further experiments on the riskawareness feature of our simulator in Appendix C.2, where we reproduce the “CVaR-reward” trade-off [23].
5.2
Baselines
We compare PPO against four baseline policies from the literature: • NoCreditPolicy: Always issues ct = 0 (lower-bound reference) [18]. • UniformCreditPolicy: Always issues ct = 0.05 $/kWh [10]. • RuleBasedPolicy: Issues cmax = 0.10 when price stress > 0.5, else ct = 0 [14]. 11
• BudgetAwareRulePolicy: Same as rule-based but scales credit proportionally to remaining budget fraction [20]. These baselines span the range from no intervention (No Credit) to budget-reactive threshold policies, providing meaningful comparison points for evaluating learned behavior. See Appendix B.
5.3
Learnability Results
Figure 7: PPO learning and final evaluation. (a) PPO episode reward during training (smoothed). The agent consistently improves over the heuristic baselines within 5 × 105 steps. (b) Final performance comparison over 100 evaluation episodes. PPO achieves the highest mean reward.
Figure 8: Baseline and PPO policy comparison over 50 evaluation episodes. (a) Mean episode reward: higher is better. (b) CVaR0.95 on consumer bills: lower indicates better consumer protection. Error bars show ±1 standard deviation. Figure 7 shows learning convergence and final performance. PPO achieves a mean episode reward that is statistically significantly higher than all four baselines (p < 0.05, Welch’s t-test over 100 evaluation episodes). The NoCreditPolicy achieves positive revenue (the retail–wholesale 12
spread is positive in normal hours) but has the highest risk (CVaR) on consumer bills, because it offers no protection during price spikes. The BudgetAwareRulePolicy provides partial risk reduction but exhausts the budget prematurely during multi-hour spike storms, while PPO learns to reserve credits for the highest-impact intervals (Figure 8). Key findings: 1. Revenue preservation: Despite issuing more credits than the NoCreditPolicy, PPO maintains positive electric utility revenue in all scenarios. 2. Budget efficiency: PPO uses 72–85% of the daily budget (vs. 100% for UniformCredit and < 40% for NoCreditPolicy), suggesting the learned policy is selective rather than exhaustive. 3. CVaR reduction: PPO reduces CVaR0.95 of per-building episode bills by 18–24% relative to NoCreditPolicy across normal, moderate, and high-volatility pricing seeds. These results confirm that the environment design produces meaningful and learnable signal. We anticipate that risk-aware RL algorithms will achieve further improvements in consumer protection at the potential cost of revenue, a trade-off that is a key direction for future work.
6
Limitations and Future Work
Feedback parameter calibration. The demand-persistence decay γ and price-elasticity coefficient λ are set to literature-informed defaults (γ = 0.9, λ = 0); empirical calibration of these parameters to specific DR program data remains future work. Risk-aware algorithm benchmarks. The present experiments use standard PPO as a proofof-concept. A natural and important direction for future work is benchmarking risk-aware algorithms (CVaR-PPO [25], WCSAC, distributional RL [8]) against the risk-neutral baselines, leveraging the risk-aware reward term that DR-Gym provides natively. Customer model calibration. The archetype parameters are drawn from survey ranges [10] rather than fit to a specific dataset. Calibration to Pecan Street or LBNL DR pilot data is a possible direction for a future version.
7
Conclusion
We have presented DR-Gym, an open-source Gymnasium-compatible environment for demandresponse optimization at the market level of an electric utility. The environment combines physicsbased building demand profiles (CityLearn/EnergyPlus/ResStock) [27], a regime-switching price spike model validated against ERCOT day-ahead market statistics, a four-archetype heterogeneous customer model with fatigue, and a configurable multi-objective reward that supports a diverse specification of RL objectives relevant to an electric utility aggregator. DR-Gym fills a gap in the RL environment ecosystem: while other works and similar environments address device-level scheduling, no existing open-source environment targets the marketlevel electric utility setting, acting under price uncertainty with an explicit consumer-protection objective. We designed our environment to be a general testbed for sequential decision-making research, compatible with standard RL libraries (Stable-Baselines3, CleanRL, Ray RLlib) through the Gymnasium interface. Experiments with PPO confirm that the environment is learnable and that principled policies outperform rule-based heuristics on both aggregate reward and consumer tail-risk. We hope DR-Gym will serve as a productive platform for risk-neutral and risk-aware RL, multi-objective policy optimization, and equity-aware DR mechanism design.
13
Broader Impact Statement DR-Gym is designed to study protective demand-response mechanisms that reduce consumer electricity bill volatility during extreme weather events. The direct societal benefit is improved consumer protection for price-vulnerable households. However, an RL agent trained in this environment could, in principle, also be used to optimize electric utility revenue at the expense of consumer welfare if reward weights are misconfigured. We strongly recommend setting reward weights with wC ≥ wR to ensure consumer-welfare priority. The optional risk-aware penalty (see Equation 10) provides an additional structural guard against policies that improve average outcomes at the cost of high tail-risk consumers. Acknowledgments The authors thank the CityLearn team and NREL for making the ResStock dataset publicly available. This work was supported in part by the National Science Foundation under the Electrical, Communications and Cyber Systems (ECCS) division.
References [1] Lori Antini, Jan 2023. URL https://www.eia.gov/todayinenergy/detail.php?id=55139#. [2] Ioannis Antonopoulos, Valentin Robu, Benoit Couraud, Desen Kirli, Sonam Norbu, Aristides Kiprakis, David Flynn, Salvador Elizondo-Gonzalez, Steve Wattam, and Merlinda Andoni. Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review. Renewable and Sustainable Energy Reviews, 130:109899, 2020. doi: 10.1016/j.rser.2020.109899. URL https://www.sciencedirect.com/science/article/pii/ S136403212030191X. [3] Ranjit Bharvirkar, Charles Goldman, Grayson Heffner, and Richard Sedano. Retail demand response in Southwest Power Pool. Technical Report LBNL-1470E, Lawrence Berkeley National Laboratory, 2009. URL https://eta-publications.lbl.gov/sites/default/files/ report-lbnl-1470e.pdf. [4] Severin Borenstein, Michael Jaske, and Arthur Rosenfeld. Dynamic pricing, advanced metering, and demand response in electricity markets. Technical Report CSEM WP 105, Center for the Study of Energy Markets, UC Berkeley, 2002. URL https://escholarship.org/uc/ item/11w8d6m4. [5] Fiona Burlig and Others. gridstatus: Extract data from isos and other energy grid sources, 2024. URL https://github.com/gridstatus/gridstatus. [6] Alejandro Campoy-Nieves, Antonio Manjavacas, Javier Jiménez-Raboso, Miguel MolinaSolana, and Juan Gómez-Romero. Sinergym – a virtual testbed for building energy optimization with reinforcement learning. Energy and Buildings, 327:115075, January 2025. ISSN 0378-7788. doi: 10.1016/j.enbuild.2024.115075. URL http://dx.doi.org/10.1016/j. enbuild.2024.115075. [7] Drury B. Crawley, Linda K. Lawrie, Frederick C. Winkelmann, W. F. Buhl, Y. Joe Huang, Curtis O. Pedersen, Richard K. Strand, Richard J. Liesen, Daniel E. Fisher, Michael J. Witte, and Jason Glazer. EnergyPlus: Creating a new-generation building energy simulation program.
14
Energy and Buildings, 33(4):319–331, 2001. doi: 10.1016/S0378-7788(00)00114-6. URL https: //www.sciencedirect.com/science/article/abs/pii/S0378778800001146. [8] Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 2892–2901, 2018. URL https://arxiv.org/abs/1710.10044. [9] Farama Foundation. Gymnasium: A standard interface for reinforcement learning environments. https://gymnasium.farama.org, 2023. [10] Ahmad Faruqui and Sanem Sergici. Household response to dynamic pricing of electricity: a survey of 15 experiments. Journal of Regulatory Economics, 38(2):193–225, 2010. doi: 10.1007/s11149-010-9127-y. URL https://link.springer.com/article/10.1007/ s11149-010-9127-y. [11] Federal Energy Regulatory Commission (FERC) and North American Electric Reliability Corporation (NERC). The February 2021 cold weather outages in Texas and the south central United States. Technical report, FERC and NERC, November 2021. URL https://www.ferc.gov/news-events/news/ final-report-february-2021-freeze-underscores-winterization-recommendations. [12] Javier Garcia and Fernando Fernandez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437–1480, 2015. URL http://jmlr.org/ papers/v16/garcia15a.html. [13] Girish Ghatikar, Mary Ann Piette, and John H. Dudley. Demand response opportunities and enabling technologies for data centers: Findings from field studies. Technical Report LBNL1004492, Lawrence Berkeley National Laboratory, 2012. URL https://gridintegration. lbl.gov/publications/demand-response-opportunities-and. [14] Haider Tarish Haider, Ong Hang See, and Wilfried Elmenreich. A review of residential demand response of smart grid. Renewable and Sustainable Energy Reviews, 59:166– 178, 2016. ISSN 1364-0321. doi: https://doi.org/10.1016/j.rser.2016.01.016. URL https: //www.sciencedirect.com/science/article/pii/S1364032116000447. [15] Gonzague Henri, Tanguy Levent, Avishai Halev, Réda Alami, and Philippe Cordier. pymgrid: An open-source python microgrid simulator for applied artificial intelligence research. CoRR, abs/2011.08004, 2020. URL https://arxiv.org/abs/2011.08004. [16] Ronald Huisman and Ronald Mahieu. Regime jumps in electricity prices. Energy Economics, 25 (5):425–434, 2003. doi: 10.1016/S0140-9883(03)00041-0. URL https://www.sciencedirect. com/science/article/abs/pii/S0140988303000410. [17] Daniel S. Kirschen and Goran Strbac. Fundamentals of Power System Economics. John Wiley & Sons, 1st edition, 2004. ISBN 0-470-84572-4. URL https://onlinelibrary.wiley.com/ doi/book/10.1002/0470020598. [18] D.S. Kirschen. Demand-side view of electricity markets. IEEE Transactions on Power Systems, 18(2):520–527, 2003. doi: 10.1109/TPWRS.2003.810692. [19] Antoine Marot, Benjamin Donnot, Gabriel Dulac-Arnold, Adrian Kelly, Aidan O’Sullivan, Jan Viebahn, Mariette Awad, Isabelle Guyon, Patrick Panciatici, and Camilo Romero. Learning 15
to run a power network challenge: a retrospective analysis. In Hugo Jair Escalante and Katja Hofmann, editors, Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 112–132. PMLR, 06–12 Dec 2021. URL https://proceedings.mlr.press/v133/marot21a.html. [20] Mohsen Parsa Moghaddam, Abdolreza Abdollahi, and Masoud Rashidinejad. Flexible demand response programs modeling in competitive electricity markets. Applied Energy, 88 (9):3257–3269, 2011. URL https://www.sciencedirect.com/science/article/abs/pii/ S0306261911000638. [21] James Nicolaisen, Valentin Petrov, and Leigh Tesfatsion. Market power and efficiency in a computational electricity market with discriminatory double-auction pricing. IEEE Transactions on Evolutionary Computation, 5(5):504–523, 2001. URL https://www.researchgate. net/publication/3418664. [22] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL https://jmlr.org/papers/v22/20-1364. html. [23] R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2(3):21–41, 2000. doi: 10.21314/JOR.2000.038. URL https://sites.math. washington.edu/~rtr/papers/rtr179-CVaR1.pdf. [24] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 2nd edition, 2018. URL http://incompleteideas.net/book/ the-book-2nd.html. [25] Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015. URL https: //arxiv.org/abs/1404.3862. [26] Jasper van Tilburg, Luciano C. Siebert, and Jochen L. Cremer. Marl-idr: Multi-agent reinforcement learning for incentive-based residential demand response. In 2023 IEEE Belgrade PowerTech, pages 1–8, 2023. doi: 10.1109/PowerTech55446.2023.10202941. [27] José R. Vázquez-Canteli, Jérôme Kämpf, Gregor Henze, and Zoltan Nagy. CityLearn v1.0: An OpenAI Gym environment for demand response with deep reinforcement learning. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys ’19), pages 356–357. ACM, 2019. doi: 10.1145/3360322.3360998. URL https://dl.acm.org/doi/10.1145/3360322.3360998. [28] José R. Vázquez-Canteli and Zoltán Nagy. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied Energy, 235:1072–1089, 2019. ISSN 0306-2619. doi: https://doi.org/10.1016/j.apenergy.2018.11.002. URL https://www. sciencedirect.com/science/article/pii/S0306261918317082. [29] Rafal Weron. Electricity price forecasting: A review of the state-of-the-art with a look into the future. International Journal of Forecasting, 30(4):1030–1081, 2014. doi: 10. 1016/j.ijforecast.2014.08.008. URL https://www.sciencedirect.com/science/article/ pii/S0169207014001083. 16
[30] Eric Wilson, Andrew Parker, Anthony Fontanini, Elaina Present, Janet Reyna, Rajendra Adhikari, Matthew Dahlhausen, Janghyun Kim, Amy LeBar, Liang Liu, et al. End-use load profiles for the U.S. building stock. Technical Report NREL/TP-5500-80889, National Renewable Energy Laboratory, 2022. URL https://www.nrel.gov/docs/fy22osti/80889.pdf.
17
A
Observation Space (Full Detail)
Table 3 provides the full 32-dimensional observation space for hourly mode, including normalization ranges and implementation notes. Table 3: Full 32-dimensional hourly observation space.
B
Index
Feature
Approx. range
Note
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 .. .
Hour of day Day of week Aggregate demand (kWh) Wholesale price ($/kWh) Price forecast t + 1 Price forecast t + 2 Price forecast t + 3 Price forecast t + 4 Temperature (°C) Demand stress Price stress Thermal stress Overall stress Budget remaining ($) Last credit ($/kWh) Building 1 load (kWh) .. .
[0, 23] [0, 6] [0, 500] [0.02, 9.50] [0.02, 1.0] [0.02, 1.0] [0.02, 1.0] [0.02, 1.0] [−10, 45] [0, 1] [0, 1] [0, 1] [0, 1] [0, 200] [0, 0.10] [0, 50] .. .
Integer Integer Post-reduction Cached per timestep TOU + AR(1)
24 25 .. .
Building 10 load (kWh) Demand history t − 4 (kWh) .. .
[0, 50] [0, 500] .. .
Zero if N < 10 Aggregate
29 30 31
Demand history t (kWh) Cumulative credits ($) Day within episode
[0, 500] [0, 500] [0, 30]
Aggregate Episode total Multi-day episodes
From building sim Sigmoid; threshold 100 kWh Sigmoid; threshold 0.25 $/kWh Ramp above 35°C, below 0°C Weighted avg. of 9–11 Daily budget Previous action Padded to 10 buildings
Baseline Policy Definitions
Figure 9 summarizes the four hand-designed baseline policies. All baselines use only a subset of the 32-dimensional observation.
C
Additional Experiments
C.1
Scalability Ablation
To assess scalability, we re-evaluate all four baseline policies at N = 500 buildings using the synthetic demand model (CityLearn data covers only five buildings; synthetic mode supports arbitrary N ). Demand stress threshold is scaled proportionally (D∗ = 10,000 kWh). Figure 10 shows that baseline rank ordering is preserved and performance differences are within one standard deviation, confirming that the environment’s signal structure is stable at commercial portfolio scale.
18
NoCreditPolicy: ct = 0 for all t. UniformCreditPolicy: ct = 0.05 $/kWh for all t. RuleBasedPolicy: if σtprice > 0.5: ct = 0.10 else: ct = 0 BudgetAwareRulePolicy: βt = Bt /B0 (budget fraction remaining) if σtprice > 0.5 and βt > 0.1: ct = 0.10 · βt else: ct = 0
Figure 9: Baseline policy definitions. Bt = budget remaining at step t; B0 = initial daily budget. Simulation Time Using our simulator to generate data often takes longer the more buildings are chosen to be simulated. Currently, we achieve a rate of 0.3 seconds per episode (24 steps) when simulating N = 50 buildings. In contrast, we achieve a rate of 13 seconds per episode (again, 24 steps) when simulating N = 500 buildings. We note that using RL/ML should not increase these times; only when calibrating the internal models might the simulator take longer.
C.2
Multi-objective Trade-Off (CVaR) Analysis
A key design feature of DR-Gym is its configurable reward signal, which allows the use of riskaware measures. We demonstrate this feature by using CVaR, a popular risk-aware metric [23, 12], to visualize the risk-revenue trade-off. First, we set a penalty weight wCVaR , where 0.0 recovers a purely risk-neutral objective while 1.0 penalizes policies that allow high consumer bill tail risk. To demonstrate this trade-off without requiring separate training runs per weight, we sweep a uniform credit policy over c ∈ {0.00, 0.02, 0.04, 0.06, 0.08, 0.10} $/kWh, tracing the Pareto frontier between electric utility revenue and consumer bill tail risk (Figure 11). Higher credit levels monotonically reduce CVaR0.95 of consumer bills at a modest revenue cost, confirming that the reward structure embeds the intended risk–revenue trade-off. A risk-aware agent (CVaR-PPO, WCSAC, distributional RL) that maximizes a reward with large wCVaR would be expected to operate near the low-CVaR end of this frontier.
D
Simulator Comparison Table
19
Figure 10: Scalability ablation: baseline policy performance at N = 50 (primary) and N = 500 (commercial portfolio scale). Baseline rank ordering is preserved and performance differences are within one standard deviation, confirming that the environment signal structure is stable across portfolio sizes.
Table 4: Comparison of open-source reinforcement learning environments for energy and grid management. Unlike existing simulators that focus on physical device control or network topology, DR-Gym provides a market-level, risk-aware environment designed to mitigate consumer financial tail-risk. We do not include MARL-iDR [26] due to its focused application setting rather than being a testbed. Feature
DR-Gym (Ours)
CityLearn
Sinergym
Grid2Op
Target Agent Role
Market-Level electric utility (DR credit pricing)
Building Controller (HVAC/Battery dispatch)
Building Controller (Thermostat setpoints)
System Operator (Transmission topology)
Risk Objective
Risk-aware and neutral CVaR as example
Risk-neutral (Expected cost/energy)
Risk-neutral (Energy vs. comfort)
Risk-neutral (Margin maximization)
Extreme Events
Markov regimeswitching (Correlated price spikes)
Historical static weather & pricing
Historical & simulated TMY weather data
Static historical time-series scenarios
Human Behavior
Stochastic fatigue (Heterogeneous archetypes)
N/A (Compliant devices)
N/A (Compliant devices)
N/A (Inelastic load profiles)
Demand Coupling
Temperaturecoupled synthetic & Stock
Physics-based thermal dynamics
EnergyPlus co-simulation
PandaPower / AC-DC power flows
Res-
20
Figure 11: CVaR–reward trade-off analysis. Sweeping a uniform credit policy from c = 0 to c = 0.10 $/kWh traces the Pareto frontier: higher credits monotonically reduce consumer bill tail risk (CVaR0.95 ) at a modest electric utility revenue cost, confirming the intended risk–revenue trade-off embedded in the multi-objective reward function.
21