Safe Deep Reinforcement Learning for Building Heating Control and Demand-side Flexibility

Safe Deep Reinforcement Learning for Building Heating Control and Demand-side Flexibility Colin Jüni ∗ Mina Montazeri ∗ Yi Guo ∗∗ Federica Bellizio ∗ Giovanni Sansavini ∗∗∗ Philipp Heer ∗

arXiv:2604.16033v1 [eess.SY] 17 Apr 2026

∗

Urban Energy Systems Laboratory, Swiss Federal Laboratories for Materials Science and Technology, Dubendorf, Switzerland. ∗∗ School of Automation, Beijing Institute of Technology, Haidian District, Beijing, China ∗∗∗ Reliability and Risk Engineering Laboratory, ETH Zürich, CH-8092, Switzerland.

Abstract: Buildings account for approximately 40 % of global energy consumption, and with the growing share of intermittent renewable energy sources, enabling demand-side flexibility, particularly in heating, ventilation and air conditioning systems, is essential for grid stability and energy efficiency. This paper presents a safe deep reinforcement learning-based control framework to optimize building space heating while enabling demand-side flexibility provision for power system operators. A deep deterministic policy gradient algorithm is used as the core deep reinforcement learning method, enabling the controller to learn an optimal heating strategy through interaction with the building thermal model while maintaining occupant comfort, minimizing energy cost, and providing flexibility. To address safety concerns with reinforcement learning, particularly regarding compliance with flexibility requests, we propose a real-time adaptive safety-filter to ensure that the system operates within predefined constraints during demand-side flexibility provision. The proposed real-time adaptive safety filter guarantees full compliance with flexibility requests from system operators and improves energy and cost efficiency — achieving up to 50 % savings compared to a rule-based controller — while outperforming a standalone deep reinforcement learning-based controller in energy and cost metrics, with only a slight increase in comfort temperature violations. Keywords: Safe Reinforcement Learning, Demand-side Flexibility, Real-time Adaptive Safety Filter, Building Thermal Control 1. INTRODUCTION In 2022, the building sector accounted for 34% of global energy demand and 37% of the energy- and process-related carbon dioxide emissions (United Nations Environment Programme, 2024; Nejat et al., 2015), with consumption projected to rise by 50% in the next 30 years according to the International Energy Agency (Agency, 2013). Meanwhile, the energy landscape is evolving, with a growing share of intermittent Renewable Energy Sources (RESs) and Distributed Energy Resources (DERs), alongside the gradual phase-out of conventional power plants. These developments introduce operational uncertainty and pose critical challenges to power system operation (Ostergaard et al., 2021). Simultaneously, advances in digitalization open new opportunities to leverage demand-side flexibility to support electrical system operation (Golmohamadi et al., 2024), reducing reliance on conventional power generation, thus mitigating the need for costly grid reinforcements (Fidalgo and Azevedo, 2022). In the building sector, Demand-Side Management (DSM) programs are becoming increasingly popular due to the integration of energy management systems, which aim to optimize net

energy consumption and enhance integration into a more flexible and low-carbon energy system (Guelpa and Verda, 2021). To effectively control building energy systems, different control strategies are required to manage Heating, Ventilation, and Air Conditioning (HVAC) systems in a dynamic environment. Conventionally, Rule-based Control (RBC) relies on predefined rules and static thresholds to manage building energy systems. While widely adopted but simple, these controllers struggle to adapt to fluctuating energy prices, occupancy patterns, or renewable energy availability (Belic et al., 2015). To improve efficiency, Model Predictive Control (MPC) has been introduced, leveraging forecasted data to optimize control decisions. However, its reliance on accurate system models and high computational complexity limits its real-time adaptability and scalability (Drgoňa et al., 2020). In contrast, Reinforcement Learning (RL) provides a data-driven alternative, autonomously learning optimal control policies by interacting with an environment. Deep Reinforcement Learning (DRL) further enhances this capability by using deep neural networks to handle complex, high-dimensional state and action spaces,

making it particularly suited for dynamic building energy management (Fu et al., 2022). Numerous studies have leveraged DRL for building energy management to improve thermal comfort and reduce energy use (Gupta et al., 2021; Silvestri et al., 2024; Wei et al., 2017; Kazmi et al., 2018). Gupta et al. (Gupta et al., 2021) implemented a DRL-based heating controller that penalizes both comfort violations and energy costs, demonstrating up to 30% improvement in comfort and 12% reduction in energy costs compared to traditional thermostats. Similar results were reported by Silvestri et al. (Silvestri et al., 2024), Wei et al. (Wei et al., 2017), and Kazmi et al. (Kazmi et al., 2018), who demonstrated the effectiveness of DRL in reducing energy consumption and temperature violations in HVAC control scenarios, including water temperature control in residential buildings. While DRL has shown strong potential for managing complex, dynamic building environments (Yu et al., 2021), its exploratory nature raises concerns about safety and constraint satisfaction, as it may lead to policy violations that impact occupant comfort or energy flexibility provision. This is particularly critical in DSM programs, where buildings must respond to flexibility requests from a power system operator by imposed energy constraints (Wang and Hong, 2020; Zhao et al., 2023). In parallel, numerous control strategies beyond reinforcement learning have been explored to enable energy flexibility provision from buildings, such as load shifting techniques (Yang et al., 2024). Zhengguang et al. (Liu et al., 2023) reviewed building energy flexibility and showed that the thermal inertia of a building plays a major role when considering different DSM scenarios. Zhang et al. (Zhang et al., 2022) investigated an MPC-based approach for flexibility provision, demonstrating the ability of MPC to provide a multitude of flexibility services, including load shedding, shifting and load tracking. Friansa et al. (Friansa et al., 2024) investigated a DRL-based controller to enhance energy flexibility in a building equipped with photovoltaic generation and a flexible HVAC system, improving flexibility while maintaining occupant comfort However, many of these approaches either do not explicitly enforce energy constraints or depend on accurate system models, which may limit scalability across diverse building types. To address the lack of safety guarantees, Safe Reinforcement Learning (SRL) techniques have been developed and integrated with safety filters and constraint-handling mechanisms to ensure control policies remain within operational limits (Gu et al., 2024). These approaches have been successfully applied in domains such as robotics, autonomous drones, and video games, where safety violations can result in problematic failures (Qi et al., 2024; Inamdar et al., 2024). By incorporating explicit energy constraints, SRL enables buildings to dynamically adjust their heating strategies to fulfill flexibility requests without violating contractual obligations with power system operators. These safety-aware DRL frameworks allow buildings to act as active participants in balancing and congestion mechanisms, supporting a more resilient and decentralized energy system. In contrast to model-based control methods - which require accurate system models (Qi et al., 2024; Zhao et al., 2021) - model-free SRL offers a promising alternative, enabling the controller to learn and adapt strategies while explicitly ensuring compliance energy con-

straints. Despite its strong potential, the application of SRL in building energy management remains largely unexplored, particularly in the context of real-time flexibility provision. Huo et al. (Huo et al., 2024) proposed a physics-inspired safe DRL method for temperature control, using prior knowledge and hard steady-state rules to guarantee compliance with grid-interaction constraints. Paesschesoone et al. (Paesschesoone et al., 2024) proposed a safe DRL controller with a predictive safety filter based on MPC to enhance the efficiency and safety of existing energy flexibility controllers. Using a data-driven MPC safety filter to adjust the DRL agent’s action, adherence of energy constraints can be guaranteed. Similarly, Wang et al. (Wang et al., 2025) applied an MPC-based filter layer to correct potentially unsafe states using an approximate building model. While these safe DRL approaches show promising results, they rely on system identification and physical modeling, or handcrafted rules — all of which limit their scalability and applicability across diverse building types. In this paper, we design and implement a safe DRLbased controller for flexibility provision within an energy hub. To ensure safe operation without relying on physical models or prior system knowledge, we introduce a realtime adaptive safety filter that adjusts the DRL agent’s actions based on real-time observations of room temperature and dynamic electricity price. For simulation, we model the thermal dynamics of the building using Physically Consistent Neural Networks (PCNNs), which integrates physical system knowledge with data-driven learning. The key contributions of this work are threefold: (1) A model-free real-time adaptive safety filter that requires no prior system knowledge or model identification. The filter adjusts control actions in realtime based on room temperature and electricity price to ensure compliance with flexibility requests from power system operators. (2) The integration of the safety filter into a DRLbased control framework for demand-side flexibility in building space heating, enabling safe operation in real-time without compromising control performance. (3) A close-to-reality simulation using data from a living laboratory building 1 , demonstrating the controller’s ability to optimize energy usage, ensure thermal comfort, and deliver flexibility services. 2. PROBLEM FORMULATION In this section, the control strategy for the flexible operation of a heat pump for space heating is presented. The controller minimizes the electricity costs while ensuring the constraints regarding system dynamics, thermal comfort, and flexibility provision. 2.1 Optimization-Based Control The control problem is formulated to determine optimal heat pump input ut ∈ R over the time horizon t ∈ T , subject to system dynamics and operational constraints: 1

UMAR apartment unit at the NEST building on the Empa campus in Dübendorf, Switzerland.

min J (ut , ζ(t))

,

(1a)

Ttmin ≤ Troom,t ≤ Ttmax , Troom,t+1 = h(Troom,t , ut ), X f (ut ) ≤ V (F ),

(1b) (1c)

ut

subject to

(1d)

t∈H

where J is the objective function, i.e. the electricity costs to be minimized, and ζ(t) is the real-time electricity price at time t. We use Troom,t to denote the indoor room temperature at timestep t, and Ttmin and Ttmax are the minimum and maximum temperature limits for thermal comfort. The system dynamics h(Troom,t , ut ) capture the thermal behavior of the room, linking the current temperature and control input to the predicted room temperature at the next time step. This function depends on the building thermal model of the room. The term f (ut ) represents the energy consumption associated with the control input, which is accumulated over the duration of a flexibility provision time window t ∈ H ⊆ T . The constraint in (1d) ensures that the total energy consumed within the flexibility time window does not exceed the allowable energy budget V (F ), as defined by the flexibility provision message F . This constraint introduces a global coupling across time steps, making the control problem temporally dependent for allocation of energy throughout the flexibility time window. More details and an explanation of flexibility constraints will be presented in Section 3. 2.2 Control Architecture The controller optimizes heat pump operation to ensure energy efficiency under varying ambient and operational conditions. To this end, we propose a model-free DRL control architecture combined with a safety filter to ensure compliance with energy flexibility constraints. The overall design is shown in Fig. 1.

which uses a hybrid thermal model of a room to simulate the temperature at the next time step t + 1 based on the previous temperature and ambient data at timestep t. Detailed mathematical description of the framework is provided in Section 3. Although we consider the flexibility provision in the state and in the reward function of the DRL controller, there is no guarantee that the energy flexibility requests are fulfilled at all times. Therefore, an additional safety measure is needed to fulfill the flexibility constraints at all times and thus ensure efficient operation of the grid. The Real-time Adaptive Safety Filter (RASF), introduced in Section 4, is the critical component that guarantees all actions applied to the environment comply with the predefined flexibility constraints outlined in the flexibility provision message by adjusting the proposed DRL action ut to a safe control input usafe,t . 2.3 Thermal Modeling of Room Temperature To simulate the thermal dynamics of a room, we use a hybrid modeling approach based on PCNNs (Di Natale et al., 2022). PCNNs combine a linear model incorporating physical knowledge and a non-linear data-driven neural network model to capture unforced dynamics that physical equations cannot quickly characterize. This structure enables accurate, interpretable temperature prediction while maintaining flexibility accross different buildings (Karniadakis and Kevrekidis, 2021). The room temperature at the next timestep is modeled as Troom,t+1 = Dt+1 +Pt+1 , where Pt+1 captures the physical principles and Dt+1 accounts for the unforced dynamics learned from historical data via a Neural Network (NN). For more details of this hybrid model using a PCNN, please refer to Di Natale et al. (Di Natale et al., 2022). 2.4 Flexibility Provision Flexible building operation refers to a building’s ability to adjust its energy consumption patterns in response to signals or requests from the system operators (Ostergaard et al., 2021). In our study, the controller incorporates flexibility provision messages by the operators while maintaining thermal comfort and minimizing electricity costs. We consider flexibility provision messages in the form of F = (ts , te , ϕflexibility ), where ts and te represent the starting and ending times of a flexibility provision time window H, respectively, and ϕflexibility denotes the flexibility factor. The flexibility factor ϕflexibility ∈ [ϕlow , ϕhigh ] specifies the energy consumption during the flexibility time window compared to the Business-as-usual (BAU) scenario without external intervention, as visualized in Fig. 2.

Fig. 1. Overview of the proposed safe DRL-based control scheme for space heating and demand-side flexibility. The DRL controller is the primary decision-making instance, computing optimal control action ut based on the current state of the system x. By interacting with the environment, the DRL controller learns an optimal policy, maximizing the reward function to take an action ut based on the state x and reward r at the previous time step. The agent interacts with the building model,

The deep reinforcement learning-based controller incorporates the flexibility provision messages F as part of the state space. This approach allows the DRL agent to learn policies that naturally comply with flexibility constraints during training, such as preheating the room before a flexibility provision time window starts. However, this does not guarantee that flexibility constraints are fully satisfied in all scenarios. Thus, an additional safety filter is required to ensure strict compliance with flexibility constraints.

3. DEEP REINFORCEMENT LEARNING CONTROLLER As mentioned above, the DRL algorithm aims to determine the optimal control set points for the heat pump, specifically the valve opening level. Since we consider the valve opening as the DRL’s action (u = [ut ]) and it is a continuous variable, we apply DRL using a Deep Deterministic Policy Gradient (DDPG) algorithm which is wellsuited for continuous action spaces. The DDPG framework consists of two neural networks: a critic network, which estimates the value function (Q-value) for state-action pairs, and an actor network, which learns a deterministic policy to select actions given a state. We define the state of the DRL agent at time t as follows: St = [Isolar,t , Tamb,t , Troom,t , Tneigh,t , Xtime,t ut , cheat,t , ttostart,t , ttoend,t , EBAU,t , Et ],

(2)

where Isolar,t represents the solar irradiation, Tamb,t is the ambient temperature, Troom,t is the controlled room temperature, and Tneigh,t is the temperature of the neighboring room, influencing heat transfer. The variable Xtime,t specifies the time of day, weekday, and season, while cheat,t denotes the heating or cooling scenario. The terms ttostart,t and ttoend,t define the flexibility provision window. The BAU energy EBAU,t is the amount of energy used without flexibility provision. The simulator computes the BAU energy consumption by running the DRL agent without considering flexibility constraints on historical data. The energy that has already been used during the flexibility time window is described by Et , so that the agent knows how much more energy it can use minimally or maximally. The reward function helps the agent to maintain a balance between energy consumption and thermal comfort while fulfilling flexibility provisions. Thus, the reward function is given by: Rt = β · Rtemp,t − δ · Rprice,t − Rflex,t ,

(3)

where Rtemp,t penalizes deviations from predefined comfort temperature bounds, Rprice,t accounts for electricity costs, and Rflex,t penalizes non-compliance with flexibility constraints. The coefficients β and δ assign different weights to comfort violation and electricity cost. Each reward component is defined as:

Rtemp,t = max(Troom,t − Ttmax , 0) − min(Troom,t − Ttmin , 0), Rprice,t = ut · ζ(t), Rflex,t = α. The penalty term α for flexibility constraints is defined as:    α1 , if Et > EBAU,t ,  α = α2 , if Et > EBAU,t and ttoend,t = 0, (4)    0, otherwise. To train the neural networks in the DDPG algorithm, we apply an actor-critic learning framework. The actor network, denoted as µ(St |θµ ), represents the policy and determines the optimal valve opening level ut based on the given state St . The parameter set θµ consists of the weights and biases of the neural network that parametrizes the policy function µ. The critic network, denoted as Q(St , ut |θQ ) assesses the long-term impact of selecting a specific valve opening level by estimating the expected future reward for a given state-action pair. The parameter set θQ contains the weights and biases of the critic network that approximates the Q-function. The Q-function represents the expected cumulative reward that the agent can achieve starting from a given state and taking a specific action, while following the current policy. The actor network is trained to maximize the expected cumulative reward, ensuring that the valve operates efficiently while maintaining thermal comfort and meeting flexibility constraints. This is formulated as the following objective function: "∞ # X µ t J(θ ) = E γ Rt | S0 , µ , (5) t=0

where γ is the discount factor that determines the importance of future rewards. To optimize this objective, the actor network is updated using the deterministic policy gradient: h i ∇θµ J ≈ E ∇u Q(S, u | θQ ) u = µ(S)∇θµ µ(S | θµ ) . (6) The critic network is trained using the Bellman equation, which ensures the Q-value estimates align with the expected future rewards. The target Q-value is computed as: yt = Rt + γQ(St+1 , µ(St+1 | θµ ) | θQ ). (7) The critic network minimizes the loss function:   (8) L(θQ ) = E (yt − Q(St , ut | θQ ))2 . By continuously updating both networks, the DRL controller learns to adjust the valve opening dynamically in response to ambient conditions, electricity prices, and flexibility constraints. This enables efficient heat pump operation while maintaining comfort and optimizing energy consumption. 4. REAL-TIME ADAPTIVE SAFETY FILTER DESIGN

Fig. 2. Timeline representation of a flexibility provision message and the associated flexibility time window with starting and ending times ts and te . The flexibility provision message is received at t1 , with the states ttostart and ttoend at time t2 . The orange line shows the BAU energy consumption trajectory, while the green line represents the flexibility provision trajectory.

The RASF ensures that the actions proposed by the DRL agent always fulfill the flexibility constraints. At each time, the agent proposes an action ut , which represents the opening factor of the valve of the heat pump. The proposed safety filter evaluates this action and adjusts it in a realtime manner, based on real-time conditions, to produce a

safe control input usafe,t , thereby guaranteeing compliance with the predefined flexibility constraints. The framework with the embedded RASF is displayed in Fig. 1. The proposed safety filter operates as follows: The remaining average safe action u1,t is computed based on the remaining energy budget Eremaining,t = EBAU,t − Et to fulfill the flexibility request and the remaining time ttoend,t of the flexibility window, so that Eremaining,t u1,t = . (9) ttoend,t Thus, the control input usafe,t precisely meets the flexibility request, as it is derived as the average input needed per timestep over the remaining duration of the flexibility window. To adapt to varying operational conditions, the safety filter adjusts the tolerance τt in real-time, increasing it when greater flexibility is both necessary and beneficial. In particular, higher tolerance is required when the room temperature approaches its comfort limits, as the system needs more freedom to act and avoid violating thermal comfort constraints. At the same time, low electricity prices offer an opportunity to consume energy more freely without incurring high costs. By increasing τt in these situations, the filter allows the controller to respond more aggressively when either comfort risk or economic benefit is high. This behavior is modeled through a weighted combination of two normalized indicators: thermal flexibility and price favorability. The tolerance is defined as: τt = τbase,t · (w1 · µtemp,t + (1 − w1 ) · µprice,t )1.5 , (10) where µtemp,t = (1 − Troom,t ), µprice,t = (1 − ζ(t)). The base tolerance τbase,t is defined based on the position within the flexibility time window, decreasing as the flexibility period progresses. In the early stages, i.e., time steps near the start of the flexibility window ts as illustrated in Fig. 2, a higher tolerance is applied to allow more deviation and provide greater control flexibility. Toward the end of the window, the tolerance is gradually tightened to ensure full adherence to the cumulative flexibility constraint. The coefficient µtemp,t is a temperature-based weight and µprice,t is a cost-based weight. The indoor room temperature Troom,t and the electricity price ζ(t) are normalized between [0.1, 0.9] using min-max normalization and w1 is a weighting factor that balances the influence of the electricity price and the temperature, respectively. This dynamic adaptation prevents violations while allowing the DRL controller to explore efficient energy management strategies. With the introduced tolerance τt , the maximal and minimal actions of the valve of the heat pump are defined by: umaximal,t = min (u1,t · (1 + τt ), 1) , uminimal,t = max (u1,t · (1 − τt ), 0) . (11) The maximal and minimal control inputs defined by the RASF account for scenarios where energy consumption needs to decrease or increase, respectively. Finally, depending on the action proposed by the DRL agent ut , the safety filter adjusts the action to usafe,t , which is then fed to the building environment to ensure the flexibility request is fulfilled and the predefined energy flexibility is provided. For the case where energy consumption needs to decrease during the flexibility time

(a)

(b)

Fig. 3. (a) The NEST building with the UMAR unit marked in white (© Zooey Braun, Stuttgart) and (b) Outline of the UMAR apartment unit. The apartment has two bedrooms with a floor area of 17.6 m2 and a large window facing east-southeast. It has a combined heating and cooling system through ceiling panels. window, ϕflexibility ≤ 1, the action is defined by usafe,t = min (ut , umaximal,t ), Conversely, when energy consumption needs to increase, ϕflexibility ≥ 1, the action is given by usafe,t = max (ut , uminimal,t ). 5. CASE STUDY The historical data from the Urban Mining and Recycling (UMAR) unit, a residential space of the Empa NEST Demonstrator in Dubendorf, Switzerland, shown in Fig. 3 are used (UMA; Nes). We consider the heating of one bedroom, i.e., room 272, to test the proposed control strategy. In this room, we can control the valve to regulate the water flow into the ceiling panels.

5.1 Simulation Setup The proposed controller is applied to the building model of the UMAR unit, which is trained on historical heating data between December 2021 and March 2022 with 15-minute resolution. The data used are normalized between [0.1, 0.9] using min-max normalization to avoid biased results based on the different scales of the variables. To evaluate the performance of the proposed SRL controller, we define specific conditions for comfort room temperature, electricity pricing, and flexibility provision. The comfort range for room temperature is defined as [T min , T max ] = [23.5, 25]◦ C 2 and a dynamic price tariff is used as electricity price. The specifications of the flexibility request in the form of F = (ts , te , ϕflexibility ) are communicated every day at 8 am. The start and end times of flexibility windows are randomly selected for the training of the DRL agent, with a maximum duration of ten hours, to simulate various flexibility requests. The flexibility coefficient ϕflexibility is also randomly selected within the given boundaries [ϕlow , ϕhigh ] = [0.7, 1.3], corresponding to lower or higher energy consumption compared to the BAU scenario. The weighting factors in the reward function, balancing comfort violations and electricity price, are set to β = 20 and δ = 0.8, while the penalty for not fulfilling flexibility constraints is set to α1 = 1 and α2 = 10. The 2

Due to the big windows of UMAR units of NEST facing towards solar irradiation most of the day, the room does not need much heating power, even on colder winter days. Therefore, we chose a higher-than-normal range for the comfort temperature in order to stimulate the controller to take action.

weighting factor for tolerance calculation in the RASF is chosen as w1 = 0.5 for this case study. To evaluate the performance of the proposed DRL-based controller with the safety filter, we compare it against a baseline Rule-based (RB) controller, analyzing the temperature evolution, the electricity cost, and the flexibility provision. The RB controller follows a conventional hysteresisbased strategy. The valve of the heat pump is opened when the room temperature Troom,t falls below the lower comfort boundary Ttmin , and closed when Troom,t exceeds the upper comfort boundary Ttmax . As a result, the baseline controller does not optimize heating based on electricity price variations and does not incorporate demand-side flexibility provision. It only reacts to the temperature threshold, leading to heat pump ON/OFF cycles. The BAU scenario, used for flexibility time windows, is computed by projecting future actions based on historical data without assuming flexibility provision or other external disturbances. 5.2 Numerical Results This subsection presents the performance of the proposed DRL-based control approach with a real-time adaptive safety-filter. We compare four control strategies: (1) A baseline RB controller based on fixed, manually defined rules for heating control, without consideration of energy cost or flexibility provision. (2) A DRL controller that does not consider flexibility provision. The controller aims to maintain occupant thermal comfort while minimizing electricity costs. (3) A DRL controller that considers flexibility provision but does not include the safety filter. The system operator communicates flexibility provision messages as introduced in Section 2.4 and Section 5.1 and the DRL-based controller applies its policy to comply with the flexibility constraints. (4) A DRL controller with the real-time adaptive safety filter to ensure compliance with the flexibility constraints. Fig. 4 presents the room temperature evolution, control actions, and ambient conditions for the four different control strategies, highlighting their responses to flexibility requests and varying electricity prices over a three-day period. To complement this, Fig. 6 summarizes the key performance indicators across the same simulation period, providing a quantitative comparison of energy consumption, cost, and comfort violations. Fig. 5 illustrates the average cumulative reward of the DRL-based controller for Case III during its training over a series of episodes. One episode corresponds to three days. It can be seen that the DRL controller outperforms the RB controller in terms of reward after 400 episodes. In Case III, the DRL controller is confronted with flexibility constraints. The controller therefore must adjust its operation based on external flexibility disturbances. The flexibility time windows are placed when the agent generally takes action to heat the room (Case II). We observe that the DRL-based controller responds to the flexibility provision request, trying to reduce the energy consumption by 30% compared to historical data for this

time window with ϕflexibility = 0.7, as shown in Fig. 4. As a result, the agent tries to preheat the room before flexibility time windows during low electricity prices in order to comply with flexibility constraints, as seen at 4 Compared to the baseline RB controller, cost savings ○. are increased heavily, and comfort temperature violations are decreased. However, despite prioritizing the flexibility provision over comfort temperature in the reward function, the controller fails to fulfill all flexibility requests. In Case IV, we introduce the DRL-based controller combined with the RASF introduced in Section 4, ensuring strict compliance with the flexibility constraints during flexibility time windows. In Fig. 4, we see that the safety filter ensures fulfillment of the flexibility constraints, for 2 & ○, 3 example at ○ where the agent without safety filter would not comply with them. Compared to Case III, the comfort temperature violation increases minimally, 2 and energy consumption and cost of electricity are see ○, comparable, as shown in Fig. 6. The controller with the safety filter still outperforms the baseline RB controller by reducing cost by 64.5% while improving thermal comfort by 45.5%. In Case II, the agent maintains temperatures near the lower comfort bound, as shown in Fig 4, reducing fluctuations compared to the RB controller. It preheats during 1 low-price periods (see ○) to reduce later heating needs, effectively lowering demand during peak prices. As shown in Fig. 6, this results in 54.9% cost savings and minimal comfort violations. Compared to Cases III and IV, Case II shows higher energy use and cost due to the absence of flexibility constraints. As seen in Fig. 6, the results show that DRL-based controllers significantly reduce energy consumption and cost compared to the baseline RB controller. However, enforcing flexibility provision increases comfort temperature violations due to the newly imposed constraints. With the integration of the safety-filter, the performance drops minimally, but ensures compliance with flexibility constraints. Despite its trade-off, Case IV maintains a balance between energy and grid efficiency, ensuring adherence to flexibility requests while keeping costs lower than the baseline. These findings highlight the importance of integrating safety mechanisms in DRL-based controllers to balance energy efficiency, comfort, and flexibility provision. Table 1 summarizes the key performance indicators over a onemonth simulation for the four cases, confirming the same trends. 5.3 Discussion The results demonstrate that the DRL-based controller outperforms the RB baseline in thermal comfort, energy efficiency, and cost savings — even when subject to flexibility constraints. Its predictive, cost-aware actions help minimize temperature fluctuations and reduce energy consumption and cost by up to 50% over a one month simulation. While the standalone DRL controller performs well, its ability to comply with flexibility constraints is not guaranteed. The integration of the proposed safety filter ensures full compliance with flexibility requests, while maintaining performance close to the unconstrained controller. This demonstrates the filter’s effectiveness in en-

Table 1. Comparison of Key Performance Indicators for Simulation over one Month Metric

Baseline RB

DRL (No Flex. Provision)

DRL (Flex. Provision)

Comfort Violations (Kh)

97.12

75.62

100.56

103.56

Energy Consumption (kWh)

134.62

67.95

63.68

69.02

Cost (CHF)

40.75

21.41

18.07

19.42

forcing safety without significantly compromising comfort or efficiency. Future work should investigate the safety filter’s robustness under varying flexibility patterns, comfort-cost tradeoffs, and real-world pricing signals. Additionally, deploying the filter in a distributed multi-agent setting, coordinating across multiple buildings with differing thermal dynamics and occupant preferences, would further enhance its value in grid-responsive applications. Notably, as seen in the simulation over one month, flexibility provision during periods of extreme ambient conditions can lead to increased comfort violations. This suggests that flexibility provision may not be appropriate under such conditions, emphasizing the need for improved coordination and communication strategies between power system operators and buildings.

DRL (Flex. Provision) + RASF

electricity costs. Combined with the real-time adaptive safety filter, the control architecture successfully fulfills the flexibility constraints, providing the predefined energy flexibility and enhancing the operation of the entire electrical system. Due to its model-free nature, the approach can be generalized to different buildings, but adaptation requires availability of thermal models and historical data. DECLARATION OF GENERATIVE AI AND AI-ASSISTED TECHNOLOGIES IN THE WRITING PROCESS During the preparation of this work, the authors used ChatGPT to check grammar. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the final version of the publication.

6. CONCLUSION In this work, we propose a data-driven safe DRL-based method to obtain optimal control policy for the building space heating, considering demand-side flexibility provision. We implement a real-time adaptive safety filter to ensure the fulfillment of flexibility requests from electrical system operators, enabling safe and efficient operation. The DRL-based controller, together with the real-time adaptive safety filter, optimizes the control problem to maximize the thermal comfort of the occupants and minimize electricity costs for the heat pump operation. The proposed DRL-based controller outperformed traditional RB control, achieving better thermal comfort and lower

REFERENCES (????). Nest - gemeinsam an der zukunft bauen. https:// www.empa.ch/web/nest/. (????). Urban mining and recycling unit. http:// nest-umar.net/. Agency, I.E. (2013). Transition to sustainable buildings: Strategies and opportunities to 2050. Organisation for Economic Co-Operation and Development. Belic, F., Hocenski, Z., and Sliskovic, D. (2015). HVAC control methods - A review. In 19th International Conference on System Theory, Control and Computing, 679–686.

Fig. 4. Room temperature evolution, control action and ambient variables under DRL-based control with safety filter (green), DRL-based control without safety filter (blue), and baseline control (orange) with flexibility time windows (red shaded area) with F = [18 : 00, 04 : 00, 0.7]. The grey dashed lines represent the temperature comfort range.

Fig. 5. Mean reward of DRL-based controller during the training of the agent in Case III compared to the reward of the baseline RB controller. For the DRL agent by considering flexibility provisions, after about 2’800 episodes, the mean reward curve starts to converge at a reward of approximately -80, where a higher reward value shows better performance in terms of the integrated cost and comfort measure defined in (3).

Fig. 6. Comparison of Key Performance Indicators over three days. Di Natale, L., Svetozarevic, B., Heer, P., and Jones, C. (2022). Physically consistent neural networks for building thermal modeling: Theory and analysis. Applied Energy, 325, 119806. Drgoňa, J., Arroyo, J., Cupeiro Figueroa, I., Blum, D., Arendt, K., Kim, D., Ollé, E.P., Oravec, J., Wetter, M., Vrabie, D.L., and Helsen, L. (2020). All you need to know about model predictive control for buildings. Annual Reviews in Control, 50, 190–232. Fidalgo, J.N. and Azevedo, F. (2022). Decision support system for long-term reinforcement planning of distribution networks. Electric Power Systems Research, 209, 107999. Friansa, K., Pradipta, J., Mahesa Nanda, R., Nashirul Haq, I., Armanto Mangkuto, R., Fauzi Iskandar, R., Wasesa, M., and Leksono, E. (2024). Enhancing university building energy flexibility performance using reinforcement learning control. IEEE Access, 12, 192377–192395. Fu, Q., Han, Z., Chen, J., Lu, Y., Wu, H., and Wang, Y. (2022). Applications of reinforcement learning for building energy efficiency control: A review. Journal of Building Engineering, 50, 104165. Golmohamadi, H., Golestan, S., Sinha, R., and BakJensen, B. (2024). Demand-side flexibility in power systems, structure, opportunities, and objectives: A review for residential sector. Energies, 17(18). Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., and Knoll, A. (2024). A review of safe reinforcement

learning: Methods, theory and applications. Guelpa, E. and Verda, V. (2021). Demand response and other demand-side management techniques for district heating: A review. Energy, 219, 119440. Gupta, A., Badr, Y., Negahban, A., and Qiu, R.G. (2021). Energy-efficient heating control for smart buildings with deep reinforcement learning. Journal of Building Engineering, 34, 101739. Huo, X., Liu, B., Dong, J., Lian, J., and Liu, M. (2024). Optimal management of grid-interactive efficient buildings via safe reinforcement learning. arXiv preprint arXiv:2409.08132. Inamdar, R., Sundarr, S.K., Khandelwal, D., Sahu, V.D., and Katal, N. (2024). A comprehensive review on safe reinforcement learning for autonomous vehicle control in dynamic environments. e-Prime - Advances in Electrical Engineering, Electronics and Energy, 10. Karniadakis, G. and Kevrekidis, I.G.and Lu, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3, 422–440. Kazmi, H., Mehmood, F., Lodeweyckx, S., and Driesen, J. (2018). Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems. Energy, 144, 159–168. Liu, Z., Chen, Y., Yang, X., and Yan, J. (2023). Power to heat: Opportunity of flexibility services provided by building energy systems. Advances in Applied Energy, 11, 100149. Nejat, P., Jomehzadeh, F., Taheri, M.M., Gohari, M., and Abd. Majid, M.Z. (2015). A global review of energy consumption, co2 emissions and policy in the residential sector. Renewable and Sustainable Energy Reviews, 43, 843–862. Ostergaard, J., Ziras, C., Bindner, H.W., Kazempour, J., Marinelli, M., Markussen, P., Rosted, S.H., and Christensen, J.S. (2021). Energy security through demandside flexibility: The case of denmark. IEEE Power and Energy Magazine, 19(2), 46–55. Paesschesoone, S., Kayedpour, N., Manna, C., and Crevecoeur, G. (2024). Reinforcement learning for an enhanced energy flexibility controller incorporating predictive safety filter and adaptive policy updates. Applied Energy, 368, 123507. Qi, Q., Yang, X., and Xia, G. (2024). Safe reinforcement learning filter for multicopter collision-free tracking under disturbances. arXiv preprint arXiv:2410.06852. Silvestri, A., Coraci, D., Brandi, S., Capozzoli, A., Borkowski, E., Köhler, J., Wu, D., Zeilinger, M.N., and Schlueter, A. (2024). Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control. Applied Energy, 368, 123447. United Nations Environment Programme (2024). Global status report buildings and constructions 2022. Wang, X., Wang, P., Huang, R., Zhu, X., Arroyo, J., and Li, N. (2025). Safe deep reinforcement learning for building energy management. Applied Energy, 377, 124328. Wang, Z. and Hong, T. (2020). Reinforcement learning for building controls: The opportunities and challenges. Applied Energy, 269, 115036. Wei, T., Wang, Y., and Zhu, Q. (2017). Deep reinforcement learning for building HVAC control. In 54th

ACM/EDAC/IEEE Design Automation Conference, 1– 6. Yang, S., Gao, H.O., and You, F. (2024). Demand flexibility and cost-saving potentials via smart building energy management: Opportunities in residential space heating across the US. Advances in Applied Energy, 14. Yu, L., Qin, S., Zhang, M., Shen, C., Jiang, T., and Guan, X. (2021). A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal, 8, 12046–12063. Zhang, K., Prakash, A., Paul, L., Blum, D., Alstone, P., Zoellick, J., Brown, R., and Pritoni, M. (2022). Model predictive control for demand flexibility: Real-world operation of a commercial building with photovoltaic and battery systems. Advances in Applied Energy, 7, 100099. Zhao, W., He, T., Chen, R., Wei, T., and Liu, C. (2023). State-wise safe reinforcement learning: A survey. arXiv preprint arXiv:2302.03122. Zhao, W., He, T., and Liu, C. (2021). Model-free safe control for zero-violation reinforcement learning. In 5th Annual Conference on Robot Learning.