Multi-Agent Deep Reinforcement Learning for Multi Objective Battery Management in Dairy Farms Marcos Eduardo Cruz Victorio[0000−0003−2604−176X] and Karl Mason[0000−0002−8966−9100]
arXiv:2607.06489v1 [cs.AI] 7 Jul 2026
University of Galway, Ireland [email protected]
Abstract. The dairy industry in Ireland has a large potential for the integration of renewable energy and the reduction of carbon emissions. However, researchers of distributed generation control are mainly focused on residential and commercial applications. To contribute to the effective integration of renewable energy in the dairy sector, this paper presents a multi-objective optimisation control system based on differential evolution and multi agent Deep Reinforcement Learning. The proposed control is organised in two layers: the upper layer uses dynamic pricing, and the lower layer is based on multi-agent reinforcement learning for battery management. This paper also simulates the electrical response of the proposed control system in a rural distribution circuit. The simulation results show that the proposed control framework can improve profits from energy arbitrage up to 18% compared to using Rule-based models, increase the use of distributed generation without significantly increasing cost, and comply with the Irish grid code in terms of voltage variation. Keywords: Multi-Objective Optimisation · Deep Reinforcement Learning · Distributed Generation.
1
Introduction
Currently, there is a large focus on research to deliver the energy transition to combat climate change. In the case of electricity, research on the energy transition focuses on the integration of Distributed Energy Resources (DER)s. In this context, there is a large potential in the agricultural sector for the integration of DERs, due to its energy intensive nature and limited use of smart grid technologies. In an electrical grid, there are multiple, often competing, objectives that need to be balanced in the smart grid [1], such as grid compliance, minimisation of cost and minimisation of carbon emissions. Although previous approaches have been proposed for grid multi objective energy optimisation (MOO) [2, 3], their performance is limited by the precision of their models. In this context, Deep Reinforcement Learning (DRL) methods can be used for energy optimisation in conditions with limited information of the system dynamics. The applications
2
M. E. Cruz Victorio et al.
of DRL in smart grids include maximising the use of renewable energy [4] and minimising costs in electric vehicles [5]. Among DRL methods, Proximal Policy Optimisation (PPO) emerges as an ideal method for energy management [6]. MMO applications in smart grid have been presented with some limiations [7, 8]. For example, in [9] MOO is applied to reduce line congestion and costs, however, this approach did not account for multiple generators and storages in the grid. In [10], a MOO framework is proposed for cost minimisation and voltage regulation, however, this approach is limited to the retail electricity market. To further develop smart grid applications in the rural sector, our proposed control framework combines heuristic optimisation with DRL agents to maximise the use of distributed generation and affordability in dairy farms, accounting for uncertain electricity prices and multiple generation and storage sources. The following section describes the proposed energy management framework.
2
Methodology
The proposed control system is divided into two layers. The lower layer performs local optimisation in the distribution system using a multi-agent system, where each agent controls a single battery based on the price of electricity provided by the upper layer. The upper layer adjusts the internal price of electricity to regulate the interaction with the main grid. These control layers form a distributed control [11] system as shown in Figure 1.
𝜋𝑔𝑟𝑖𝑑
b)
Lower layer
2 PV
min 𝑤1 𝐹1 + 𝑤2 𝐹2
𝜋𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙
3
4.9 km 4.4 km
0.6 km
5 6
min ∑𝑃𝜋𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙
0.6 km
7
1.3 km
0.5 km
𝑉𝑏𝑢𝑠 , 𝛿𝑏𝑢𝑠 , 𝑃𝑏𝑢𝑠
PV
9
𝑆𝑂𝐶
PV
Battery
10
1.7 km
PV
11
12 1.5 km
3.0 km
Battery
0.3 km
0.8 km
Battery
13
14
Battery
0.3 km
8 Circuit Simulation
Battery
2.8 km
4
Multi Agent DRL Layer
𝑃𝑙
Environment layer
0.2 km
Multi-Objective Optimisation Layer
𝑃𝑔𝑟𝑖𝑑
Upper layer
a)
1
Main grid
Battery
2.0 km
PV 0.2 km PV
Fig. 1: a) Proposed control framework. b) Simulated distribution circuit model. Numbers in black indicate the Bus, the numbers in blue indicate line distance between buses and the arrows indicate electrical loads.
MORL for Battery Management
2.1
3
Multi-Objective Layer
The upper control layer performs dynamic pricing by solving the next MOO problem using heuristic optimisation, namely differential evolution:
min α, β
w1 F1 (t) + w2 F2 (t), ∀t ∈ T
(1a)
s.t.
α, β ∈ [0, 1],
(1b)
w1 + w2 = 1
(1c)
where w1 and w2 are weights for the objectives F1 and F2 , at each time step t for a duration of T . The first objective is the cost minimisation of the entire system and the second objective is the minimisation of power flow Pgrid with the main grid, expressed as: F1 = c1
X
F2 = c2
Pgrid (t)πgrid
X
(Pgrid (t))2
(2)
(3)
where parameters α and β modify the internal price of electricity using predictive and corrective measures of internal generation, and c1 and c2 normalise the objectives to one for α, β = 0. The power flow Pgrid with the main grid depends on the line losses and power Load, PL , internal Power generation PG and Power from batteries P : Pgrid (t) = PL (t) − PG (t) − P
(4)
The predictive measure, defined as πp in euro/kW , is based on a forecast with length K of the internal load and generation, PLf and PGf , calculated as follows:
πp (t + k) = (1 − α)πgrid (t + k) + (PLf (t + k) − PGf (t + k))α, ∀k ∈ K
(5)
The internal price πinternal is further modified by a corrective measure to account for forecast errors and the reaction of other agents, based on Pgrid expressed in euro/kW , as follows: πinternal (t) = (1 − β)πp (t) +
Pgrid (t) + Pgrid (t − 1) β 2
(6)
This modifies the internal price according to the current net demand. To increase the price and grid stability, the price changes with a moving average model, using the current and previous price.
4
2.2
M. E. Cruz Victorio et al.
Multi-Agent Layer
The lower layer uses a distributed control system, with multiple independent DRL agents, to control each battery in the distribution circuit. In this paper, we use PPO for its ability to handle continuous action and state spaces and its stability during training [12, 6]. The DRL agent learns a policy to interact with its environment, in this case the circuit, observing a state S and taking an action a sampled from A, to maximise the accumulated reward R. The objective function L of PPO during training is described as follows [12]: L(a) = Et [min(rt (a)[R − Rexp ], clip(rt (a), 1 − δ, 1 + δ)[R − Rexp ]]
(7)
where E is the expectation function at time t; rt is the ratio of probabilities of the new policy over the old policy of selecting action a; Rexp is the expected reward observing the state S and δ is a hyperparameter that limits policy changes between updates. In this paper, the states observed by the DRL agent are expressed as follows: S = {PRB , SOC, πinternal }
(8)
where PRB is a rule-based power schedule and SOC is the battery state of charge. The rule-based battery power schedule charges at the lowest forecasted price and discharges at the highest. Based on the state observed at each hour, the agent selects a power reference P for the battery sampled from the action space A ∈ [−Pmax , Pmax ]. This is used to define the power schedule of each battery. During training, the power schedule is used to calculate the agent’s reward. In this case, the reward is the total cost of energy, that is: X min R = P πinternal (t), ∀t ∈ T P s.t. 0 ≤ SOC ≤ SOCmax
(9a) (9b)
where SOCmax is the maximum SOC. SOC is calculated as follows with charge and discharge efficiencies, ηc and ηd : SOC(t) + ηc P P > 0 SOC(t + 1) = (10) SOC(t) + ηd P else
3
Test Case
The Multi-Objective layer implements differential evolution using Scipy with parameters specified in Appendix A, and the DRL agents are trained using PPO from stable baselines 3 (SB3). w1 is varied between 0.1 and 0.9 in increments of 0.1 to test different weights of each objective, running each configuration 10
MORL for Battery Management a)
5
b)
d)
c)
e)
Fig. 2: Simulation results. a) Voltage amplitude response during peak PV generation in volts per unit (p. u.). b) Phase angle response during peak PV generation. c) MOO results for the distribution grid. The orange dot represents the results with no dynamic pricing. d) Training response of the DRL agent. e) Profits from energy arbitrage for a single prosumer in average. times using random seeds. We tested the Irish single market price data from 2022 publicly available from the Single Electricity Market Operator (SEMO), with equal buying and selling prices. The circuit simulation model is shown in Figure 1 using PyPSA and the distributed generation and storage capacity connected to each Bus, that is, a voltage measurement point, is presented in Appendix B. ηc and ηd are set to 0.95 and 1/0.95, equivalent to a round-trip battery efficiency of 90%, found in Tesla Powerwalls. The simulated load represents 50 Irish farms distributed across the circuit model [8], and the simulated renewable generation is based on Ireland. The nominal voltage is 10KV and the lines have an impedance of 0.157 + i0.123 Ω/km. The price forecast is provided to each DRL agent with lengths of 0 (no forecast), 6, 12 and 24 hours with 0, 5, 10
6
M. E. Cruz Victorio et al.
and 15% added error to represent realistic conditions. PV generation and load forecast is provided similarly for dynamic pricing. The DRL agent uses two fully connected layers, testing 5 cases: 1) using only the rule-based model. 2) using the DRL agent with 7-day trajectories for training with 64 neurons in each layer. 3) 14-day trajectories with 62 neurons in each layer. 4) 30-day trajectories with 128 neurons in each layer. 5) 30-day trajectories with 256 neurons in each layer. The training lasts 2 million steps, where each step represents one hour. In total, 640 configurations are tested.
4
Simulation Results
During peak renewable generation during 4th and 5th of May of 2022, the simulation results show that voltage and phase angle variations are in compliance with the limits of the Irish grid of 1 volt per unit (p. u.) ±10% and phase angle of 0 ±0.69 radians , on a base of 230 Volts per phase, established in the EN 50160 standard. The simulated scenario represents the highest variation in the electrical response for the data tested and is shown in Figure 2. In terms of energy arbitrage in each battery, results show highest profits using a 24-hour price forecast in case 5 with 1866 euros. In comparison, using only the rule-based model, the highest profit achieved is 1569 euros. The results using the DRL agents also show that errors of 5% do not significantly affect total costs. Similarly, Figure 2 shows that there is no significant difference between the 12-hour and 24-hour forecast. The total cost of energy for the entire system for one year is 55, 263 euros and the power variability is 34.73 M W 2 . The simulation results show that the Multi-Objective layer can reduce power variability of the entire system with the main grid by 0.85M W 2 , without disrupting the cost minimisation achieved by the Multi-Agent layer, using dynamic pricing with parameters α, β = [0.022, 0.305], as shown in Figure 2.
5
Conclusion
This paper presents a hierarchical and distributed energy management framework for the rural sector combining Deep Reinforcement Learning and differential evolution for Multi-Objective Optimisation. The use of DRL agents achieves up to 18% increase in profits from energy arbitrage compared to using a rule-based power schedule. The use of dynamic pricing can reduce the power variation with the main grid by 0.85M W 2 and maintain a similar cost compared to using only the DRL agents, which promotes the use of distributed generation and voltage stability. In addition, the simulation of the voltage response shows that the proposed control system is in compliance with the Irish grid standard EN 50160. Furthermore, simulation results show that a relatively small difference in the accuracy of the electricity price forecast did not significantly affect profits from energy arbitrage. Acknowledgments. This research was conducted with the financial support of the Science Foundation Ireland under Grant number [21/FFP-A/9040].
MORL for Battery Management
7
References [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
Jingwen Chen et al. “Multi-Objective Optimization Scheduling of Active Distribution Network Considering Large-Scale Electric Vehicles Based on NSGAII-NDAX Algorithm”. In: IEEE Access 11 (2023), pp. 97259–97273. Muhammad Imran Akbar et al. “A Novel Hybrid Optimization-Based Algorithm for the Single and Multi-Objective Achievement With Optimal DG Allocations in Distribution Networks”. In: IEEE Access 10 (2022), pp. 25669–25687. Ruipeng Xu et al. “Multi-Objective Hierarchically-Coordinated Volt/Var Control for Active Distribution Networks With Droop-Controlled PV Inverters”. In: IEEE Transactions on Smart Grid 13.2 (2022), pp. 998–1011. Xiaoyu Li, Xueshan Han, and Ming Yang. “Day-Ahead Optimal Dispatch Strategy for Active Distribution Network Based on Improved Deep Reinforcement Learning”. In: IEEE Access 10 (2022), pp. 9357–9370. Yi Wang et al. “Coordinated Electric Vehicle Active and Reactive Power Control for Active Distribution Networks”. In: IEEE Transactions on Industrial Informatics 19.2 (2023), pp. 1611–1622. Nawazish Ali, Rachael Shaw, and Karl Mason. “A Deep Reinforcement Learning Approach to Battery Management in Dairy Farming via Proximal Policy Optimization”. In: Advances in Practical Applications of Agents, Multi-Agent Systems, and Digital Twins: The PAAMS Collection. Ed. by Philippe Mathieu and Fernando De la Prieta. Cham: Springer Nature Switzerland, 2025, pp. 15–26. isbn: 978-3-031-70415-4. Thi Ngoc Yen Huynh et al. “Multi objectives reinforcement learning for smart buildings: A systematic review of algorithms, applications and future perspectives”. In: Energy and Buildings 345 (2025), p. 116045. Hossein Khaleghy, Eoghan Clifford, and Karl Mason. “A Machine Learning Approach to Dairy Farm Energy Disaggregation”. In: 2024 IEEE 6th International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA). 2024, pp. 55–60. Ilias Sarantakos et al. “A Robust Mixed-Integer Convex Model for Optimal Scheduling of Integrated Energy Storage—Soft Open Point Devices”. In: IEEE Transactions on Smart Grid 13.5 (2022), pp. 4072–4087. Zhijun Zhang et al. “Economic-Driven Hierarchical Voltage Regulation of Incremental Distribution Networks: A Cloud-Edge Collaboration Based Perspective”. In: IEEE Transactions on Industrial Informatics 18.3 (2022), pp. 1746–1757. Jiang Wu et al. “Dynamic Pricing and Prices Spike Detection for Industrial Park With Coupled Electricity and Thermal Demand”. In: IEEE Transactions on Automation Science and Engineering (2022), pp. 1326–1337. John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707.06347 [cs.LG]. url: https://arxiv.org/abs/1707.06347.
8
M. E. Cruz Victorio et al.
Appendix A The parameters for differential evolution used in the Multi-Objective Layer are described in Table 1. Value Parameter Population 30 Mutation [0.5,1] Convergence Tolerance 0.01 Maximum number of iterations 1000 Crossover rate = 0.7
Table 1: Differential Evolution Parameters
Appendix B The distributed generation and storage capacity in the circuit model shown in Figure 1 is described in Table 2. Battery Solar PV Bus Pmax (kW) SOCmax (kWh) Bus Pmax (kW) 2 20 27 3 22.44 6 30 40.5 7 39.27 7 50 67.5 9 33.66 10 40 54 11 28.05 12 40 54 12 28.05 14 10 13.5 14 16.83
Table 2: Distributed generation specifications by Bus. The left side describes the parameters for the batteries and the right side describes the PV generation.