A Hierarchical MARL-Based Approach for Coordinated Retail P2P Trading and Wholesale Market Participation of DERs Patrick Wilka , Ethan Cantora , Yikui Liub,∗ and Jie Lia a Department Electrical and Computer Engineering, Rowan University, 201 Mullica Hill Rd, Glassboro, New Jersey, 08028, United States
arXiv:2604.20586v1 [cs.LG] 22 Apr 2026
b School of Electrical Engineering, Sichuan University, 24 South Section 1, 1st Ring Road, Chengdu, 610065, China
ARTICLE INFO
ABSTRACT
Keywords: aggregator electricity market peer-to-peer reinforcement learning
The ongoing shift towards decentralization of the electric energy sector, driven by the growing electrification across end-use sectors, and widespread adoption of distributed energy resources (DERs), necessitates their active participation in the electricity markets to support grid operations. Furthermore, with bi-directional energy and communication flows becoming standard, intelligent, easy-to-deploy, resource-conservative demand-side participation is expected to play a critical role in securing power grid operational flexibility and market efficiency. This work proposes a market engagement framework that leverages a hierarchical multi-agent deep reinforcement learning (MARL) approach to enable individual prosumers to participate in peer-to-peer retail auctions and further aggregate these intelligent prosumers to facilitate effective DER participation in wholesale markets. Ultimately, a Stackelberg game is proposed to coordinate this hierarchical MARL-based DER market participation framework toward enhanced market performance.
1. Introduction Historically, electricity markets were structured as vertically integrated monopolies, where a single utility owned and controlled all generation, transmission, and distribution segments. As this model limited competition and consumer choice, liberalization reforms separated generation, transmission, and distribution sectors, enabling multiple largescale suppliers to compete. Over time, however, the growing electrification and increasing integration of smaller-scale distributed energy resources (DERs) introduce new challenges to the emerging electricity market paradigm.
1.1. Background of DERs in Wholesale and Retail Markets Renewable DERs such as solar photovoltaics (PV) offer environmental benefits and diversify the energy mix. While their near-zero marginal cost can lower electricity prices, their intermittency could induce higher price volatility, especially in markets with large-scale renewable adoptions [1]. Although policies like Renewable Portfolio Standards (RPSs) have successfully accelerated decarbonization, they have also inadvertently increased retail electricity prices instead of reducing them. For instance, after 7 years of RPS implementation, retail electricity prices rose by 11%, partly due to indirect integration costs, stranded asset payments, and the complexity of managing intermittent supplies [2, 3]. Additionally, despite declining costs for both renewables and conventional generations, the U.S. retail electricity prices continued to rise, reaching 12.72 cents per kWh in 2023, a more than 85% increase since 2000 [4]. These increases stem from the investments in grid modernization, resiliency, and cyberphysical security measures. Given the depreciated ∗ Corresponding author
[email protected] (P. Wilk); [email protected] (E. Cantor); [email protected] (Y. Liu); [email protected] (J. Li) ORCID (s): 0000-0002-3292-6153 (Y. Liu)
P. Wilk et al.: Preprint submitted to Elsevier
value of U.S. electrical infrastructure at $1.5 trillion, its replacement cost may approach $5 trillion [5]. In parallel, the nation’s electricity sector has also seen a rapid proliferation of consumer side DERs, such as rooftop solar, microturbines, battery storage, electric vehicles, and flexible demand resources [6]. Such DERs turn the consumers into prosumers who can generate, store, and manage their energy assets. Although prosumers can reduce their energy bills and support grid operational efficiency by contributing to local balancing, the standard top-down control and static utility tariff structures fail to harness such flexibility and complicate distribution-side coordination [7]. Amid these challenges, local electricity markets (LEMs) emerged as a promising approach [8, 9]. By coordinating DERs at the distribution level, participants can trade energy among themselves rather than relying on utility transactions. Such local balancing can alleviate grid congestion, delay infrastructure investment, and maximize localized renewable consumption [10]. At scale, LEMs offer essential services to the upstream grid, providing critical support as renewables displace synchronous generators and the infrastructure becomes more stressed [11]. Within LEMs, peer-to-peer (P2P) trading is an innovative paradigm, facilitated by smart meters, communication networks, and advanced data analytics, allowing prosumers to directly negotiate and transact energy with each other [12, 13]. With the continued development of LEMs, growing research efforts have been directed toward more effective market design and coordination frameworks: • From Iterative-Based to Non-Iterative Coordination of DER Aggregators: Effective P2P markets require a robust mechanism for optimal bidding, price formation, and fair clearing to prevent market manipulation. Prosumers decide how much energy, at what price, and with which peer to trade. However, poorly coordinated P2P transactions may induce unexpected demand or generation peaks, causing grid operational challenges or Page 1 of 11
MARL-Based Coordinated P2P Electricity Trading
system stability issues [14]. Aggregators have emerged to bundle DERs by pooling heterogeneous prosumers into a “virtual” resource and negotiating on their behalf in wholesale markets. Through price-based mechanisms, aggregators can harness prosumers’ flexibility to align their retail bidding strategies with wholesale market signals [15]. Trials like EcoGrid EU confirm residential loads can be price sensitive [16]. However, research on aggregator-based coordination often relies on iterative pricing schemes, such as ADMM [17], that solve Nash bargaining problems by iteratively sharing price and energy data between the aggregator and participants [18, 19]. Moreover, these models [20, 21, 22], using prices to influence market behavior, can steer prosumer behavior toward the aggregator’s objectives but often require continuous, bidirectional communication, raising scalability concerns. • From Centralized to Hybrid Coordination Schemes: DER coordination in LEMs broadly follows centralized, decentralized, or hybrid approaches. Centralized models [11] typically rely on extensive information sharing by collecting all data, optimize dispatches, and manage local transactions. While this may achieve system-wide optima, it faces scalability bottlenecks, single-point failures, and privacy concerns. Decentralized models rely on bilateral contracts or consensus, alleviating scalability and privacy issues but risking suboptimal and convergence outcomes [13]. Hybrid methods seek to strike a balance by combining centralized coordination with decentralized decision-making, thereby retaining system-level coherence while preserving local autonomy [23]. • From Model-Based to Data-Driven Models: DER coordination studies often rely on model-based optimization under ideal conditions, assuming adequate knowledge of DER characteristics and predictable prosumer behavior [22, 23, 24, 25, 26]. Although theoretically capable of delivering optimal solutions, these approaches can be computationally expensive, require extensive modeling, and become unwieldy at large scales, posing challenges for adoption by resource-limited prosumers. Data-driven methods, specifically deep reinforcement learning (DRL), have recently gained attention in energy management and market applications [27]. Using the wealth of data, DRL agents could discover optimal decision-making policies from real or simulated experience, learning to manage uncertainties without requiring explicit system models [28]. DRL has been applied to optimize residential energy trading [29], EV charging strategies [30], and building energy management [31, 32]. However, scaling single-agent RL to a multi-agent context (MARL) is challenging due to large state-action spaces, non-stationarity, and agent heterogeneity. Current MARL falls into three categories: (i) Independent Learning [33, 34, 35], which can lead to uncoordinated outcomes and unstable training; (ii) Centralized Critic [36, 37], which mitigates non-stationarity but imposes P. Wilk et al.: Preprint submitted to Elsevier
computation and communication costs, and raises privacy issues; and (iii) Value Decomposition and Policy Sharing, suitable for cooperation but less effective when agents have divergent objectives [38]. Despite these developments, existing studies still largely treat retail P2P trading and wholesale market participation as two separate processes, making it difficult to capture the cross-layer interaction between local prosumer decisions and aggregator-level market actions. More specifically, a substantial proportion of studies, from the prosumer perspective, focuses on retail P2P trading mechanisms and corresponding pricing strategies, while either disregarding the utility or treating it as a fixed-price, capacity unconstrained trading counterparty for prosumers. The rest of the studies, from the aggregator perspective, focuses on pricing prosumers and participating in the wholesale market to maximize aggregator profit, while neglecting other trading channels available to prosumers. Intuitively, this separation results in inefficient coordination across market layers. Retail P2P trading decisions may deviate significantly from aggregator-level wholesale strategies, leading to economic losses for both prosumers and aggregators.
1.2. The Proposed Solution The next-generation electricity market scheme, spanning both wholesale and retail layers, holds the potential to create a level playing field where all customers have equal participation opportunities. We argue that this requires intelligent, easy-to-deploy decision-making tools and involves the gamification of the energy ecosystem to stimulate adoption. This work aims to explore a market solution to streamline DER aggregator operations and enhance prosumer engagement while reducing dependence on detailed system modeling and high-bandwidth communication, the two hurdles to effective DERs’ market participation. To this end, a new market design that explicitly bridges retail P2P trading and wholesale market participation is proposed. On this basis, a hierarchical DRL framework, i.e.,the Strategic distributed Energy resource Aggregation Model for Localized Efficient System Synchronization (SEAMLESS), combining aggregator-level Proximal Policy Optimization (PPO) policies with prosumer-level Local StrategyDriven Multi-Agent Deep Deterministic Policy Gradient (LSD-MADDPG) coordination is proposed, enabling noniterative coordination under limited-information sharing. The major contributions of this work are as follows: (i) Effective Market Design: A new market design is proposed to unify the retail P2P trading with wholesale transactions, empowering prosumers to compete for lower costs through mutual trades while the aggregator handles unmatched supply and demand in wholesale markets. Via dynamic pricing, the aggregator fosters prosumer participation and fair bidding while competing in the wholesale market, bridging grid operation layers. This decentralized, non-iterative bidding design, guided by aggregator signals, incentivizes DERs’ P2P participation while reducing market manipulation risks through Page 2 of 11
MARL-Based Coordinated P2P Electricity Trading
limited information sharing and centralized handling of unmatched P2P supply and demand. (ii) Limited-Information Sharing for Privacy-Preservation: LSD-MADDPG, a DRL solution customized for the retail P2P management, employs a decentralized actorcritic architecture that reduces non-stationarity and uncoordinated outcomes through limited information sharing, balancing cooperative and competitive objectives of individual DERs with reduced computational costs. In LSD-MADDPG, prosumer agents access only their local states and customized “strategies”, i.e., the aggregator state and price signals, during training. Such a design reduces information exchange complexity and offers a more realistic, secure, and scalable approach than fully centralized or distributed methods. (iii) A Hierarchical DRL Design: SEAM-LESS innovatively employs parallel PPO and LSD-MADDPG learning to coordinate DERs’ wholesale and P2P market participation. A Stackelberg game is used to model the coordination between aggregator-level PPO for wholesale market bidding with prosumer-level LSD-MADDPG for retail P2P decision-making and clearing. PPO’s stable policy updates enable the aggregator, as the Stackelberg leader, to provide consistent price signals, effectively guiding prosumers’ market behaviors. The remainder of this paper is organized as follows: Section 2 introduces the hierarchical DER aggregation framework, detailing the market rules and pricing structures. Section 3 formulates the Markov game for aggregator and prosumer coordination and the proposed Stackelberg game, defining SEAM-LESS architecture. Section 4 conducts case studies, discussing simulation results and comparative analyses of the proposed with a conventional centralized critic MADDPG and rule-based market solutions. Finally, Section 5 concludes with key insights and future research directions.
2. Hierarchically Coordinated Wholesale and P2P Market Participation of DERS Figure 1 illustrates the physical and market structures of the proposed hierarchical DER market participation framework. In Figure. 1a, the physical network shows how electric energy flows between DERs/prosumers connected to the distribution and transmission grid. In Figure. 1b, the hierarchical wholesale-retail market reveals how the prosumers, aggregator, utility, and ISO/RTO interact via digital platforms. Specifically, the DER aggregator interfaces with the retail P2P market, utility, and wholesale market, facilitating transactions in terms of energy quantity and price. Decoupling the physical and financial layers promotes market functioning via digital clearinghouses while maintaining the grid in its technical limits. In the retail market, the P2P protocol matches bids and offers to form financial agreements among prosumers, while the aggregator observes prosumer consumption and generation patterns, adapts its wholesale market strategies, and exploits price differentials across market layers. Through these actions, the aggregator promotes local energy exchanges, P. Wilk et al.: Preprint submitted to Elsevier
coordinates dynamically with the wholesale market, and seeks to profit. The P2P market aims to maximize the combined net benefits of buyers in set 𝑋 and sellers in set 𝑌 (1). Each buyer 𝑥 derives value 𝑉𝑥 (𝑄𝑥 ) from the energy it purchases, while each seller 𝑦 incurs cost 𝐶𝑦 (𝑄𝑦 ) for producing the energy it sells. Each transaction 𝑄𝑥,𝑦 between buyer 𝑥 and seller 𝑦 has a price 𝑃𝑥,𝑦 . [ ] ∑ ∑ max 𝑉𝑥 (𝑄𝑥 ) − 𝑃𝑥,𝑦 ⋅ 𝑄𝑥,𝑦 𝑥∈𝑋 𝑦∈𝑌 (1) [∑ ] ∑ + 𝑃𝑥,𝑦 ⋅ 𝑄𝑥,𝑦 − 𝐶𝑦 (𝑄𝑦 ) 𝑦∈𝑌
𝑥∈𝑋
∑
where 𝑄𝑥 = 𝑦∈𝑌 𝑄𝑥,𝑦 is the total quantity purchased by ∑ buyer 𝑥 and 𝑄𝑦 = 𝑥∈𝑋 𝑄𝑥,𝑦 is the total quantity sold by seller 𝑦. The DER aggregator seeks to maximize its own profit (2) by managing unsettled bids/offers from the P2P market via wholesale transactions at price 𝑝𝑤 𝑎 . The aggregator sells energy to the P2P market at price 𝑝𝑠𝑎 and acquires surplus at 𝑝𝑏𝑎 . Unsettled supply from P2P sellers is 𝑄𝑎,𝑦 , and unsettled demand from P2P buyers is 𝑄𝑎,𝑥 . ∑ ∑ 𝑝𝑏 ⋅ 𝑄𝑎,𝑦 max 𝑝𝑠𝑎 ⋅ 𝑄𝑎,𝑥 − 𝑦∈𝑌 𝑎 𝑥∈𝑋 ) (∑ (2) ∑ 𝑄 𝑄 − −𝑝𝑤 𝑎,𝑦 𝑎,𝑥 𝑎 𝑦∈𝑌
𝑥∈𝑋
2.1. P2P Retail Market Design The retail P2P market is executed in discrete, equal time intervals. During each time interval, participants submit their bids/offers, and the P2P market settles all trades in a single round before proceeding to the next time interval. A P2P market participant could declare itself as a buyer or a seller based on its assets’ evolving conditions (e.g., generation/storage capacity, demand flexibility) and strategic considerations at each time interval. The P2P market design is elaborated via the following three steps, where 𝑛 indexes prosumers (buyers and sellers) in the P2P market participant set 𝑁 = 𝑁𝐵 ∪ 𝑁𝑂 , with 𝑁𝐵 being the buyer set, 𝑁𝑂 being the seller set, and 𝑁𝐵 ∩ 𝑁𝑂 = ∅. Step 1: Bid/Offer Submission and Sorting. Each buyer 𝑛∈𝑁𝐵 submits a bid 𝑏𝑛 characterized by a maximum price 𝑝𝑏𝑛 , at which it is willing to pay for a desired quantity 𝑞𝑏𝑛 < 0 (negative denotes energy purchasing). Similarly, each seller 𝑛 ∈ 𝑁𝑂 submits an offer 𝑜𝑛 with a minimum acceptable price 𝑝𝑜𝑛 for quantity 𝑞𝑜𝑛 > 0 (positive denotes energy selling). All bids are collected in (3) and all offers in (4). { ( )| } 𝐵 = 𝑏𝑛 𝑝𝑏𝑛 , 𝑞𝑏𝑛 || 𝑛 ∈ 𝑁𝐵 (3) | { ( )| } 𝑂 = 𝑜𝑛 𝑝𝑜𝑛 , 𝑞𝑜𝑛 || 𝑛 ∈ 𝑁𝑂 (4) | Once all bids and offers are submitted, the P2P platform sorts all bids in descending price order (5), and all offers in ascending price order (6), enabling efficient matching of the highest-paying buyers with the lowest-priced sellers. { } 𝐵𝑘∗ = 𝑏∗1,𝑘 , 𝑏∗2,𝑘 , … , 𝑏∗𝑛 ,𝑘 with 𝑝𝑏∗ ≥ 𝑝𝑏∗ ≥⋯≥ 𝑝𝑏∗ (5) 𝑏
1,𝑘
2,𝑘
𝑛𝑏 ,𝑘
Page 3 of 11
MARL-Based Coordinated P2P Electricity Trading
share the same lowest price at round 𝑘, the total supply is the summation (11), and each seller’s proportion of the total supply at this price is (12). } { | (7) 𝐵𝑝,𝑘 = 𝑏∗𝑖,𝑘 ∈ 𝐵𝑘∗ || 𝑝𝑏∗ = 𝑝𝑏∗ 𝑘 | 𝑖,𝑘 { } | 𝑂𝑝,𝑘 = 𝑜∗𝑗,𝑘 ∈ 𝑂𝑘∗ || 𝑝𝑜∗ = 𝑝𝑜∗ (8) 𝑘 | 𝑗,𝑘 ) ( ∑ 𝑄(𝑘) = (9) −𝑞𝑏∗ ∗ 𝑇𝐷 𝑏𝑖,𝑘 ∈𝐵𝑝,𝑘
𝑖,𝑘
(𝑘) 𝛼𝑏(𝑘) ∗ = −𝑞𝑏∗ ∕𝑄𝑇 𝐷
𝑄(𝑘) = 𝑇𝑆
(10)
𝑖,𝑘
𝑖,𝑘
∑
(11)
𝑞∗ 𝑜∗𝑗,𝑘 ∈𝑂𝑝,𝑘 𝑜𝑗,𝑘
(𝑘) 𝛽𝑜(𝑘) ∗ = 𝑞𝑜∗ ∕𝑄𝑇 𝑆
(a) Physical Network Structure
(12)
𝑗,𝑘
𝑗,𝑘
The total transaction quantity that can be matched at round 𝑘 is limited by the smaller of total demand and total supply (13). 𝑞𝑘 is then allocated proportionally to individual participants for updated quantities for buyers (14) and sellers (15). ( ) (𝑘) 𝑞𝑘 = min 𝑄(𝑘) , 𝑄 (13) 𝑇𝐷 𝑇𝑆 𝑞𝑏𝑡𝑥,(𝑘) = 𝛼𝑏(𝑘) ∗ ∗ × 𝑞𝑘 ,
∀ 𝑏∗𝑖,𝑘 ∈ 𝐵𝑝,𝑘
(14)
𝑞𝑜𝑡𝑥,(𝑘) = 𝛽𝑜(𝑘) ∗ ∗ × 𝑞𝑘 ,
∀ 𝑜∗𝑗,𝑘 ∈ 𝑂𝑝,𝑘
(15)
𝑖,𝑘
𝑖,𝑘
𝑗,𝑘
(b) Hierarchical Market Structure Figure 1: Proposed Wholesale and P2P Market Participation Framework of DERs.
𝑗,𝑘
The transaction price 𝑝𝑘 is set as the midpoint between the highest buyer price and the lowest seller price (16). This design ensures fairness, as buyers pay less than or equal to their maximum willingness-to-pay, and sellers receive at least their minimum acceptable price. 𝑘
} { 𝑂𝑘∗ = 𝑜∗1,𝑘 , 𝑜∗2,𝑘 , … , 𝑜∗𝑛 ,𝑘 with 𝑝𝑜∗ ≤ 𝑝𝑜∗ ≤⋯≤𝑝𝑜∗ 𝑜
1,𝑘
𝑛𝑏 ,𝑘
2,𝑘
(6)
Here, 𝑛𝑏 = |𝑁𝐵 | and 𝑛𝑜 = |𝑁𝑂 | represent the number of bids and offers in current time interval. The superscript (*) denotes the sorted lists, and 𝑘 indexes the round of the matching process. 𝑏∗𝑖,𝑘 is the 𝑖-th highest-priced bid and 𝑜∗𝑗,𝑘 is the 𝑗-th lowest priced offer at round 𝑘. Step 2: Price Matching and Proportional Allocation. For the 𝑘-th round, identify: • The highest bid 𝑏∗𝑘 = 𝑏∗1,𝑘 with 𝑝𝑏∗ = 𝑝𝑏∗ , 𝑞𝑏∗ = 𝑞𝑏∗ 𝑘 𝑘 1,𝑘 1,𝑘 • The lowest bid 𝑜∗𝑘 = 𝑜∗1,𝑘 with 𝑝𝑜∗ = 𝑝𝑜∗ , 𝑞𝑜∗ = 𝑞𝑜∗ 𝑘
1,𝑘
𝑘
1,𝑘
If 𝑝𝑏∗ ≥ 𝑝𝑜∗ , a buy/sell match is possible, meaning at 𝑘 𝑘 least one buyer is willing to pay no lower than the seller requires; otherwise, no transaction occurs. For multiple buyers or sellers sharing the same price, we group them as (7) and (8). All buyers in 𝐵𝑝,𝑘 have the same highest price at round 𝑘, the total demand at this price is calculated as (9), and each buyer’s proportion of the total demand at this price is (10). Similarly, all sellers in 𝑂𝑝,𝑘 P. Wilk et al.: Preprint submitted to Elsevier
(16)
𝑝𝑘 = (𝑝𝑏∗ + 𝑝𝑜∗ )∕2 𝑘
Finally, buyer payments and seller revenues for round 𝑘 are (17) and (18), respectively. 𝑃 𝐴(𝑏∗𝑖,𝑘 )(𝑘) = 𝑞𝑏𝑡𝑥,(𝑘) × 𝑝𝑘 ∗
(17)
𝑅𝑉 (𝑜∗𝑗,𝑘 )(𝑘) = 𝑞𝑜𝑡𝑥,(𝑘) × 𝑝𝑘 ∗
(18)
𝑖,𝑘
𝑗,𝑘
The remaining quantities for each participant are updated with transacted amounts (19) and (20), and any fully cleared bids/offers (e.g., 𝑞𝑏∗ = 0 or 𝑞𝑜∗ = 0) are removed from the 𝑖,𝑘 𝑗,𝑘 active lists. 𝑃 𝐴𝑛 and 𝑅𝑉𝑛 are updated via (21) and (22) to record the total amount paid by a buyer and the total revenue earned by a seller across all P2P transactions up to round 𝑘. 𝑞𝑏∗ ← 𝑞𝑏∗ + 𝑞𝑏𝑡𝑥,(𝑘) (19) ∗ 𝑖,𝑘
𝑖,𝑘
𝑖,𝑘
𝑞𝑜∗ ← 𝑞𝑜∗ + 𝑞𝑜𝑡𝑥,(𝑘) ∗ 𝑗,𝑘 𝑗,𝑘 𝑗,𝑘
(20)
𝑃 𝐴𝑛 ← 𝑃 𝐴𝑛 + 𝑃 𝐴(𝑏∗𝑖,𝑘 )(𝑘) ,
𝑛 ∈ 𝑁𝐵
(21)
𝑅𝑉𝑛 ← 𝑅𝑉𝑛 + 𝑅𝑉 (𝑜∗𝑗,𝑘 )(𝑘) ,
𝑛 ∈ 𝑁𝑂
(22)
Page 4 of 11
MARL-Based Coordinated P2P Electricity Trading
After adjusting the lists, set 𝑘 = 𝑘 + 1 and repeat this matching process until 𝑝𝑏∗ < 𝑝𝑜∗ , or one list is empty. 𝑘 𝑘 Step 3: Unmatched Quantities. After all P2P transactions are cleared, some bids and/or offers may not be fully matched, and will proceed to the aggregator for further settlement.
2.2. Aggregator’s Wholesale Market Participation During a time interval, each participant’s unmatched quantity 𝑞𝑏𝑎 or 𝑞𝑜𝑎 is handled by the aggregator, which sums 𝑛 𝑛 them as in (23) for wholesale trading. ∑ ∑ 𝑞𝑎𝑤 = 𝑞𝑏𝑎 + 𝑞𝑜𝑎 (23) 𝑛∈𝑁𝐵
𝑛∈𝑁𝑂
𝑛
𝑛
If 𝑞𝑎𝑤 > 0, the aggregator is a net seller during that time
interval in the wholesale market and otherwise, a net buyer. The aggregator chooses a price 𝑝𝑤 𝑎 at which it attempts to transact in the wholesale market. To be a successful wholesale seller, the aggregator must offer a price no higher than the wholesale clearing price; and for a successful purchase, it must bid no lower than the wholesale clearing price. If the aggregator fails to clear in the wholesale market, it incurs a penalty paid to the P2P participants, proportional to their unfulfilled quantities 𝑞𝑜𝑎 and 𝑞𝑏𝑎 . The aggregator 𝑛 𝑛 must then buy from or sell to the utility at preset prices on behalf of these participants. If successful, the aggregator transacts at prices relative to the wholesale clearing price 𝑃𝑤 to buy (24) and sell (25) unmatched P2P quantities. Δ𝑏 and Δ𝑜 represent revenue margins around the wholesale price, granting the aggregator flexibility to profit by acquiring P2P energy below 𝑃𝑤 and selling above it. ) ( (24) 𝑝𝑏𝑎 = 𝑃𝑤 × 1 − Δ𝑏 ) ( (25) 𝑝𝑠𝑎 = 𝑃𝑤 × 1 + Δ𝑜 The payments of the P2P buyer to and revenue of the P2P seller from the aggregator are (26) and (27), respectively. 𝑃 𝐴𝑎𝑛 = 𝑞𝑏𝑎 × 𝑝𝑠𝑎 ,
𝑛 ∈ 𝑁𝐵
(26)
𝑅𝑉𝑛𝑎 = 𝑞𝑜𝑎 × 𝑝𝑏𝑎 , 𝑛
𝑛 ∈ 𝑁𝑂
(27)
𝑛
If the aggregator fails to clear in the wholesale market, it pays penalty 𝑃 𝑁𝑛 , proportional to the unfulfilled quantities (28-29). ( ) 𝑃 𝑁𝑛 = 𝑝𝑎𝑝 ⋅ − 𝑞𝑏𝑎 , 𝑛 ∈ 𝑁𝐵 (28) 𝑛
𝑃 𝑁𝑛 = 𝑝𝑎𝑝 ⋅
( 𝑎) 𝑞𝑜 , 𝑛
𝑛 ∈ 𝑁𝑂
(29)
The aggregator’s profit (30) includes payments from/to P2P buyers/sellers, transactions settled in the wholesale market, and penalties incurred if the wholesale market attempt fails. ∑ ∑ ( ) ∑ 𝑅𝑉𝑛𝑎 + 𝑃𝑤 ⋅𝑞𝑎𝑤 − 𝑃 𝑁𝑛 𝑇𝐴 = − 𝑃 𝐴𝑎𝑛 − 𝑛∈𝑁𝐵 𝑛∈𝑁𝑂 𝑛 (30) Final payouts of P2P buyer (31) and seller (32) incorporate P2P transactions and settlements with the aggregator. 𝑇 𝐵𝑛 = 𝑃 𝐴𝑛 + 𝑃 𝐴𝑎𝑛 + 𝑃 𝑁𝑛 ,
𝑛 ∈ 𝑁𝐵
P. Wilk et al.: Preprint submitted to Elsevier
𝑇 𝑆𝑛 = 𝑅𝑉𝑛 + 𝑅𝑉𝑛𝑎 + 𝑃 𝑁𝑛 ,
𝑛 ∈ 𝑁𝑂
(32)
The proposed hierarchical market framework spans a 24-hour horizon. For each hour 𝑡, the aggregator posts its purchase and sell prices, 𝑝𝑏𝑎 (𝑡) and 𝑝𝑠𝑎 (𝑡), to the P2P market before participants submit bids and offers for that hour. After the P2P market clears, settled prices and quantities are allocated among participants, with any unmatched bids and/or offers tracked as residuals. Once all 24 hours’ P2P transactions are processed, the aggregator compiles hourly residuals and transacts with the wholesale market or utility on behalf of prosumers, aiming to optimize its profit (30). Following wholesale settlements, the aggregator distributes net profits or costs back to the P2P participants. This approach accommodates the single round 24-hour day-ahead wholesale market while allowing hourly price-setting between the aggregator and local P2P participants.
3. DRL-Based Stackelberg Game Framework For Market Interactions The above proposed retail P2P market and the aggregator’s participation in the wholesale market can be formulated as a Markov game. In this setting, both individual prosumers (i.e., P2P market participants) and the aggregator operate as strategic RL agents, making decisions along with evolving system and market states. This captures dynamic behaviors and interactions between the aggregator and participants and among participants. We first describe the Markov game formulations for aggregator and prosumer coordination, then present the proposed SEAM-LESS architecture based on the Stackelberg game framework.
3.1. Markov Game for Market Interactions 3.1.1. P2P Market Agents State and Action Space The state space of a P2P market agent includes all variables influencing operational and market dynamics: avg
𝑠𝑛 = {𝑝𝑏𝑎 , 𝑝𝑠𝑎 , 𝑞𝑛tot , 𝑝𝑛 , 𝑞cm , 𝑇 𝑃𝑛 }
(33)
For agent 𝑛, the total quantity 𝑞𝑛𝑡𝑜𝑡 is 𝑞𝑏𝑛 if it is a buyer or 𝑞𝑜𝑛 if it is a seller. The total payoff (or profit) 𝑇 𝑃𝑛 is defined in (34). The average P2P trade price of agent 𝑛 is 𝑝𝑎𝑣𝑔 𝑛 (35), and the aggregated total net quantity of all prosumers is 𝑞𝑐𝑚 (36). { 𝑇 𝐵𝑛 , if 𝑛 ∈ 𝑁𝐵 𝑇 𝑃𝑛 = (34) 𝑇 𝑆𝑛 , if 𝑛 ∈ 𝑁𝑂 𝑡𝑜𝑡 𝑝𝑎𝑣𝑔 𝑛 = 𝑇 𝑃𝑛 ∕𝑞𝑛 ∑ 𝑞𝑐𝑚 = 𝑞𝑛𝑡𝑜𝑡 𝑛
(35) (36)
Agent 𝑛’s action space 𝑎𝑛 = [𝑎𝑝𝑛 ] is designed to adjust its bid/offer within predefined bounds [𝑝𝑚𝑖𝑛 , 𝑝𝑚𝑎𝑥 ] (37) where 𝑎𝑝𝑛 ∈ [−1, 1] represents a normalized action. 𝑝𝑏𝑛 ∕𝑝𝑜𝑛 = ((𝑎𝑝𝑛+1)∕2)⋅(𝑝max−𝑝min )+𝑝min , 𝑛 ∈ 𝑁𝐵 ∕𝑁𝑂 (37)
(31) Page 5 of 11
MARL-Based Coordinated P2P Electricity Trading
3.1.2. Aggregator Agent State and Action Space The aggregator agent acts as an intermediary between the P2P and wholesale markets, aiming to align local trading with broader system objectives and wholesale price signals. Its state space is defined as (38). 𝑤 𝑤 𝑠𝑎𝑔 = {𝐹𝑀𝑃 , 𝐹𝐼𝑃 , 𝑞𝑎,𝑜 , 𝑞𝑎,𝑏 } (38) 𝐹𝑀𝑃 provides the aggregator with insight into the expected wholesale clearing price in the current trading period, while the subsequent period’s wholesale clearing price forecast 𝐹𝐼𝑃 further guides the aggregator’s buy/sell price settings to shape its behavior in the retail P2P market. The 𝑤 (39) and demand 𝑞 𝑤 (40) from aggregated supply 𝑞𝑎,𝑜 𝑎,𝑏 the P2P market define the net local energy balance to be managed by the aggregator: ∑ 𝑤 𝑞𝑎,𝑜 = 𝑞𝑜𝑎 (39) 𝑛∈𝑁𝑂
𝑤 𝑞𝑎,𝑏 =
∑
𝑛
(40)
𝑞𝑎 𝑛∈𝑁𝐵 𝑏𝑛
The aggregator’s action space (41) includes setting the wholesale market bid and adjusting its P2P buy/sell prices: (41)
𝑜 𝑏 𝑎𝑎𝑔 = {𝑎𝑤 𝑎𝑔 , 𝑎𝑎𝑔 , 𝑎𝑎𝑔 }
The wholesale bidding price 𝑝𝑤 𝑎 (43) is constrained be𝑚𝑎𝑥 𝑤 tween 𝑝𝑚𝑖𝑛 𝑎𝑔 and 𝑝𝑎𝑔 via the normalized action 𝑎𝑎𝑔 ∈ [0, 1]. 𝑚𝑖𝑛 𝑤 𝑚𝑎𝑥 𝑝𝑤 𝑎 = 𝑝𝑎𝑔 ⋅ 𝑎𝑎𝑔 + 𝑝𝑎𝑔
(42)
The aggregator’s buy action 𝑎𝑏𝑎𝑔 ∈ [0, 1] and offer action ∈ [0, 1] in P2P market are defined as fractions of the maximum allowable markup 𝜌𝑚𝑎𝑥 applied to the forecasted wholesale price 𝐹𝑀𝑃 . The adjustments Δ𝑏 and Δ𝑜 , used to determine the aggregator’s buy and sell prices in the P2P market (as defined in (24) and (25)), are then given by (43) and (44). (43) Δ𝑏 = 𝑎𝑏𝑎𝑔 ⋅ 𝜌max 𝑎𝑜𝑎𝑔
( ) Δ𝑆𝑛 = max 0, 𝑝𝑜𝑛 − 𝑝𝑏𝑎 , Π𝑆𝑛 = 𝑞𝑜𝑎 × Δ𝑆𝑛 , 𝑛 ( ) 𝐵 𝑠 𝐵 𝑎 Δ𝑛 = max 0, 𝑝𝑎 − 𝑝𝑏𝑛 , Π𝑛 = 𝑞𝑏 × Δ𝐵 𝑛, 𝑛 { Π𝑆𝑛 , if 𝑛 ∈ 𝑁𝑂 Π𝑛 = Π𝐵 𝑛 , if 𝑛 ∈ 𝑁𝐵 𝑟𝑛 = 𝑟∗𝑛 − Π𝑛
𝑛 ∈ 𝑁𝑂 (46) 𝑛 ∈ 𝑁𝐵 (47) (48) (49)
The aggregator reward 𝑟𝑎𝑔 depends on successful arbitrage and alignment with wholesale market conditions (50), representing net gains from wholesale transactions and P2P redistributions. 𝑟𝑎𝑔 = 𝑇 𝐴 (50)
3.2. Hierarchical DRL-Based Stackelberg Game Framework A hierarchical DRL framework, i.e., SEAM-LESS, is proposed under a Stackelberg game setup to solve the Markov game, as shown in Figure 2. The aggregator agent, as the game leader, sets price signals to define its wholesale and retail market policies, while the prosumer agents, as the game followers, learn local control policies to govern their decision-making in the retail market. To establish a fair and privacy-preserving retail market, prosumers are designed with limited access to the information of their peers. Although a centralized operator with full observability could solve the retail market clearing problem, it inevitably compromises the privacy of market participants, lacks scalability, and faces increased computational complexity as the number of P2P participants increases. Instead, LSD-MADDPG, a model-free MARL scheme proposed in [38], is adopted and customized.
(44)
Δ𝑜 = 𝑎𝑜𝑎𝑔 ⋅ 𝜌max
3.1.3. Reward Structures The base reward (45) quantifies each agent’s cost savings or revenue via P2P trading compared to the aggregator fallback. A P2P participant’s cost or revenue (𝑃 𝐴𝑛 or 𝑅𝑉𝑛 ) is compared to that it would have obtained from the aggregator for the portion traded in the P2P market. ⎧ 𝑅𝑉𝑛 − (𝑞𝑜𝑛 − 𝑞𝑛𝑢𝑛 ) × 𝑝𝑏𝑎 ,
⎪ 𝑟∗𝑛 = ⎨
if 𝑛 ∈ 𝑁𝑂
(45)
if 𝑛 ∈ 𝑁𝐵 ⎩ To ensure continuous feedback from action selection, a penalty Π𝑛 is introduced when the P2P participant’s unmatched energy 𝑞𝑛𝑢𝑛 stems from an uncompetitive P2P bid/offer (46-48). This helps agents detect uncompetitive pricing, balancing exploration (agents can safely try moderate price variations) with exploitation (excessively high or low prices incur aggregator fallback). The result is a clear signal reflecting the “missed opportunity” of potential savings. The final agent reward 𝑟𝑛 is the base reward minus penalty (49). If no trade occurs, 𝑟∗𝑛 = 0. ⎪−𝑃 𝐴𝑛 − (−𝑞𝑏𝑛 + 𝑞𝑛𝑢𝑛 ) × 𝑝𝑠𝑎 ,
P. Wilk et al.: Preprint submitted to Elsevier
Figure 2: SEAM-LESS Stackelberg Game Framework.
The proposed SEAM-LESS framework employs PPO for the aggregator and LSD-MADDPG for prosumers, leveraging their suitability for a non-iterative, hierarchical DRL approach in a Stackelberg game structure. In practical wholesale markets, bids are submitted once per period (e.g., hourly) and remain fixed, aligning with RL’s single-action policies that ensure timely, computationally efficient decisions, unlike iterative methods that risk manipulation through intra-period adjustments. This design is adopted in the proposed hierarchical wholesale-retail framework, where PPO’s clipped surrogate objective provides stable Page 6 of 11
MARL-Based Coordinated P2P Electricity Trading
price signals, critical for guiding prosumer behavior as the Stackelberg leader, while its proficiency in continuous action spaces and high-dimensional inputs supports precise wholesale bidding. LSD-MADDPG’s decentralized architecture limits information sharing to local observations and aggregator signals, unlike conventional MADDPG, which assumes that each agent’s critic accesses all other agents’ observations and actions, ensuring privacy and scalability. Furthermore, its ability to balance cooperative and competitive behaviors enables effective P2P coordination, validated by the authors’ prior success in community energy management [39]. In LSD-MADDPG’s actor-critic scheme, prosumer agents’ critics use their local observations and shared “strategies” from the aggregator to facilitate coordinated policy learning. This design respects practical physical constraints and upholds data confidentiality, critical to an effective retail energy market. By designing rewards to align individual prosumer goals with the aggregator’s high-level incentives, LSD-MADDPG ensures that local decisions collectively yield successful P2P settlements. During training, the aggregator agent broadcasts its price signals globally to the P2P market as a game leader using a PPO policy, while LSD-MADDPG agents, as game followers, observe their own local states and aggregator’s signals and state to make decisions. This design ensures that the aggregator maintains full transparency, a fundamental principle for an open and effective P2P market, while helping prosumers understand market conditions and the impact of their actions on market clearing. Consequently, it reduces market manipulation and promotes rational bidding, enhancing P2P market stability and integrity. By combining cooperative (agents collectively responding to aggregator signals and state) and competitive (each agent pursues individual market gains) market behaviors, SEAM-LESS excels in this mixed MARL environment. As shown in Algorithm 1, the aggregator’s PPO policy operates at a higher tier as the leader. The aggregator observes the wholesale and retail market conditions and sets a wholesale market bid/offer price 𝑎𝑤 𝑎𝑔 and P2P retail market 𝑜 𝑏 incentive signals [𝑎𝑎𝑔 , 𝑎𝑎𝑔 ]. These signals are broadcasted to all the prosumers in the P2P market, which run LSDMADDPG follower policies. Each LSD-MADDPG agent receives its local observation 𝑠𝑛 and the aggregator’s updated state 𝑠𝑎𝑔 , enabling it to determine actions 𝑎𝑛 that adjust its P2P trading profile. Any residual energy not settled in the P2P market is managed by the aggregator. A dynamic feedback loop emerges as the wholesale transaction success of the aggregator is dependent on the responsiveness of all the LSD-MADDPG agents. This closed-loop interaction fosters a self-regulating market where agents learn to trade optimally under evolving incentives.
4. Case Study The effectiveness of the proposed SEAM-LESS for coordinated wholesale and P2P retail market participation of DERs is evaluated against two rule-based (RB) and one conventional MARL market strategies: P. Wilk et al.: Preprint submitted to Elsevier
Algorithm 1 Training of SEAM-LESS 1: Initialize each LSD-MADDPG agent 𝑛 and PPO agent: 𝑄 𝜇 • Initialize actor network 𝜇𝑛 with 𝜃𝑛 𝑛 and critic network 𝑄𝑛 with 𝜃𝑛 𝑛 ; 𝜇′
𝜇
• Initialize target networks 𝜇𝑛′ and 𝑄′𝑛 with weights 𝜃𝑛 𝑛 ← 𝜃𝑛 𝑛 and 𝑄′ 𝑄 𝜃𝑛 𝑛 ← 𝜃𝑛 𝑛 ;
• Initialize LSD experience replay buffer ; • Initialize PPO parameters, including actor network 𝜋𝜃 and critic network 𝑉𝜙 . PPO uses its own data buffer for trajectories. 2: Training Loop for each episode 𝑒 = 1,2,. . . : • Reset the environment: get initial observation 𝑠0𝑛 for each LSDMADDPG agent 𝑛, and get 𝑠0𝑎𝑔 initial state for PPO. • For each timestep 𝑡 until the terminal state: A. LSD-MADDPG Agents’ Actions: Each agent 𝑛 observes (𝑠𝑡𝑛 , 𝑠𝑡𝑎𝑔 ). Select action 𝑎𝑡𝑛 = 𝜇𝑛 (𝑜𝑡𝑛 ). B. Partial Environment } { Step: i. Execute actions 𝑎𝑡𝑛 in LSD-MADDPG layer; i. Update 𝑠𝑡𝑎𝑔 resulting in 𝑠𝑡,∗ 𝑎𝑔 . 𝑡 C. PPO Actions: PPO observes 𝑠𝑡,∗ 𝑎𝑔 . Select action 𝑎𝑎𝑔
∼
𝜋𝜃 (𝑎𝑡𝑎𝑔 |𝑠𝑡,∗ 𝑎𝑔 ). D. Finalized Environment Step: ′ ′ i. Execute actions 𝑎𝑡𝑎𝑔 . Next states for LSD 𝑠𝑡𝑛 and PPO 𝑠𝑡𝑎𝑔 ; i. Rewards 𝑟𝑡𝑛 and 𝑟𝑡𝑎𝑔 . Get terminal signals done. E. Updates for LSD-MADDPG and PPO: i. Every 𝛿 steps for LSD-MADDPG sample: ′ ′ – Batch of 𝜌 transitions (𝑠𝑗𝑛 , 𝑎𝑗𝑛 , 𝑟𝑗𝑛 , 𝑠𝑗𝑛 , 𝑠𝑗𝑎𝑔 , 𝑠𝑗𝑎𝑔 ) from ; 𝑄 1 ∑𝜌 (𝑦𝑗 − – Compute critic loss: 𝐿(𝜃𝑛 𝑛 ) = 𝑗=1 𝑛 𝜌 𝑄𝑛 (𝑠𝑗𝑛 , 𝑎𝑗𝑛 , 𝑠𝑗𝑎𝑔 ))2 ; 𝑄
𝑄
– Perform gradient descent on 𝐿(𝜃𝑛 𝑛 ) to update 𝜃𝑛 𝑛 ; 𝜇 – Compute policy gradient: ∇𝜃 𝜇𝑛 𝐽 (𝜃𝑛 𝑛 ) 𝑗 𝑗 𝑗 𝑗 𝔼[∇𝜃𝜇𝑛 𝜇𝑛 (𝑠𝑛 )∇𝑎𝑗 𝑄𝑛 (𝑠𝑛 , 𝑎𝑛 , 𝑠𝑎𝑔 )]; 𝑛
𝜇
=
𝜇
– Perform gradient ascent on ∇𝜃𝜇𝑛 𝐽 (𝜃𝑛 𝑛 ) to update 𝜃𝑛 𝑛 ; – Update Target Networks. i. After collecting 𝑇ℎ𝑜𝑟𝑖𝑧𝑜𝑛 : – Compute advantages 𝐴𝑡 and targets for value function; – PPO Objective: 𝐿𝐶𝐿𝐼𝑃 (𝜃) [ ( Clipped ( ) )] 𝔼̂ 𝑡 min 𝑟𝑎𝑔,𝑡 (𝜃)𝐴𝑡 , 𝑐𝑙𝑖𝑝 𝑟𝑎𝑔,𝑡 (𝜃), 1 − 𝜖, 1 + 𝜖 𝐴𝑡 – Update Target Networks 𝜋𝜃 , 𝑉𝜙 by gradient steps. 3: Termination and Finalization • 𝑒 ← 𝑒 + 1 and go to 2, until 𝑒 = 𝑒𝑓 𝑖𝑛𝑎𝑙
=
• RB Agg: All retail energy trades are handled by the aggregator, with no P2P transactions among prosumers. The aggregator is assumed to always clear its trades in the wholesale market and sells to/buys from the prosumers at the maximum markup 𝜌𝑚𝑎𝑥 . • RB P2P: All retail P2P participants settle at a uniform clearing price, providing consistent trading conditions for buyers and sellers. Any surplus or deficit after P2P trades is handled by the aggregator, who always clears in the wholesale market and sells to/buys from the prosumers at the maximum markup 𝜌𝑚𝑎𝑥 . • MARL-Conv: Instead of LSD-MADDPG which maintains information privacy during centralized training, a conventional MADDPG is adopted. In this setting, all MADDPG agents share observations and actions with their centralized critic, while PPO shares only its actions. The simulation parameters for MADDPG and LSDMADDPG, as well as PPO are presented in Table 1. Each episode was simulated over a 24-hour period, from (00:00) to (23:59), corresponding to the hourly day-ahead market.
Page 7 of 11
MARL-Based Coordinated P2P Electricity Trading Table 2 Bidding Specifications Attributes Prosumer Min/Max Bidding Price Aggregator Min/Max Price Markup Aggregator Min/Max Price
Figure 3: Time-series of P2P participants energy overlaid with hourly wholesale price forecasts in Case I.
Table 2 provides the permissible bidding specifications for the P2P and wholesale markets, including prosumers and aggregator price ranges and maximum markup. These values can be adapted to specific agreements or market rules, thereby defining the feasible cost bounds for participants in different markets.
4.1. Effectiveness of the Proposed SEAM-LESS Three cases are studied to compare the four market strategies: Case I: Each P2P participant is designated as either a buyer or seller: prosumers 1 and 3 hold negative quantities (as buyer), while prosumers 2 and 4 always maintain positive quantities (as seller). Figure 3 illustrates the evaluation data, where all prosumers’ hourly quantities and wholesale prices are randomly varied by ±5 units during training. Case II: In contrast to Case I, where sellers hold market power due to scarce and valuable supply, Case II tests the inverse scenario by negating all participants’ quantities, so prosumers 1 and 3 become sellers and prosumers 2 and 4 become buyers. This setup evaluates how buyers wield market power in the P2P market with reduced demand. Case III: This case uses PJM wholesale market data [13] from a 2023 year-long dataset, as well as practical PV generation and energy consumption profiles of buildings from [39] to simulate the prosumers. Prosumers 1 and 2, with integrated solar PV systems, may switch their P2P market role from buyer to seller when their onsite PV generation exceeds energy demand during a time interval, while the remaining two prosumers remain as buyers throughout the day, demonstrating dynamic P2P interactions. Episodes are randomly sampled from yearly data to capture seasonal variations in market prices, loads, and supply conditions. Table 1 RL Training Hyperparameters MADDPG / LSD-MADDPG Total Timesteps: 60,000 Episode Length: 24 Learning Rate Actor: 10−4 Learning Rate Critic: 10−3 Noise-rate: 0.1 Gamma: 0.95 Tau: 0.01 Buffer size: 5 × 105 Batch size: 256
PPO Total Timesteps: 60,000 Episode Length: 24 Learning Rate Actor: 10−4 Learning Rate Critic: 10−3 GAE factor: 0.1 Clip rate: 0.95 K epochs: 0.01 L2 regularization: 10−3 Batch size: 256
P. Wilk et al.: Preprint submitted to Elsevier
Value 0 – 200 $∕MWh 𝜌max = 50% 0 – 100 $∕MWh
Table 3 compares the net monetary gains or losses for each prosumer participating in the P2P market and the aggregator in Case I. The row labeled “Sum P2P” aggregates all P2P transactions into a single net value. RB Agg is used as a benchmark to evaluate market manipulation, ensuring prices remain within acceptable limits, while RB P2P serves as a reference for a midpoint singleprice market. MARL-Conv and SEAM-LESS results are comparable, both favoring sellers with market power by securing higher revenues compared to RB markets. Conversely, buyers benefit most from the RB P2P market, where a uniform P2P trading price eliminates market power disparities, resulting in the lowest buyer costs and proportional quantity splits among peers. RB Agg maximizes aggregator profit by funneling all energy through the aggregator at its bid/offer spread, elevating buyer costs and reducing seller revenues the most. Compared to RB P2P’s uniform pricing, RL-based market solutions introduce competition by enabling distinct price bids. SEAM-LESS notably enhances seller revenues while maintaining buyer costs similar to those under RB Agg, demonstrating that prosumer agents effectively exploit market power without excessive manipulation, aided by the aggregator fallback price mechanism. This is evident by the aggregator’s increased profit in SEAM-LESS over RB P2P, as mismatches in competitive pricing cause some trades to revert to the aggregator for settlement. Despite limited information sharing among prosumers, as compared to MARL-Conv, SEAM-LESS successfully learns optimal bidding strategies for prosumers. Figure. 4 illustrates that P2P settling prices often surpass the wholesale clearing price, boosting seller revenues while buyer costs align more closely with the aggregator’s sell price. As shown in Table 4, SEAM-LESS yields the best Sum P2P reward (5.91), surpassing all other methods. Notably, Buyer 1 shifts from a negative reward of -0.22 to a positive reward of 0.18, and Buyer 3 increases from -0.87 to 0.47, while Seller 2 climbs above 3.14 compared to 2.60 in MARL-Conv, reflecting increased profit and lower penalties. These penalties specifically target aggregator fallback when uncompetitive prices arise, thereby guiding P2P trades learning toward stable pricing strategies. Table 3 Case I Monetary Costs and Revenue (in $) Aggregator Buyer 1 Seller 2 Buyer 3 Seller 4 Sum P2P
RB Agg RB P2P MARL-Conv SEAM-LESS 2.19 1.22 1.25 1.27 -1.99 -1.81 -1.98 -1.97 0.24 0.50 0.74 0.69 -3.21 -2.92 -3.29 -3.22 0.24 0.50 0.75 0.70 -4.72 -3.74 -3.78 -3.79
Page 8 of 11
MARL-Based Coordinated P2P Electricity Trading
6. The onsite energy availability during these peak solar periods temporarily shifts market power and elevates P2P clearing prices in favor of sellers, but aggregator fallback quickly shapes the overall cost distribution and ultimately decreases total P2P market cost relative to the RB Aggregator approach.
Figure 4: SEAM-LESS settling price evolution of Case I. Table 4 Case I Rewards Comparison Aggregator Buyer 1 Seller 2 Buyer 3 Seller 4 Sum P2P
RB Agg RB P2P MARL-Conv SEAM-LESS 21.99 12.25 12.57 12.72 -13.11 -0.81 -0.22 0.18 -16.37 2.54 2.60 3.14 -14.61 1.23 -0.87 -0.47 -16.37 2.54 3.00 3.05 -60.45 5.50 4.50 5.91
Figure 5: SEAM-LESS settling price evolution of Case II.
Figures. 4 (for Case I) and 5 (for Case II) illustrate the settling price evolution of the aggregator and P2P market participants, collectively demonstrating how market power emerges when one side’s quantity is limited yet bounded by the aggregator’s fallback. In Case I, two buyers’ total demand exceeds two sellers’ total supply, forcing buyers to compete for scarce retail energy supply, driving up P2P bidding prices (and in turn retail P2P clearing prices) and weakening their bargaining position. By inverting participant quantities in Case II, the retail P2P market shifts to more supply than demand. Sellers must undercut each other to secure retail trades, lowering their offer prices and thus retail P2P clearing prices, favoring buyers with cheaper deals. Despite these shifts, neither side can push prices unboundedly high or low, as the aggregator’s purchase/sell prices ultimately constrain the P2P market power. In Case III, with the integration of solar PVs, prosumers 1 and 2’s net energy remains demand-heavy excepting for the peak solar hours (i.e., hours 9-13), when a limited surplus starts to push up the P2P prices in hour 9, as shown in Figure. P. Wilk et al.: Preprint submitted to Elsevier
Figure 6: Price dynamics with solar PV integration, highlighting elevated P2P clearing prices at peak hours.
Table 5 presents all prosumers’ monetary costs and the aggregator’s revenue in Case III with MARL-Conv yielding the highest P2P outcome, albeit with a lower aggregator revenue, while SEAM-LESS better balances aggregator profit with competitive P2P pricing, comparable to RB P2P. Table 6 shows the rewards of all participants, with negative rewards reflecting missed opportunities, i.e., surplus not sold during the short peak solar window, leading to unrealized P2P transactions at favorable times. When focusing on participants’ rewards as defined in Section III and not their actual profits or costs (i.e., relative to aggregator prices 𝑝𝑏𝑎 , 𝑝𝑠𝑎 ), SEAM-LESS demonstrates higher overall peer benefits. For instance, Prosumer 2, which shows a negative reward under MARL-Conv, trades more surplus in SEAM-LESS, lowering its penalty from missed transactions. Furthermore, SEAM-LESS narrows the shortfall, indicating fewer aggregator fallbacks. RB P2P and SEAM-LESS both better balance aggregator’s profit with improved P2P outcomes, whereas MARL-Conv sees a larger aggregator-P2P discrepancy, as its aggregator prices yield less aggregator profit but do not necessarily maximize P2P surplus. Results confirm that solar peaks offer brief seller leverage, allowing SEAMLESS to retain robust performance by efficiently matching generation to demand while preserving data privacy. Table 5 Case III Monetary Costs and Revenues (in $) Aggregator Prosumer 1 Prosumer 2 Prosumer 3 Prosumer 4 Sum P2P
RB Agg RB P2P MARL-Conv SEAM-LESS 0.14 0.12 0.06 0.12 -0.09 -0.08 -0.06 -0.07 -0.10 -0.10 -0.08 -0.10 -0.02 -0.02 -0.02 -0.02 -0.14 -0.13 -0.11 -0.13 -0.35 -0.33 -0.27 -0.32
Summarizing the case studies, Case I showcases a scenario of fixed retail buyers and sellers, illustrating how SEAM-LESS promotes P2P trades while balancing benefits Page 9 of 11
MARL-Based Coordinated P2P Electricity Trading Table 6 Case III Reward Comparison Aggregator Prosumer 1 Prosumer 2 Prosumer 3 Prosumer 4 Sum P2P
RB Agg RB P2P MARL-Conv SEAM-LESS 21.99 18.72 9.43 18.09 -95.45 3.16 0.49 5.82 -46.86 0.11 -0.78 -0.54 -3.56 1.00 0.81 0.92 -16.25 5.02 4.42 2.86 -162.11 9.29 4.94 9.07
Table 7 Reward Comparison of 10 Prosumers Case RB Agg RB P2P MARL-Conv SEAM-LESS Aggregator 78.92 9.75 12.91 11.41 -3.43 0.82 0.27 0.90 Prosumer 1 Prosumer 2 -5.72 1.43 0.77 0.69 Prosumer 3 -3.77 0.91 0.18 0.60 Prosumer 4 -3.40 0.74 0.39 -0.09 Prosumer 5 -3.79 0.86 -0.02 0.56 -10.31 1.43 0.91 0.72 Prosumer 6 Prosumer 7 -7.36 0.98 0.72 0.41 Prosumer 8 -5.61 0.69 0.65 0.49 Prosumer 9 -6.59 0.90 0.56 0.62 Prosumer 10 -8.44 1.14 0.86 0.63 Sum P2P -58.4199 9.9025 5.2911 5.5179
among different parties; Case II demonstrates the scenario of shifted market power from sellers to buyers as compared to Case I, and shows that SEAM-LESS prevents extreme prices while enabling moderate leverage of such market power; Case III presents the market dynamics integrating solar PV systems, enabling prosumers to autonomously switch their roles as a seller or a buyer in the P2P market and engage in effective trading within the SEAM-LESS framework.
4.2. Scalability of SEAM-LESS To further evaluate the scalability of the proposed SEAMLESS framework, a case study with 10 prosumers was conducted, with quantities varied by ±5 units during training and evaluated on a test dataset [40]. The results are summarized in Table 7, showing that SEAM-LESS effectively balances aggregator and prosumer benefits (compared with rule-based markets), performs comparably to MARL-Conv while preserving individual prosumers’ privacy and achieving higher P2P rewards. Compared to RB P2P, SEAM-LESS encourages price competition, yields lower total P2P rewards due to restricted data sharing and varied pricing strategies, and elevated aggregator reward. By preserving privacy, SEAM-LESS promotes solution scalability and adaptability, avoiding MARL-Conv’s centralized data issues. While this may sacrifice some collective efficiency, it is an ideal feature for deploying privacy-sensitive, resource-conservative, large-scale retail markets.
5. Conclusion We introduced SEAM-LESS, a scalable and privacypreserving hierarchical MARL framework for DERs’ coordinated participation in wholesale and retails markets. SEAM-LESS integrates a PPO-based aggregator’s wholesale bidding and LSD-MADDPG based strategies for prosumers’ retail P2P market clearing. SEAM-LESS ensures that individual prosumers adaptively align their retail market P. Wilk et al.: Preprint submitted to Elsevier
behaviors with their collective actions to advance wholesale market clearing goals through aggregator coordination. Unlike centralized training in conventional MADDPG, SEAM-LESS balances cooperative and competitive elements through localized decision-making, mitigating data bottlenecks, communication lags, and single points of failure. The case studies demonstrated SEAM-LESS’s robust performance across diverse market scenarios, highlighting its ability to coordinate DERs, aggregator fallback, and P2P trading in a stable and economically efficient manner. Moving forward, SEAM-LESS could be extended to incorporate an even richer portfolio of DERs and to handle more complex grid constraints into the market model design, further fostering stable, economically efficient, and environmentally sustainable energy trading ecosystems in more sophisticated system contexts.
References [1] J. Blazquez, R. Fuentes-Bracamontes, C. A. Bollino, N. Nezamuddin, The renewable energy policy paradox, Renewable and Sustainable Energy Reviews 82 (2018) 1–5. doi : https : / / doi.org / 10.1016 / j.rser.2017.09.002. [2] M. Greenstone, I. Nath, Do renewable portfolio standards deliver costeffective carbon abatement?, University of Chicago, Becker Friedman Institute for Economics Working Paper. (2019-62). URL https:// ssrn.com/abstract=3374942 [3] H. Wang, Do mandatory u.s. state renewable portfolio standards increase electricity prices?, Growth and Change 47 (2) (2016) 157– 174. doi:https://doi.org/10.1111/grow.12118. [4] Statista, Historical electricity prices in the united states from 1990 to 2023, https://www.statista.com, accessed: Jan. 2025 (2023). [5] J. D. Rhodes, The old, dirty, creaky u.s. electric grid would cost $5 trillion to replace. where should infrastructure spending go?, accessed: Jan. 2025 (Dec. 2018). URL https://energy.utexas.edu/ news/old- dirty- creaky- us- electric- grid- would- cost- 5- trillionreplace-where-should-infrastructure
[6] W. Strielkowski, L. Civín, E. Tarkhanova, M. Tvaronavičienė, Y. Petrenko, Renewable energy in the sustainable development of electrical power sector: A review, Energies 14 (24). doi:10.3390/en14248240. [7] U. J. Hahnel, M. Herberz, A. Pena-Bello, D. Parra, T. Brosch, Becoming prosumer: Revealing trading preferences and decision-making strategies in peer-to-peer energy communities, Energy Policy 137 (2020) 111098. doi:https://doi.org/10.1016/j.enpol.2019.111098. [8] T. Pinto, Z. Vale, S. Widergren, Local Electricity Markets, Elsevier, Amsterdam, The Netherlands, 2021. URL https : / / www.sciencedirect.com / book / edited - volume / 9780128200742 / local electricity-markets
[9] Y. Ye, D. Papadaskalopoulos, Q. Yuan, Y. Tang, G. Strbac, Multiagent deep reinforcement learning for coordinated energy trading and flexibility services provision in local electricity markets, IEEE Transactions on Smart Grid 14 (2) (2023) 1541–1554. doi:10.1109/ TSG.2022.3149266. [10] T. Chen, W. Su, Indirect customer-to-customer energy trading with reinforcement learning, IEEE Transactions on Smart Grid 10 (4) (2019) 4338–4348. doi:10.1109/TSG.2018.2857449. [11] G. Strbac, D. Papadaskalopoulos, N. Chrysanthopoulos, A. Estanqueiro, H. Algarvio, F. Lopes, L. de Vries, G. Morales-Espana, J. Sijm, R. Hernandez-Serna, J. Kiviluoma, N. Helisto, Decarbonization of electricity systems in europe: Market design challenges, IEEE power & energy magazine 19 (1). doi:10.1109/MPE.2020.3033397. [12] A. Ghasemi, A. Shojaeighadikolaei, K. Jones, M. Hashemi, A. G. Bardas, R. Ahmadi, A multi-agent deep reinforcement learning approach for a distributed energy marketplace in smart grids, in: 2020 IEEE International Conference on Communications, Control, and
Page 10 of 11
MARL-Based Coordinated P2P Electricity Trading Computing Technologies for Smart Grids (SmartGridComm), 2020, pp. 1–6. doi:10.1109/SmartGridComm47815.2020.9302981. [13] D. Qiu, Y. Ye, D. Papadaskalopoulos, G. Strbac, Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach, Applied Energy 292 (2021) 116940. doi:https://doi.org/10.1016/j.apenergy.2021.116940. [14] D. Papadaskalopoulos, G. Strbac, Nonlinear and randomized pricing for distributed management of flexible loads, IEEE Transactions on Smart Grid 7 (2) (2016) 1137–1146. doi:10.1109/TSG.2015.2437795. [15] G. Yang, S. Du, Q. Duan, J. Su, Deep reinforcement learningbased trading strategy for load aggregators on price-responsive demand, Computational Intelligence and Neuroscience 2022 (1) (2022) 6884956. doi:https://doi.org/10.1155/2022/6884956. [16] G. Le Ray, E. M. Larsen, P. Pinson, Evaluating price-based demand response in practice—with application to the ecogrid eu experiment, IEEE Transactions on Smart Grid 9 (3) (2018) 2304–2313. doi: 10.1109/TSG.2016.2610518. [17] M. Khojasteh, P. Faria, F. Lezama, Z. Vale, A novel adaptive robust model for scheduling distributed energy resources in local electricity and flexibility markets, Applied Energy 342 (2023) 121144. doi: https://doi.org/10.1016/j.apenergy.2023.121144. [18] U. Agwan, L. Spangher, W. Arnold, T. Srivastava, K. Poolla, C. J. Spanos, Pricing in prosumer aggregations using reinforcement learning, e-Energy ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 220–224. doi:10.1145/3447555.3464853. [19] H. Wang, J. Huang, Incentivizing energy trading for interconnected microgrids, IEEE Transactions on Smart Grid 9 (4) (2018) 2647– 2657. doi:10.1109/TSG.2016.2614988. [20] L. Jia, Q. Zhao, L. Tong, Retail pricing for stochastic demand with unknown parameters: An online machine learning approach, in: 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2013, pp. 1353–1358. doi : 10.1109 / Allerton.2013.6736684. [21] S.-J. Kim, G. B. Giannakis, An online convex optimization approach to real-time energy pricing for demand response, IEEE Transactions on Smart Grid 8 (6) (2017) 2784–2793. doi:10.1109/TSG.2016.2539948. [22] N. Liu, X. Yu, C. Wang, C. Li, L. Ma, J. Lei, Energy-sharing model with price-based demand response for microgrids of peer-to-peer prosumers, IEEE Transactions on Power Systems 32 (5) (2017) 3569– 3583. doi:10.1109/TPWRS.2017.2649558. [23] P. Horrillo-Quintero, P. García-Triviño, D. Carrasco-González, C. A. García-Vázquez, L. M. Fernández-Ramírez, Smart energy coordination in microgrid clusters using hybrid model predictive control and differential evolution optimization, Energy Conversion and Management 351 (2026) 121039. doi : https : / / doi.org / 10.1016 / j.enconman.2026.121039. [24] H. Khajeh, H. Firoozi, M. R. Hesamzadeh, H. Laaksonen, M. ShafieKhah, A local capacity market providing local and system-wide flexibility services, IEEE Access 9 (2021) 52336–52351. doi:10.1109/ ACCESS.2021.3069949. [25] A. Vicente-Pastor, J. Nieto-Martin, D. W. Bunn, A. Laur, Evaluation of flexibility markets for retailer–dso–tso coordination, IEEE Transactions on Power Systems 34 (3) (2019) 2003–2012. doi:10.1109/ TPWRS.2018.2880123. [26] Y. Zhou, J. Wu, G. Song, C. Long, Framework design and optimal bidding strategy for ancillary service provision from a peer-to-peer energy trading community, Applied Energy 278 (2020) 115671. doi: https://doi.org/10.1016/j.apenergy.2020.115671. [27] Z. Guo, P. Pinson, S. Chen, Q. Yang, Z. Yang, Chance-constrained peer-to-peer joint energy and reserve market considering renewable generation uncertainty, IEEE Transactions on Smart Grid 12 (1) (2021) 798–809. doi:10.1109/TSG.2020.3019603. [28] D. Cao, W. Hu, J. Zhao, G. Zhang, B. Zhang, Z. Liu, Z. Chen, F. Blaabjerg, Reinforcement learning and its applications in modern power and energy systems: A review, Journal of Modern Power Systems and Clean Energy 8 (6) (2020) 1029–1042. doi:10.35833/ MPCE.2020.000552.
P. Wilk et al.: Preprint submitted to Elsevier
[29] R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd Edition, MIT Press, Cambridge, MA, USA, 2018. URL http: //incompleteideas.net/book/the-book-2nd.html [30] T. Chen, W. Su, Local energy trading behavior modeling with deep reinforcement learning, IEEE Access 6 (2018) 62806–62814. doi: 10.1109/ACCESS.2018.2876652. [31] H. Hua, Y. Qin, C. Hao, J. Cao, Optimal energy management strategies for energy internet via deep reinforcement learning approach, Applied Energy 239 (2019) 598–609. doi:https://doi.org/10.1016/ j.apenergy.2019.01.145. [32] Z. Wan, H. Li, H. He, D. Prokhorov, Model-free real-time ev charging scheduling based on deep reinforcement learning, IEEE Transactions on Smart Grid 10 (5) (2019) 5246–5257. doi : 10.1109 / TSG.2018.2879572. [33] A. Anvari-Moghaddam, A. Rahimi-Kian, M. S. Mirian, J. M. Guerrero, A multi-agent based energy management solution for integrated buildings and microgrid system, Applied Energy 203 (2017) 41–56. doi:https://doi.org/10.1016/j.apenergy.2017.06.007. [34] S. Brandi, M. S. Piscitelli, M. Martellacci, A. Capozzoli, Deep reinforcement learning to optimise indoor temperature control and heating energy consumption in buildings, Energy and Buildings 224 (2020) 110225. doi:https://doi.org/10.1016/j.enbuild.2020.110225. [35] J.-G. Kim, B. Lee, Automatic p2p energy trading model based on reinforcement learning using long short-term delayed reward, Energies 13 (20). doi:10.3390/en13205359. [36] P. Hernandez-Leal, M. Kaisers, T. Baarslag, E. M. de Cote, A survey of learning in multiagent environments: Dealing with non-stationarity (2019). URL https://arxiv.org/abs/1707.09183 [37] J. Vazquez-Canteli, T. Detjeen, G. Henze, J. Kämpf, Z. Nagy, Multiagent reinforcement learning for adaptive demand response in smart cities, Journal of Physics: Conference Series 1343 (1) (2019) 012058. doi:10.1088/1742-6596/1343/1/012058. [38] R. Lu, Y.-C. Li, Y. Li, J. Jiang, Y. Ding, Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management, Applied Energy 276 (2020) 115473. doi:https://doi.org/10.1016/j.apenergy.2020.115473. [39] P. Wilk, N. Wang, J. Li, Multi-agent reinforcement learning for smart community energy management, Energies 17 (20). doi : 10.3390 / en17205211. [40] PJM Interconnection, PJM Data Miner - Settlements Verified Hourly LMPs, https : / / dataminer2.pjm.com / feed / rt _ da _ monthly _ lmps, accessed: Jan. 19, 2025 (2025).
Page 11 of 11