Relational Multi-Agent Reinforcement Learning for Dynamic Pricing in High-Speed Railway Markets Enrique Adrian Villarrubia-Martina , David Muñoz-Valerob,∗, Luis Rodriguez-Beniteza , Giovanni Montanac , Luis Jimenez-Linaresa
arXiv:2607.05179v1 [cs.LG] 6 Jul 2026
a
Department of Technologies and Information Systems, Universidad de Castilla-La Mancha, Paseo de la Universidad 4, Ciudad Real, 13071, Spain b Department of Technologies and Information Systems, Universidad de Castilla-La Mancha, Avenida Carlos III, s/n, Toledo, 45071, Spain c Warwick Manufacturing Group, University of Warwick, Gibbet Hill Road, Coventry, CV4 7AL, UK
Abstract In liberalised railway systems, operators must set prices dynamically in an environment with partial observability, as they retain private information about their objectives and performance, where regulatory constraints prohibit communication or direct information exchange between competitors to prevent explicit collusion. Consequently, agents must learn to infer strategic interactions only from observable market data which presents a significant challenge for multi-agent reinforcement learning, where standard approaches typically treat observations as unstructured vectors, ignoring the underlying market topology that governs strategic interactions. To address this, an entity graph modelling approach is proposed, which represents the environment as a graph of operational units, rather than decision-making agents or static infrastructure, encoding competition, coordination, and connectivity relations between entities. Then, an extension of the multi-agent twin delayed deep deterministic policy gradient algorithm with graph-based representation learning processes the features of the entities through a multi-layer relational graph convolutional network and aggregates them via a learnt attention mechanism. ∗
Corresponding author Email addresses: [email protected] (Enrique Adrian Villarrubia-Martin), [email protected] (David Muñoz-Valero), [email protected] (Luis Rodriguez-Benitez), [email protected] (Giovanni Montana), [email protected] (Luis Jimenez-Linares)
Experimental results in a rail pricing reinforcement learning environment show that this novel framework achieves higher revenue and stability in two different settings of increasing market complexity compared to a representative selection of relational and non-relational baselines. The code is publicly available at: https://github.com/Kinrre/RelationalRailPricing-RL. Keywords: Dynamic Pricing, Multi-Agent Reinforcement Learning, Deep Reinforcement Learning, Graph Neural Networks, Railway Systems 1. Introduction The Multi-Agent Reinforcement Learning (MARL) community has increasingly adopted graph-based approaches to model complex environments [1, 2]. Graphs are a common and very powerful practice to naturally and efficiently capture the structural relations that are fundamental to multi-agent interaction and reasoning [3]. The environments are characterised by a highly dynamic nature, where agents are constantly moving and their neighbourhood relations evolve rapidly, which creates a need for architectures capable of capturing abstract relational representations. Graph Neural Networks (GNNs) are framed as the ideal solution, precisely because they operate natively on graph structures, allowing the learning process to automatically adapt to the changing topology of the environment [4]. As agents modify their connections, GNNs maintain the ability to model interactions through message passing operations, which transforms relational variability from a limitation into an advantage, without the need to adapt the architecture of the model. Focusing on railway systems, the field of application selected for experimentation, graph-based representations have emerged as the standard approach for modelling complex transportation networks. Station-based graphs, where vertices encode physical stations and edges denote direct rail connections, remain the predominant paradigm in recent railway optimisation literature [5]. This approach is widely used for network topology analysis, vulnerability assessment, and capacity planning [6]. However, these models, where stations are encoded as nodes, are typically not designed to capture operational aspects, such as service differentiation, multi-operator dynamics, and market-level interactions in liberalised high-speed passenger markets [7]. To bridge this gap, this work introduces an entity graph-based 2
modelling approach, where operational entities constitute the vertices, and competitive, coordination, and connectivity relations are encoded as heterogeneous edges. With these relations encoded in the graph structure, GNN layers propagate messages across the network while attention mechanisms can selectively aggregate the most relevant features for each agent. This produces learnable relational representations that can be used to support strategic decision-making in multi-agent environments. Based on the entity graph-based modelling framework [8, 9], the proposed model learns a relational state representation for each agent through a three-stage process. First, it individually encodes the characteristics of all entities in the environment. Then, it enriches them by allowing entities to exchange contextual information with their neighbours through the network of relations. Finally, it applies an attention mechanism that allows each agent to selectively combine the most relevant information from all entities, thus generating a compact and personalised state of the environment. These relational representations are subsequently integrated into an actor-critic architecture augmented with graphs that operates under the Centralised Training with Decentralised Execution (CTDE) [10, 11] paradigm. This architecture allows agents to access global information during training, when graph-based embeddings are constructed and refined, while maintaining decentralised policies that preserve privacy at runtime. The proposed method extends the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) [12] algorithm with graph representation learning capabilities, capturing the relational structure of the environment and addressing the challenge of multi-agent strategic interaction under partial observability with information asymmetry, which is typical of competitive market environments such as the liberalised passenger rail transport market. 1.1. Contributions The contributions of this work can be summarised as follows: • An entity graph modelling approach for MARL that encodes the environment as a graph of operational units rather than decision-making agents or static infrastructure. This approach models competition, coordination, and connectivity relations as heterogeneous edges between entities, providing a relational inductive bias [13] to capture finegrained strategic dependencies in partially observable environments. 3
• An extension of the MATD3 algorithm with graph-based representation learning. The proposed method specifically incorporates a relational state representation module that processes heterogeneous entity features through a multi-layer Relational Graph Convolutional Network (R-GCN) and aggregates them via a learnt attention mechanism to learn policies that dynamically prioritise strategically relevant services. In addition, it incorporates a gradient-stopping operation to decouple representation learning from policy optimisation, which, although it can come at the cost of a reduced total revenue, stabilises the training dynamics of each agent individually. • An experimental study of the framework in a railway pricing reinforcement learning environment [14] in two different scenarios, with increased revenue and stability in both settings. The performance of the method is compared against a representative selection of relational and non-relational MARL baselines. Several ablation studies have been conducted to measure the relative contribution of each edge type, the impact of the network depth on agent performance, the importance of the learnt attention mechanism, and the effect of decoupling the representation learning from the actor. Furthermore, the learnt graph embeddings are also explored. The rest of this work is structured as follows: First, Section 2 reviews related work. Then, Section 3 introduces the formal framework of partially observable Markov Game (MG). Then, Section 4 presents the entity-graph relational actor-critic framework. After that, Section 5 evaluates the effectiveness of the framework, and finally, Section 6 outlines the conclusions and future research directions. 2. Related work First, Section 2.1 reviews MARL in the context of dynamic pricing. Secondly, Section 2.2 examines how graph-based representations can capture the relational structure inherent in multi-agent environments. Lastly, Section 2.3 discusses the application of Deep Reinforcement Learning (DRL) to railway systems.
4
2.1. Multi-agent reinforcement learning for dynamic pricing MARL extends the single-agent reinforcement learning framework to scenarios in which multiple agents interact in a shared environment to achieve individual or collective goals [15]. In contrast to single-agent settings, MARL environments are inherently non-stationary, as each agent’s policy evolves during training, thus altering the environment for other agents. This interaction is formally modelled through game theory, and settings involving both competitive and cooperative objectives are referred to as coopetition [16]. A particularly influential paradigm for addressing non-stationarity that maintains scalable, decentralised execution is CTDE [10, 11]. Under CTDE, agents share global information during training to stabilise learning, while retaining independent policies at execution time. Within this paradigm, MADDPG [11] adapts the deterministic policy gradient approach for continuous action spaces in mixed cooperative-competitive environments, Multi-Agent Proximal Policy Optimisation (MAPPO) [17] extends proximal policy optimisation to cooperative settings, and Multi-Actor Attention Critic (MAAC) [18] incorporates attention mechanisms to dynamically weight the relevance of other agents when computing centralised Q-values. Dynamic pricing, also known as pricing intelligence, involves adjusting prices in real time based on supply and demand to maximise profits and align with market conditions. Within this context, for electric vehicle charging, [19] proposed a MARL mechanism design framework to simultaneously determine optimal charging prices across multiple stations, modelling station–user interaction as a mechanism design problem and station–station cooperation as an MG solved via MADDPG. Similarly, for hydrogen fuel cell vehicle refuelling, [20] employed a Multi-Agent System (MAS) to coordinate refuelling schedules and determine prices, improving demand satisfaction and traffic flow in microgrids. Shifting to other domains, in telecommunications, [21] framed the interaction between mobile virtual network operators and users as a Stackelberg game, solving it using a Multi-Agent Deep Q-Network (MADQN). Moreover, in smart grids, [22] proposed a distributed multi-agent optimisation approach for resolving supply-demand imbalances while strengthening privacy and autonomy. Despite these advancements, dynamic pricing in high-speed railways introduces complex relational dependencies, such as cooperative dynamics with multi-operator services and overlapping market competition. Furthermore, the partial observability and the prohibition of communication between agents in this domain demand that agents infer strategic interactions only from ob5
servable market data. This motivates the use of graph-based representations, where interacting entities and their relationships are naturally encoded as nodes and edges in a graph structure. 2.2. Graph neural networks in multi-agent reinforcement learning Multi-agent environments frequently involve relational information, where agents interact with each other through dependencies such as competition or coordination. However, traditional deep MARL approaches encode a state as a flat vector, a representation that discards the structural relations between agents, forcing neural networks to implicitly discover patterns that could be encoded explicitly. GNNs can be used as an alternative that preserves this relational structure [23]. By modelling the environment as a graph, where vertices denote decision-making entities and edges encode their relationships, agents are able to reason explicitly about their interdependencies. For instance, [1] introduced graph convolutional reinforcement learning, which constructs a graph where each agent corresponds to a vertex and edges connect agents that influence each other’s outcomes. Rather than using fixed rules to determine these connections, the method employs multi-head attention mechanisms as relation kernels, learning how agents interact with their neighbours, with temporal relation regularisation further encouraging cooperative behaviours over time. Similarly, [24] developed G2ANet, which introduces a two-stage attention network combining hard attention, to identify and eliminate irrelevant agent connections, and soft attention, to weigh the importance of remaining interactions. Other approaches have explored richer graph structures that incorporate additional context from the environment. In [9], they proposed InforMARL, which constructs an agent-entity graph connecting agents not only to other agents but also to surrounding entities such as obstacles and goals. At each time step, the framework builds a graph linking each agent to nearby entities within its local neighbourhood, employing a GNN with multi-head attention to aggregate this information into fixed-size representations for actor-critic networks. This local aggregation is more sample-efficient than concatenating global information, as it allows agents to focus on locally relevant context while maintaining a scalable representation. Additionally, [2] developed a GNN-based MARL method for resilient distributed coordination of multirobot systems operating under challenging real-world conditions. The approach employs MAPPO using a Multilayer Perceptron (MLP) critic for training and a GNN actor that incorporates both node and edge features 6
through a modified GraphSAGE [25] architecture with multi-layer message passing. A key innovation is a neighbour scoring mechanism which evaluates and selects the graph edges that agents should traverse for discrete wayfinding. Several successful applications can be found across multiple domains. In traffic signal control, [26] introduced a graph state representation, and in a similar way, [27] proposed the HG-M2I algorithm, which combines a hierarchical graph representation learning module with a multi-agent mutual information framework. In a related line of work on traffic management, [5] applied DRL with graphs for the extension planning of metro networks, improving the resilience of urban critical infrastructures. Beyond transportation, [28] proposed Multi-Relational Graph Reinforcement Learning (MRGRL) for dynamic flexible job-shop scheduling, explicitly modelling multi-type dependencies and competition patterns through heterogeneous graphs, demonstrating strong generalisation under unstable manufacturing conditions. Although these graph-based approaches are effective for reasoning about relational structure, they predominantly model agents themselves as graph nodes or static physical infrastructure, which limits their ability to capture the fine-grained operational dependencies that drive strategic interactions, as is the case of dynamic pricing in liberalised railway systems. 2.3. Deep reinforcement learning for railway systems The application of DRL to railway systems has grown considerably in recent years, with a potential for optimising operations and improving efficiency. In the context of traffic management, DRL has been applied to real-time trajectory generation for trains to ensure punctuality and energy efficiency [29], as well as to Train Timetable Rescheduling (TTR) and the Vehicle Rescheduling Problem (VRSP), where agents must restore normal operations rapidly following disruptions [30, 31]. Further applications include predictive maintenance scheduling [32, 33], autonomous train control [34, 35], and interval control under safety constraints using constrained Markov decision processes [36]. Simulation environments such as Flatland-RL [31] have played an important role in supporting the safe development and benchmarking of these approaches [37]. However, existing DRL applications in railways are predominantly concerned with operational problems rather than market-driven strategies. Dynamic pricing introduces additional challenges where agents must model the 7
distinct passenger behaviours, account for interdependencies arising from multi-operator itineraries, as well as the inherent tension between competition and cooperation in a partially observable market. To address this gap, [14] introduced RailPricing-RL, a parameterisable MARL environment designed specifically for dynamic pricing in high-speed railway markets where operators must balance competition and cooperation with connecting services, incorporating microscopic passenger decision-making via Random Utility Models (RUM). The current study extends this approach by introducing a novel relational state representation that explicitly captures these structural dependencies. 3. Preliminaries This section introduces a partially observable MG [38], a stochastic game that satisfies the Markov property, with mixed incentives in which agents have both aligned and competing objectives. In an MG, a multi-agent extension of an MDP, multiple agents act as independent decision-makers, each with their own set of actions and reward functions, as shown in Figure 1.
obs
Agent 2 reward
obs
action
Agent 1 reward
action
Environment Figure 1: MARL interaction in a partially observable MG.
A key property is that both state transitions and individual rewards depend on the joint actions of all agents, creating a complex interdependence of strategies and outcomes in which no single agent fully controls the environment dynamics. Formally, it is defined by the tuple: 8
M = ⟨S, U, P, r, Z, O, A, γ⟩ where S is the set of possible states, with s ∈ S denoting the current state of the environment. The set A = {1, . . . , n} represents the n agents, where each agent a ∈ A selects actions ua ∈ U at each time step. These actions collectively form a joint action u ∈ U ≡ U n , which induces a transition to a new state s′ according to the state transition function P (s′ | s, u) : S × U × S → [0, 1]. In contrast to Decentralised Partially Observable Markov Decision Processes (DPOMDP) [39], typically used in fully cooperative settings where all agents share a common reward function, stochastic games allow competitive, cooperative, or mixed forms of interaction, the latter often referred to as coopetition [16]. Consequently, each agent is assigned an individual reward function, defined as ra (s, u) : S × U → R, which depends on the global state and the joint action. Partial observability is intrinsic to realistic multiagent settings, contrasting with perfect information games, where the full state is observable. This is the case in railway market environments, where operators retain private information about their objectives, behaviour, performance, etc. Formally, each agent receives an observation za ∈ Z a , which provides partial information about the global state. It is determined by the agent-specific observation function O(s, a) : S × A → Z a , where Z a denotes the observation space for agent a. Furthermore, agents operate under strict information asymmetry without explicit communication, where each agent a can observe only its own reward signal ra and local observation za , whereas the rewards, policies, and value functions of other agents remain hidden. While traditional approaches treat the observations as unstructured vectors, real-world MAS often exhibit a rich relational structure with entities connected by specific relations that influence strategic interactions. To capture the structural relations inherent in multi-agent environments, the state is augmented with a graph Gt = (V, E, X), where V denotes the set of vertices, E ⊆ V × V are the edges encoding relations between vertices, and X = {xv }v∈V comprises node features. The graph evolves with the environment state at each time step t, yielding Gt = G(st ), and provides a structured representation that enables reasoning about relational dependencies beyond flat feature vectors. This is a separate augmentation of the state rather than an additional element of the tuple M . Finally, the goal of each agent a is to learn a stochastic policy π a (ua | za ) : Z a × U → [0, 1], which maps observa9
tions to a probability distribution over actions. The collection of policies for all agents, π = (π 1 , . . . , π n ), forms the joint policy. Thus, the objective of each agent is to maximise its expected discounted return, defined as: # "∞ X J(π a ) = Eu∼π,s∼P γ t rta (s, ut ) , t=0
where γ ∈ [0, 1] is the discount factor that determines the relative importance of immediate versus long-term rewards. 4. Methods This section proposes an entity graph-based relational actor-critic framework for MARL in environments with rich relational structure. First, Section 4.1 introduces the entity-based graph modelling approach, then Section 4.2 details the semantic embedding architecture, and finally Section 4.3 presents the graph-augmented actor-critic architecture that integrates these representations. 4.1. Entity-based graph modelling The first goal is to encode the environment as a heterogeneous graph to provide a relational inductive bias [13, 40, 41] overcoming the limitations of unstructured state representations common in MARL [17]. Typically, nodes in these graphs are the agents themselves [1, 24] or the static physical infrastructure [5, 26]. As a novel contribution, the entity graph, instead of focusing on decision-making agents such as railway companies, focuses on the operational nodes they control, referred to as entities. This paradigm shift moves beyond modelling decision-makers or passive infrastructure, towards encoding the strategic units of control themselves. The approach treats entities as the primary vertices in the graph, modelling the different relations between operational units within the MAS. It captures how strategic decisions affect and are affected by the broader ecosystem of interactions, rather than forcing neural networks to implicitly discover these patterns from flat feature vectors. These entities in the railway domain take the form of specific train services, which constitute the resources and functional units on which agents operate. A service is defined as a scheduled journey between stations at a specific time with allocated capacity. The value of this entity approach lies 10
in the explicit modelling of the relational dynamics that arise between these entities. Thus, competitive dependencies are captured, where services from different agents compete for the same market; coordination relations, that forms the agent’s internal service portfolio; and connectivity dependencies, where one entity requires another to operate, so that passengers can transfer between services to form multi-operator routes. As a consequence, the framework can reason about how pricing decisions propagate through the service network. To formalise this, at each time step t, the railway market is modelled as a directed graph Gt = (V, E, X) where vertices correspond to individual railway services and edges encode these market relations. Here, each vertex v ∈ V corresponds to a unique service operated within the railway network by an agent agent(v) ∈ A. A service is a scheduled, multi-stop journey along a line of consecutive stations and serves a set of origin–destination markets W (v), where each market (wo , wd ) is specified by its origin station wo and destination station wd , defining an origin-destination pair where passengers seek transportation. The edge set is formed through three specific relation types described as follows, where vi , vj are any two distinct vertices in V : • Ecomp = {(vi , vj ) : W (vi )∩W (vj ) ̸= ∅, date(vi ) = date(vj ), agent(vi ) ̸= agent(vj )} denotes competition edges connecting services scheduled on the same date that operate in at least one common market w but are managed by different agents. • Ecoord = {(vi , vj ) : date(vi ) = date(vj ), agent(vi ) = agent(vj )} represents coordination edges that link services operated by the same agent on the same date. • Econn = {(vi , vj ) : Wd (vi ) ∩ Wo (vj ) ̸= ∅, date(vi ) = date(vj )} defines connectivity edges as potential passenger connections between services scheduled on the same date, where Wo (v) = {wo : (wo , wd ) ∈ W (v)} and Wd (v) = {wd : (wo , wd ) ∈ W (v)} are the sets of origin and destination stations of the markets served by v. Competition and coordination edges are symmetric, so the reverse edge (vj , vi ) is also added, whereas the connectivity relation is naturally directed from the arriving service to the departing service. Thus, the edge set E is: E = Ecomp ∪ Ecoord ∪ Econn . 11
Figure 2 illustrates a concrete example of this entity graph, visualising the dense network of relations for a market comprising 15 services operated by three competing agents, where each node corresponds to a service, labelled by its controlling agent. Notably, the coordination edges densely interconnect within each agent’s portfolio, competition edges link services in overlapping markets, and connectivity edges weave between services to form potential multi-operator routes.
2
1
Competition Coordination Connectivity
1 1
2
1
2 1 2 3
2 3
3 3
3
Figure 2: Entity graph representation for a railway network with 15 services scheduled on the same date by three agents. Red edges indicate competition, services in the same market by different operators, orange edges denote coordination, services operated by the same agent, and green edges represent connectivity, potential transfers between services.
4.2. Relational state representation learning This framework learns a relational state representation for each agent through a three-stage process (see Figure 3): heterogeneous entity feature encoding, which processes categorical and continuous entity attributes through separate pathways (Section 4.2.1), relational context propagation, which performs message passing to create context-aware entity embeddings (Section 4.2.2), and agent-level state aggregation, which produces a fixed-size state vector for each agent, computed from the set of entity embeddings (Section 4.2.3).
12
2 3
Layer Norm Final layer
Weighted Sum
R-GCN Attention pooling
3
+
Softmax Multi-Layer GNN
MLP (Score function)
Dropout
2
ReLU
1
Intermediate layer
Layer Norm
Linear
1
R-GCN 1 Embedding
Multi-Layer Perceptron
1
Figure 3: Architecture of the relational state representation module. For each agent, the entity features are first encoded, with categorical attributes through embeddings and continuous ones through an MLP, and then concatenated and projected by a linear transformation. A multi-layer R-GCN refines them by using message passing over the entity graph. Finally, a learnt attention pooling aggregates them into a fixed-size state vector.
4.2.1. Encoding of heterogeneous entity features The initial stage transforms the raw, heterogeneous features of each entity node into a unified dense representation, which serves as the input to an individual GNN. For each agent a, this transformation is achieved through parallel pathways designed to handle the distinct properties of categorical and continuous data before fusing them into a common latent space. Each entity v is characterised by a heterogeneous feature vector comprising categorical attributes, such as origin and destination stations, seat class, operator, line, corridor, time slot and rolling stock type, and continuous market indicators used in dynamic railway pricing models. Each categorical attribute type k is transformed through a learnable embedding function: Eka : N → Rde which maps discrete category indices to dense embedding vectors of dimension de . This enables learning continuous semantic representations of 13
the respective categorical attributes, which are subsequently concatenated to form a unified categorical feature vector: hacat (v) = ∥ Eka (vk ) k∈K
where K is the set of all categorical attributes for entity v, and ∥ denotes the concatenation operation. The dynamic market data associated with service v, comprising prices ps,c and sales volume nw,c for each service-seat combination and the total service profit rw,c are flattened into a single vector xacont (v). To enforce partial observability, agent a observes complete information for its own services, but for competitor services the private ticket-sales information, that is, sales volume per service-seat and total service profit, is masked with zeros. Although zero sales is a valid observation, it cannot be confused with a masked one, as masking is applied only to competitor services which can be identified by the operator feature. This vector is then processed by an MLP to capture complex, nonlinear interactions and dependencies within the market signals, such as price elasticity effects: hacont (v) = MLPa (xacont (v)) Finally, the categorical and continuous vectors are concatenated and projected into the target hidden dimension dh by a linear transformation: a ha,(0) = Wproj [hacat (v) ∥ hacont (v)] + baproj . v a,(0)
This projection yields the initial node embedding hv ∈ Rdh for each entity, a unified embedding that captures the intrinsic properties independently of the graph structure, without incorporating any relational information from neighbouring entities. These entity-level representations then serve as input for the relational context propagation stage, where message passing over the graph enriches them with information from the entity’s neighbourhood. 4.2.2. Relational context propagation through a multi-layer GNN a,(0) The initial entity embeddings hv capture intrinsic properties but lack awareness of the broader market structure. To incorporate this relational context, a multi-layer GNN that performs message passing across the heterogeneous entity graph, is used. Specifically, a stack of R-GCN [42] layers is employed to iteratively aggregate information from neighbouring entities. 14
a,(ℓ)
At each layer ℓ ∈ {1, 2, . . . , L}, the hidden representation hv of entity v is updated through message passing over the graph structure Gt : X X (ℓ) = f ha,(ℓ) v r∈R u∈Nr (v)
1 (ℓ) + b(ℓ) , v ∈ V W (ℓ) ha,(ℓ−1) + W0 ha,(ℓ−1) v |Nr (v)| r u
where Nr (v) denotes the neighbourhood of node v under relation r connected through competition, coordination, or connectivity edges. Each re(ℓ) lation weight matrix Wr transforms the aggregated neighbour representa(ℓ) tions, whereas W0 corresponds to a self-loop that introduces a special relation type to each node, to ensure that the node at layer ℓ is also informed by the corresponding representation at the previous layer ℓ − 1. In addition, b(ℓ) is a learnable bias vector and the activation function f (ℓ) introduces nonlinearity. The normalisation constant |Nr (v)| implements a mean aggregation, ensuring that nodes with many connections do not dominate the aggregation. To stabilise training and facilitate gradient flow through deep architectures, each intermediate layer ℓ < L incorporates several architectural mechanisms. More specifically, residual connections [43] are added to mitigate the vanishing gradient problem through additive information flow across layers. In addition, layer normalisation [44] is employed to stabilise the distribution of representations as they evolve over the network during training. Furthermore, dropout regularisation prevents overfitting by randomly deactivating neurons during training. In contrast, the final layer L differs from intermediate layers, applying only layer normalisation without an activation function or residual connection. a,(L) This produces the final context-aware entity embedding hv ∈ Rdemb , where demb is the embedding dimension. These learnt embeddings encode both intrinsic entity properties, derived from the initial feature encoding, and relational context, acquired through iterative message passing across the a,(L) graph structure. Consequently, each embedding hv captures the strategic position of entity v within the market ecosystem, reflecting competitive pressures from rival services, portfolio synergies with services under the same agent’s control, and network effects arising from connectivity relations. 4.2.3. Attention-based agent-level state aggregation a,(L) Each agent GNN yields a set of context-aware embeddings {hv }v∈V for all entities in the environment. However, for decision-making, each agent 15
requires a single, fixed-size state vector that aggregates information from the set of entities relevant to its decision-making. Since the number of entities can vary, a simple concatenation or flattening of embeddings is not feasible. While fixed pooling operations, such as mean or max aggregation, can produce a fixed-size vector, they treat all entities as equally important, regardless of their strategic relevance. To overcome this, an attention mechanism is introduced that learns to dynamically weigh the importance of each entity in the environment. This allows the agent to focus on the most strategically relevant entities when constructing its state representation. For instance, the attention weights can learn to prioritise services in highly competitive markets during price wars, while directing focus towards capacity-constrained services when demand increases, or emphasising services that form multi-operator journeys when coordination opportunities arise. For each agent a, attention weights are computed by applying a learnt scoring function to each entity embedding, followed by softmax normalisation: a,(L) exp score(hv ) a , v ∈ V, (1) αv = P a,(L) exp score(h ) u u∈V where score(·) is a learnable function that maps entity embeddings to scalar importance scores. In particular, a two-layer MLP with a ReLU activation function is used for the score function. The agent-level observation z̃a is then computed as a weighted sum: X z̃a = αva ha,(L) , (2) v v∈V
yielding z̃ ∈ R , a permutation-invariant representation [45] that dynamically weights entities according to learnt strategic relevance. Moreover, the learnt attention weights are useful diagnostics, offering insight into which entities the agent considers most relevant to make a decision at each time step. a
demb
4.3. Relational actor-critic with heterogeneous graph embeddings The attention mechanism produces a context-aware state vector z̃a for each agent. Next, to enable strategic decision-making with these representations in mixed competitive-cooperative multi-agent settings under partial observability and information asymmetry, the framework employs the CTDE 16
paradigm [10, 11]. This framework gives agents access to global information during training while maintaining decentralised, privacy-preserving policies at execution time, making it well-suited for competitive market environments where operators must act independently based on local observations. Based on the MATD3 algorithm [12], the proposed method extends the standard actor-critic architecture with graph-based representation learning to capture relational environmental structure. Each agent a maintains two distinct components, a decentralised actor πθa that maps local aggregated observations z̃a to actions, and two centralised critics Qaϕ1 , Qaϕ2 that estimate action-value functions using global state information, with the minimum taken to mitigate overestimation bias. Additionally, each agent a maintains its own independent GNN (GNNψa ) that processes the entity graph Gt , with agent-specific masked features, to a,(L) produce entity embeddings {hv }v∈V . This design reflects the information asymmetry inherent in competitive markets where each agent observes only public information about competitors’ entities while maintaining complete visibility of its own operations. In particular, the actor network πθa processes the agent’s aggregated observation z̃a derived from attention-weighted pooling of entity embeddings to produce a continuous action ua . The actor maximises expected returns through policy gradients, whereas the critic minimises temporal difference error. Therefore, using both to update the GNN representations can lead to conflicting gradient signals. To address this, when computing actions for environment interaction and actor updates, the aggregated observation z̃a , along with the representation module, is detached from the computational graph. Thus, the GNN is trained exclusively through the critic loss, which provides richer signals from the global value function, decoupling policy learning from representation learning. The centralised critics Qaϕ1 (Z̃, u), Qaϕ2 (Z̃, u) take as input the joint aggregated observations Z̃ = (z̃1 , . . . , z̃n ) and joint actions u = (u1 , . . . , un ) of all agents. The critics operate on the same abstracted state space as the actors, taking as input the joint aggregated observation Z̃ to learn a global value function conditioned on the agents’ summarised local views. During critic updates, the GNN parameters ψ a are optimised only with respect to the current agent, although the critics receive the joint embedding Z̃j , which also contains the embeddings produced by the GNNs of other agents. The GNN parameters ψ a are jointly optimised with the critics parameters ϕa1 , ϕa2 to minimise the temporal difference error as the expected value over samples 17
drawn from the replay buffer D: L(ϕa1 , ϕa2 , ψ a ) = Ej∼D
" 2 X
yja − Qaϕi (Z̃j , uj )
2
# ,
(3)
i=1
where yja is the target value computed using target networks as follows: yja = rja + (1 − dj ) γ min Qaϕ′i (Z̃j+1 , πθ1′ (z̃1j+1 ) + ϵ, . . . , πθn′ (z̃nj+1 ) + ϵ), i=1,2
(4)
and ϵ ∼ clip(N (0, σ), −c, c) introduces a small amount of random noise to the target policy as a regularisation term, clipped to keep the target close to the original action. The term dj indicates a terminal transition, for transitions where the episode terminates, the target yja is equal to the immediate reward rja . The next-state embeddings are produced by the online representation module, as only target networks are used for the actors and critics, which already provide a stable objective in the learning process. Then, the actor is updated using the deterministic policy gradient: h i (5) ∇θa J(π a ) = Ej∼D ∇θa π a (z̃aj )∇ua Qϕa1 (Z̃j , uj )|ua =πa (z̃aj ) with gradient detachment on the aggregated observations to preserve the separation between policy and representation learning. In addition, the actor target network is updated with a reduced frequency d compared to the critic, which has been shown to reduce overestimation bias and improve sample efficiency in continuous control tasks [46]. Finally, to further stabilise learning, target networks are updated using soft updates with interpolation parameter τ . The complete training procedure of the framework called RACHE is detailed in Algorithm 1. 5. Experimental settings and results This section evaluates the effectiveness of the proposal in learning strategic pricing policies in two different scenarios. First, Section 5.1 describes the simulation environment, and Section 5.2 the experimental scenarios. Subsequently, Section 5.3 presents the algorithms selected for evaluation. Then, Section 5.4 details the implementation settings. Later, Section 5.5 analyses the comparative performance of the evaluated algorithms and finally, Section 5.6 examines key architectural components of RACHE. 18
Algorithm 1 Relational Actor-Critic with Heterogeneous Graph Embeddings (RACHE) 1: Initialise networks {πθa , Qϕa1 , Qϕa2 , GNNψa }n a=1 with random parameters a a a 2: Initialise target networks θ ′ ← θ a , ϕ′ 1 ← ϕa1 , ϕ′ 2 ← ϕa2 3: Initialise replay buffer D 4: for episode = 1, . . . , M do 5: for t = 1, . . . , T do 6: for each agent a = 1, . . . , n do a,(L) 7: Compute service embeddings: {hv }v∈V = GNNψa (Gt ) ▷ With
masked features Compute aggregated observation z̃at via attention using Eq. 1-2 ▷ Stop gradient Select action: uat = πθa (z̃at ) + ϵ, ϵ ∼ N (0, σ) end for Execute joint action ut , observe rewards {rta }na=1 and next observation Zt+1 12: Store transition (Zt , ut , {rta }na=1 , Zt+1 , dt ) in D ▷ Raw entity features 13: if |D| ≥ batch size then 14: for each agent a = 1, . . . , n do 15: Sample minibatch (Zj , uj , {rja }na=1 , Zj+1 , dt ) from D 16: Compute aggregated observation z̃aj+1 for all agents ▷ See Lines 7-8 17: Compute target as shown in Eq. 4 18: Update critic and GNN using Eq. 3 19: Update actor using the sampled policy gradient as in Eq. 5 ▷ Stop gradient 20: end for 21: Update critic target network: ϕ′ ai ← τ ϕai + (1 − τ )ϕ′ ai , i ∈ {1, 2} 22: if t mod d = 0 then 23: Update actor target network: θ′ a ← τ θa + (1 − τ )θ′ a 24: end if 25: end if 26: end for 27: end for 8: 9: 10: 11:
5.1. Simulation environment All experiments are conducted using the simulator RailPricing-RL [14], a parameterisable reinforcement learning environment designed to model highspeed railway networks featuring dynamic pricing, multi-operator itineraries, and diverse passenger behaviour. The environment employs microscopic 19
modelling of passenger decision-making using RUM, where passengers can choose not to travel if no itinerary meets their utility threshold, resulting in a non-zero sum game. It supports MARL algorithms through compatibility with standard RL interfaces. In this environment, each service is scheduled for a specific date. However, the simulation operates over a booking horizon that begins when the first passenger attempts to buy a ticket. This horizon is divided into discrete time steps, each representing a single day. During these steps, agents set prices, and passengers, each arriving on a specific day sampled from their segment’s booking distribution, make purchasing decisions. Now, the observation space, action space, and reward function are described. Observation space. Each agent observes all services in the network. For each service the observation includes static attributes which are the train service provider, the corridor (a set of stations within a region), the line (a sequence of stations in the corridor), the time slot, and the rolling stock, encoded as indices, as well as dynamic pricing data containing the prices per service-seat combination, the ticket sales and the cumulative service revenue. Nevertheless, the ticket sales and the cumulative service revenue are private, so agent a observes them only for its own services, while the corresponding features of competitor services are masked with zeros to preserve partial observability. Action space. The action space defines the set of decisions available to agents for modifying ticket prices in their respective services. Each agent c selects a continuous action αv,t ∈ [−1, 1] per service-seat combination, representing a normalised percentage price adjustment applied to the current price: β c c c , pv,t+1 = pv,t · 1 + αv,t · 100 where β = 25 is the default scaling factor and prices are clipped to [0, ∞) to prevent negative values. Reward. At each time step t, agent a receives a reward equal to the incremental daily revenue across its operated services: X ρvt − ρvt−1 , rta = v∈V a
20
where ρvt denotes the cumulative revenue of service v up to time step t and V a ⊆ V is the set of services operated by agent a. This formulation assumes no associated costs for providing services, so the reward captures revenue changes only. Thus, the objective is to maximise its expected total revenue over the course of an episode. 5.2. Scenarios The experiments evaluate the performance of the algorithms in two scenarios of increasing market complexity. These scenarios employ distinct supply configurations and a common demand pattern to assess algorithm behaviour under varying levels of market structure. The supply configuration of the first scenario features 18 services scheduled on the same date, operated by three agents, each one with six services, illustrated in Figure 4a. The second scenario expands it to 63 services, consisting of a daily schedule of 21 services repeated on three consecutive days. In addition, it introduces an additional agent as shown in Figure 4b. In both scenarios, agents compete directly in overlapping markets and can cooperate through multi-operator connecting itineraries. Agents are heterogeneous, controlling different sets of services with distinct action spaces. The specific characteristics of each scenario are described below. Competition Coordination Connectivity
1 1
1
2
1
2
1
3 2
1 4
3
2
4
3
3 3
1
3
(a) Base.
1
2 4
Competition Coordination Connectivity
2
1
4
1
2
2
4 4
2 2 1 2 2 2 1 1 1 1 3 3 3 3 3 4
2 2 1 2 2 2 1 1 1 1 3 3 3 3 3
4 1
1
2 4
2 2 1 2 2 2 1 1 1 1 3 3 3 3 3
(b) Large.
Figure 4: Entity graph representations for the experimental scenarios. Each node corresponds to a train service labelled by its operating agent, and edges encode competition, coordination, and connectivity relations.
21
Base. This scenario introduces a varied passenger market across five stations and seven origin-destination markets, operated by three agents, each controlling six services on a single day with a total of 18 services. Four passenger segments named business, commuter, student and leisure coexist with distinct utility functions and temporal preferences. On the one hand, business travellers are a user group characterised by inelastic demand, with strong preferences for convenient morning arrival and departure times, and commuters exhibit high penalties for travel time and transfers. On the other hand, students show high price sensitivity with flexible timing preferences and book tickets close to the departure, whereas leisure passengers tend to book earlier and over a wider window with greater diversity in the arrival time. Episodes last seven days and generate an average of 980 potential passengers across all markets. Moreover, all services offer two seat types, basic and premium, with a capacity of 32 and eight seats per service respectively, forming a joint action space of 36 dimensions. Finite seat availability introduces capacity management constraints requiring agents to adapt their pricing strategies in response to competition and inventory levels. Large. This scenario expands market complexity by introducing a fourth competing agent, resulting in a total of 63 services, 21 per day over three consecutive days, across six origin-destination markets with an increased number of passengers. The four agents are asymmetric in portfolio size, with the largest controlling a total of 21 services, and the smallest operating only nine. The same four passenger segments coexist with the same utility functions and temporal preferences described above. Episodes span ten days and generate an average of approximately 1,420 potential passengers across all markets. As in the first scenario, all services offer basic and premium seat types, with a capacity of 13 and four seats per service respectively, and a joint action space of 126 dimensions, to account for capacity management constraints. Finally, Table 1 reports the statistics of the entity graph for each scenario. In both, the coordination relation is the densest one, as it connects every service scheduled on the same date provided by the same agent. However, the other relations in the Large scenario, competition and connectivity, are more represented with the addition of a fourth agent.
22
Table 1: Statistics of the entity graph for each scenario. Competition and coordination edges are symmetric and counted in both directions, whereas connectivity edges are directed.
Statistic Agents Nodes (services) Competition edges Coordination edges Connectivity edges Total edges
Base
Large
3 18 30 (21%) 90 (61%) 27 (18%) 147
4 63 162 (26%) 294 (47%) 168 (27%) 624
5.3. Benchmark algorithms and baselines To evaluate the approach and its contribution of entity-based graph modelling, RACHE is compared against a representative selection of MARL algorithms covering deterministic and stochastic policies, with and without relational structure. The evaluated algorithms are described below: • Random: A non-learning baseline that selects pricing actions uniformly at random from the action space, establishing a lower bound on performance. • Static pricing: A revenue management strategy in which fixed pricing is used to assess whether dynamic pricing policies are valuable. • MADDPG [11]: A deterministic actor-critic algorithm extending DDPG [47] to multi-agent settings via CTDE, included as a baseline for deterministic policy methods in mixed cooperative-competitive environments. • MATD3 [12]: A deterministic actor-critic that extends MADDPG with twin critics, delayed policy updates, and target policy smoothing to mitigate overestimation bias. This is the algorithm the framework is based on. • MAAC [18]: A stochastic multi-agent extension of SAC [48] incorporating attention to dynamically weight the relevance of other agents when computing centralised Q-values to assess agent-level attention without explicit graph structure. 23
• GA-AC [24]: A graph-based stochastic actor-critic in which decisionmaking agents are graph nodes and interactions are modelled by the G2ANet two-stage attention mechanism, combining hard gating with soft scaled dot-product attention. It contrasts the agent-based graph paradigm with the entity-graph approach of RACHE. These algorithms are evaluated in the scenarios described previously to assess whether explicit relational reasoning provides benefits compared to flat observation encoders, whether modelling operational entities in an entity graph is preferable to modelling decision-making agents as graph nodes, and how deterministic policy methods compare with stochastic alternatives in this setting. All baseline implementations, except for the MAAC algorithm for which the original repository has been used, are publicly available at: https://github.com/Kinrre/RelationalRailPricing-RL. 5.4. Implementation details For a standardised comparison, the default hyperparameters of each algorithm were adopted without additional tuning. A complete list of the hyperparameters is provided in Appendix A, and all experiments were conducted on an Intel i7-13700KF CPU and an NVIDIA GeForce RTX 4080 16GB GPU. To encourage initial exploration and diversify the state-action space, each algorithm was first trained with a random policy for 1,000 episodes. The models were trained for one million steps with a replay buffer of the same size. Furthermore, policy updates were performed every two Q-network updates in the case of the MATD3 and RACHE algorithms. Finally, it is worth noting that the RACHE framework employs a two-layer R-GCN for relational context propagation across the entity graph. In the experiments, to facilitate stable training, normalisation was applied to continuous features of the observation space, such as prices, ticket sales and cumulative service revenue. In particular, it was performed after masking the private data of each agent, which can still be identified through the operator feature. The normalised observation feature ẑt at time step t was computed as follows: zt − µt , ẑt = p σt2 + ϵ where zt is the raw observation at time t, µt is the running mean of each individual continuous feature up to time t, σt2 is the running variance of each 24
individual feature up to time t, and ϵ is a small constant used for numerical stability. For the reward signal, representing the daily revenue, reward scaling was employed instead of standard normalisation. Rewards were divided by a constant factor of 1,000 in both scenarios to account for high revenue magnitudes. This approach preserves the direct relationship between revenue and the reward signal, maintaining positive signs and relative magnitudes while preventing numerical instability from large gradient updates. As a result, the value function receives a more consistent learning signal with a clearer distinction between more profitable and less profitable actions. 5.5. Performance comparisons and analysis This section analyses the performance of the evaluated algorithms in both scenarios. Table 2 summarises the total revenue obtained during the evaluation at the end of training using the last checkpoint of the models, and Figure 5 illustrates the training dynamics across all scenarios. Table 2: Total revenue obtained in the evaluation using the last checkpoint for the different scenarios. The mean and the stratified bootstrap confidence intervals at 95% were computed over three independent runs.
Algorithm
Base
Large
Random 30961 [30832, 31092] Static pricing 31309 [31284, 31335] MADDPG 32688 [31641, 33675] MATD3 37028 [36312, 37739] MAAC 31254 [30445, 32093] GA-AC 34446 [34285, 34596] RACHE (Ours) 41036 [40596, 41509] The highest values are indicated in bold.
40031 [39937, 40115] 40950 [40908, 40992] 43902 [43427, 44402] 48206 [46930, 49425] 41220 [40755, 41675] 40479 [40388, 40566] 53831 [53113, 54604]
First, with respect to the non-learning baselines, static pricing, that keeps a fixed pricing by not modifying the initial price point for each service specified by the scenarios, performs better than the random baseline, indicating that modifying prices randomly from the action space is a worse strategy. Among the learning-based algorithms, the RACHE framework achieves the highest revenue in both scenarios, followed by MATD3, which presents a substantially higher variance. This supports that the relational state representation learnt over the entity graph provides a meaningful inductive bias. 25
Average Total Revenue (95% CIs)
As expected, regarding the deterministic baselines, MATD3 surpasses MADDPG, since the combination of twin critics, delayed policy updates, and target policy smoothing helps to achieve a higher revenue. In the case of the stochastic alternatives, MAAC fails to learn an effective pricing policy, performing comparably to or even worse than the random baseline during training in the first scenario. Finally, regarding GA-AC, the agent-based graph baseline learns a meaningful policy that surpasses MADDPG in the Base scenario, with a very low variance. However, in the large one, its performance fails to improve over the course of training. This is due to the fact that, by abstracting each operator as a single node, GAAC cannot capture the fine-grained service-level relations. The entity graph addresses precisely this limitation by treating services rather than operators as the primary vertices, whose granularity scales with the market structure. Base
44000 42000
55000
40000
52500
38000
50000
36000
47500
34000
45000
32000
42500
30000
40000
28000
0.0
0.2
0.4
Large
57500
0.6
Time steps
0.8 MADDPG GA-AC
1.0 1e6
0.0
MATD3 Random
RACHE Static
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
MAAC
Figure 5: Average total revenue obtained at training for the two scenarios.
To account for which agent the total revenue comes from, Figure 6 shows the distribution of the total revenue of each individual agent in both scenarios. Foremost, it can be observed in the Base scenario that the total revenue is more evenly distributed among the agents, whereas in the Large scenario, it is imbalanced because of the asymmetric portfolio sizes. In both cases, the RACHE framework increases the revenue of every agent with respect to almost all baselines, except, for instance, the first agent of the MATD3 algorithm in the former. More precisely, the increase of total revenue in the first scenario can be attributed mainly to the third agent, although, in the second, the improvement is more uniformly distributed across all agents. In addition, as suggested by [49], Figure 7 presents a pairwise matrix 26
Average Revenue (95% CIs)
Base 17500 15000 12500 10000 7500 5000 2500 0
Large 17500 15000 12500 10000 7500 5000 2500 0
c dom Stati DDPG ATD3 MAAC GA-AC ACHE M R MA
Ran
Algorithm
Agent 1
c dom Stati DDPG ATD3 MAAC GA-AC ACHE M R MA
Ran Agent 2
Algorithm
Agent 3
Agent 4
Figure 6: Average revenue obtained at evaluation using the last checkpoint for each agent.
with the probability of improvement, that is, the probability that algorithm X obtains a higher total reward than algorithm Y . This is calculated from the Mann–Whitney U statistic collected across the three seeds for a scenario m given N and K evaluation episodes respectively, defined as: 1, if y < x, 1 S(xm,i , ym,j ) where S(x, y) = 21 , if y = x, P (Xm > Ym ) = N K i=1 j=1 0, if y > x. N X K X
The results indicate that the RACHE framework obtains a higher total reward than the rest of the considered baselines. Base
Large 1.0
0.33 0.00 0.65 0.00 0.00
0.8
Static 0.83 MADDPG 0.67 0.67
0.06 0.70 0.29 0.00
MATD3 1.00 1.00 0.94
1.00 0.94 0.00
MAAC 0.48 0.35 0.30 0.00
0.03 0.00
GA-AC 1.00 1.00 0.71 0.06 0.97
0.00
RACHE 1.00 1.00 1.00 1.00 1.00 1.00
dom Stati
Ran
Random
0.6
0.00 0.00 0.00 0.22 0.10 0.00
1.0
0.00 0.00 0.33 0.97 0.00
0.8
Static 1.00 MADDPG 1.00 1.00
0.4
0.10 0.98 1.00 0.00
MATD3 1.00 1.00 0.90
1.00 1.00 0.02
MAAC 0.78 0.67 0.02 0.00
0.67 0.00
0.2
GA-AC 0.90 0.03 0.00 0.00 0.33
0.0
RACHE 1.00 1.00 1.00 0.98 1.00 1.00
c E PG D3 AC AC DD AT MA GA- RACH MA M
0.00
P(row > col)
0.17 0.33 0.00 0.52 0.00 0.00
P(row > col)
Random
0.6 0.4 0.2 0.0
c 3 dom tati DPG TD AAC A-AC RACHE Ran S MAD MA M G
Figure 7: Probability of improvement matrix using the Mann–Whitney U statistic.
Next, the effect of the learnt policies on the final prices at the end of the 27
episode, relative to the initial fares, is analysed together with its impact on the percentage of passengers travelling. Figure 8 shows the distribution of the final price change per seat type, and Table 3 details the corresponding percentages. As shown, the deterministic algorithms increase prices more than the stochastic ones. In fact, in the Base scenario, GA-AC obtains a higher revenue than MADDPG with smaller price increments, as well as RACHE compared to MATD3 in both scenarios, indicating that larger price increases do not necessarily translate into higher revenue. In contrast, the deterministic algorithms reduce the percentage of passengers travelling more than the stochastic ones, as a consequence of their larger price increases, but still RACHE obtains a higher revenue while retaining a larger number of passengers travelling compared to MATD3. Base
Large
Final Price Change (%)
800 300
600
200
400
100
200
0
0
100
PG
DD
MA
TD3 MA
AC
MA
Algorithm
AC
GA-
HE
PG
DD
RAC
MA
Basic
TD3 MA
AC
MA
Algorithm
AC
GA-
HE
RAC
Premium
Figure 8: Distribution of the final price change per seat type relative to the initial fare.
Lastly, Table 4 reports the number of parameters for each model and the total training time including all the seeds of each algorithm in hours. Notably, the scalability of the proposed framework can be attributed to two points: first, the processing of a compact fixed-size embedding of each agent, which decreases the input size of the centralised critics, and second, to the message passing paradigm whose parameter weights are shared across all nodes and, therefore, independent of the number of services in the graph. 5.6. Further studies This section examines key architectural components in more depth. First, Section 5.6.1 analyses the importance of each relation type in the entity
28
Table 3: Percentage of passengers travelling in the evaluation using the last checkpoint for the different scenarios. The mean and the stratified bootstrap confidence intervals at 95% were computed over three independent runs.
Algorithm
Base
Random 70.15 [69.96, 70.34] Static pricing 70.67 [70.50, 70.84] MADDPG 63.16 [62.91, 63.42] MATD3 61.09 [59.94, 62.18] MAAC 69.53 [69.23, 69.82] GA-AC 69.48 [69.28, 69.68] RACHE (Ours) 63.46 [62.03, 64.81] The highest values are indicated in bold.
Large 63.07 [62.99, 63.16] 63.22 [63.12, 63.31] 52.77 [52.27, 53.29] 52.08 [51.35, 52.78] 62.15 [61.88, 62.45] 63.10 [62.99, 63.20] 53.55 [53.04, 54.04]
Table 4: Parameter count of each algorithm and total wall training time using three independent runs for the different scenarios.
Base Parameter Count
Large Wall time
MADDPG 1, 318, 695 6.2 hours MATD3 2, 208, 810 6.2 hours MAAC 1, 759, 000 7.0 hours GA-AC 4, 287, 825 6.3 hours RACHE (Ours) 1, 887, 729 5.8 hours The lowest values are indicated in bold.
Parameter Count
Wall time
5, 851, 778 10, 374, 790 5, 152, 212 10, 094, 856 2, 983, 466
9.0 hours 9.1 hours 10.0 hours 9.1 hours 7.4 hours
graph. Subsequently, Section 5.6.2 examines how the depth of message passing affects agent performance. Then, Section 5.6.3 evaluates the attentionbased pooling contribution. Next, Section 5.6.4 investigates the effect of decoupling representation learning from the actor in the training dynamics. Finally, Section 5.6.5 analyses the information encoded by the learnt graph embeddings. 5.6.1. What is the contribution of each relation type in the entity graph? An evaluation of the importance of each relation was carried out, in which each relation type was removed from the entity graph in order to assess its relative contribution. To this end, three variants of the framework are compared versus the default configuration, in which all three relation types 29
are present. Figure 9 shows the average total revenue during training when each relation type is removed in both scenarios. The default configuration with all relations results in the highest total revenue, confirming that the three relations contribute to the agent’s decision-making. More specifically, removing the competition relation produces the largest drop in performance, followed by the connectivity relation, which is expected, since they capture direct competition and potential cooperation between agents. In contrast, removing the coordination relation, inspired by agent-based graphs, which only encodes whether two services are operated by the same agent, leads to the smallest performance gap. This indicates that the strategic value of connecting two services derives from their market relationship rather than shared ownership. Average Total Revenue (95% CIs)
Base
Large
57500
42500 40000 37500 35000 32500 30000 27500 25000
55000 52500 50000 47500 45000 42500 40000 0.0
0.2
0.4
0.6
Time steps Default
0.8
1.0 1e6
No Competition
0.0 No Coordination
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
No Connectivity
Figure 9: Ablation study comparing average total revenue with each relation type removed.
5.6.2. How does message passing depth affect agent performance? To evaluate the impact of network depth on agent performance, an ablation study was conducted comparing variants with one, two, and three R-GCN layers. The depth of the GNN determines how information is aggregated for each service: a single-layer GNN aggregates information only from adjacent services, whereas a two-layer or three-layer GNN expands it to two-hop or three-hop neighbourhoods, respectively. Figure 10 illustrates the average total revenue during training for each depth setting. The two-layer configuration achieves the best performance in both scenarios, followed by the one-layer and three-layer variants. This 30
Average Total Revenue (95% CIs)
can be attributed to the fact that it captures both, direct and connecting services with one transfer, in contrast to the one-layer variant, which only captures direct services, as price-sensitive passengers are willing to accept longer journeys or transfers if the total fare is lower. In addition, the threelayer variant exhibits the worst performance, which can suggest a smoothing effect on the features of vertices, making them harder to distinguish and less meaningful for decision-making [50]. Base
44000 42000 40000 38000 36000 34000 32000 30000 0.0
0.2
0.4
Large
57500 55000 52500 50000 47500 45000 42500 40000 37500 0.6
Time steps
0.8 One Layer
1.0 1e6 Two Layers
0.0
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
Three Layers
Figure 10: Ablation study comparing average total revenue for one, two, and three layers.
5.6.3. How does attention-based pooling compare to uniform aggregation? The attention mechanism serves as a dynamic weighting function that aggregates service-level embeddings into agent-level observations. To assess the contribution of the learnt attention mechanism to agent performance, the framework with learnt attention weights was compared against a variant employing uniform attention weights. In the uniform attention variant, all services receive an equal weight during the aggregation process, resulting in the computation of a mean service embedding, rather than a weighted combination. Figure 11 presents the average total revenue during training for both approaches. The results show that the uniform attention variant leads to a slower learning process and, hence, to a worse sample efficiency compared to the learnt attention method. Moreover, an interesting finding is that, once enough experience has been collected, both variants converge to a similar asymptotic performance. As a result, the multi-layer R-GCN compensates for the lack of dynamic weighting by encoding later the relevant relational context directly into the entity embeddings. 31
Average Total Revenue (95% CIs)
Base
44000 42000
55000
40000
52500
38000
50000
36000
47500
34000
45000 42500
32000 30000
Large
57500
40000 0.0
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
Learnt Attention
0.0
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
Uniform Attention
Figure 11: Ablation study comparing average total revenue of learnt attention weights against uniform attention weights.
In addition, a visual representation of the cross-attention is shown in Figure 12, where the coordinate (i, j) represents the normalised mean attention weight that agent i assigns to services operated by agent j. To analyse the attention pattern, during the period in which the learnt mechanism provides the advantage of the greater sample efficiency, the weights are aggregated at training step 200K. Notably, the attention does not follow a uniform distribution, as there are weights concentrated along the main diagonal. This behaviour is consistent with the partial observability of the environment, since agents are only able to fully observe the services they control, so their own entities provide a richer and more informative signal. Nevertheless, attention is still assigned to competitor services, indicating that the visible public attributes of rival entities are also informative. 5.6.4. How does decoupling representation learning from the actor affect the training dynamics? The framework detaches the aggregated observation from the computational graph when computing the actor update, so that the GNN parameters are optimised only through the critics loss. The motivation behind this design choice is that, since they optimise different objectives, using both can lead to a more unstable training of the agents. To evaluate its impact, the default configuration is compared with a variant in which the representation module is jointly updated by the critic and the actor losses. Figure 13 shows the average total revenue during training for both variants. In particular, in the first scenario, the detached configuration results in 32
Base
A2
A3
0.675
0.168
0.180
0.157
0.770
0.156
1.0
A1
0.578
0.138
0.129
A2
0.065
0.764
0.094
0.077
0.6
A3
0.055
0.049
0.855
0.041
0.4
A4
0.040
0.043
0.047
0.870
A1
A2 A3 Target Agent
A4
0.8
0.050
0.204
0.120
0.676
A1
A2 Target Agent
A3
Attention
Source Agent
A1
Large
0.2 0.0
Figure 12: Normalised mean cross-attention weights between agents at training step 200K.
Average Total Revenue (95% CIs)
a policy with better performance and less variability across runs compared to the other approach, whereas in the second scenario, it is the no detach variant that achieves the higher revenue at the end of training. However, this figure only reflects the revenue obtained by the agents in total. Base
44000 42000
55000
40000
52500
38000
50000
36000
47500
34000
45000
32000
42500
30000 28000
Large
57500
40000 0.0
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
0.0
Detach
No Detach
0.2
0.4
0.6
Time steps
0.8
1.0 1e6
Figure 13: Ablation study comparing average total revenue of detach and no detach variants.
To quantify the stability difference at agent level, for each run, a stability metric is calculated as the standard deviation of the consecutive differences of the individual revenue (Table 5). Intuitively, this metric is a discrete analogue 33
of the variability of the learning curve derivative, so smaller values indicate that revenue increments from one step to the next are more consistent. More specifically, the detached variant achieves the lowest value for every agent in both scenarios, with this difference being more pronounced in the second case. This implies that decoupling representation learning from the actor stabilises the training dynamics of each agent individually, although there is a trade-off with a reduced total revenue. Table 5: Training stability per agent measured as the standard deviation of consecutive differences of the individual revenue. The mean and the standard deviation were computed over three independent runs.
Base Detach
Large No Detach
Agent 1 455.7 ± 70.7 498.9 ± 144.0 Agent 2 363.9 ± 215.5 441.2 ± 13.6 Agent 3 521.7 ± 39.2 537.1 ± 100.9 Agent 4 The lowest values are indicated in bold.
Detach
No Detach
666.4 ± 114.0 622.4 ± 61.8 450.7 ± 50.6 356.2 ± 37.7
846.1 ± 195.3 943.7 ± 17.3 649.3 ± 61.2 535.9 ± 106.1
5.6.5. What do the learnt graph embeddings capture about agent strategies? To understand the information encoded by the learnt embeddings, t-SNE [51] visualisation was applied to the attention-weighted observations of the trained agents. Specifically, these visualisations project the aggregated observation vectors z̃a that serve as input to each agent’s actor and critic networks. For each scenario, the three experimental runs with different random seeds during the evaluation at the end of training are presented, with each point representing an agent’s aggregated observation at a single time step. Each observation point is coloured in two ways: first, by its controlling agent, to reveal structural patterns in how different agents perceive the market, and second, by its associated revenue, to uncover strategic value distributions. Figure 14 illustrates the projections of the aggregated observations for both scenarios. On the one hand, the agent-coloured projections show that the aggregated observations of different agents are mapped to distinct, contiguous regions of the latent space, with little overlap between them. This can be attributed to the fact that each agent learns its own representation, as each of them has its own GNN. Moreover, the partial observability of 34
the environment that hides private ticket-sales data of competitor services encourages the same service to be encoded differently in each agent’s view of the graph. On the other hand, the reward-coloured projections present a similar pattern with high and low revenue points clustered together, rather than being uniformly distributed. Therefore, this suggests that the learnt embeddings appear to encode who is acting and how profitable the market state is. 6. Conclusions and future work This work introduces an entity relational actor-critic framework for partially observable MARL with mixed competitive-cooperative settings. It encodes the environment as graph of operational units with heterogeneous edges, rather than decision-making agents or static infrastructure, connected by competition, coordination, and connectivity relations. Each agent learns a relational state representation through a multi-layer R-GCN, which is then aggregated into an agent-level observation via a learnt attention mechanism, and trains its policy under the CTDE paradigm by extending MATD3. The framework has been evaluated in a railway pricing reinforcement learning environment in two different scenarios of increasing market complexity, where it obtained a higher total revenue compared to the other MARL baselines considered. Moreover, it achieved this with smaller price increments and a higher percentage of passengers travelling compared to MATD3. One potential benefit of the approach is the scalability with respect to the market structure, due to the compact fixed-size agent embeddings and the message passing paradigm that keep the model size independent of the number of services. This contrasts with the agent-based graph baseline GA-AC, in which each operator is encoded into a single node and whose effectiveness collapses in the larger scenario. According to the further studies, the competition and connectivity relations contribute the most to performance, while the coordination relation, which is the closest to the agent-based modelling, has the smallest impact. In addition, regarding the depth of the message passing, the two-layer configuration achieves the highest revenue, since it captures direct and connecting services with one transfer. Furthermore, the attention mechanism improves sample efficiency in the early stages of training, and detaching the representation module from the actor gradients has been shown to stabilise the individual agent training dynamics, even though it can reduce the total revenue. 35
150 t-SNE Dimension 2
100 50 0 50 100
50 0 50 100 150
200
200 200
100 0 100 t-SNE Dimension 1
200
150 100 50 0 50 100 200
100 0 100 t-SNE Dimension 1
200
200
Colored by Reward (Reward/Ep: 40160.62)
150
100
100
100
50 100
t-SNE Dimension 2
200
150 t-SNE Dimension 2
200
150
0
50 0 50 100
150 200 100 0 100 t-SNE Dimension 1
4000
50
200 200
6000
100
150 200
8000
0
200 100 0 100 t-SNE Dimension 1
200
50
150 200
100 0 100 t-SNE Dimension 1
Colored by Reward (Reward/Ep: 42350.51)
200
50
Agent 1 Agent 2 Agent 3
200
150 200
Colored by Reward (Reward/Ep: 40511.32)
t-SNE Dimension 2
100
150
Colored by Agent (Reward/Ep: 42350.51)
Agent 1 Agent 2 Agent 3
200
t-SNE Dimension 2
150 t-SNE Dimension 2
Colored by Agent (Reward/Ep: 40160.62)
Agent 1 Agent 2 Agent 3
Reward (Profit)
Colored by Agent (Reward/Ep: 40511.32) 200
200
2000 200
100 0 100 t-SNE Dimension 1
200
0
(a) Base. Agent 1 Agent 2 Agent 3 Agent 4
150
150 t-SNE Dimension 2
100 50 0 50 100
50 0 50 100 150
200
200 200
100 0 100 t-SNE Dimension 1
200
150 100 50 0 50 100 200
100 0 100 t-SNE Dimension 1
200
200
Colored by Reward (Reward/Ep: 53274.74)
150
100
100
100
0 50 100 150
t-SNE Dimension 2
200
150 t-SNE Dimension 2
200
150
50 0 50 100 150
200 100 0 100 t-SNE Dimension 1
200
200
8000 7000 6000
50
5000
0
4000
50
3000
100
2000
150
200 200
100 0 100 t-SNE Dimension 1
Colored by Reward (Reward/Ep: 52100.12)
200
50
Agent 1 Agent 2 Agent 3 Agent 4
200
150 200
Colored by Reward (Reward/Ep: 56263.26)
t-SNE Dimension 2
100
150
Colored by Agent (Reward/Ep: 52100.12)
Agent 1 Agent 2 Agent 3 Agent 4
200
t-SNE Dimension 2
200
t-SNE Dimension 2
Colored by Agent (Reward/Ep: 53274.74)
Reward (Profit)
Colored by Agent (Reward/Ep: 56263.26)
1000
200 200
100 0 100 t-SNE Dimension 1
200
200
100 0 100 t-SNE Dimension 1
200
0
(b) Large.
Figure 14: t-SNE visualisation of learnt agent observations coloured by agent and reward. The clustering patterns suggest that the embeddings capture aspects of agent identity and market profitability.
36
As future work, it is suggested to explore methods where agents could directly discover which entities are connected to each other, rather than relying on predefined edge types. For instance, this could be achieved by using attention mechanisms that dynamically learn a relational structure based on observed state transitions and rewards. The framework would also benefit from modelling the temporal aspect within the graph structure itself. Currently, the entity graph is reconstructed at each time step based on static rules, but many real-world systems present relations that evolve over time. To this end, temporal graph networks [52] could be incorporated into the model to capture relational changes as the environment evolves. Finally, only off-policy methods are considered in this work, so future directions could extend the framework to on-policy algorithms in order to assess their potential advantages in this setting. CRediT authorship contribution statement Enrique Adrian Villarrubia-Martin: Writing – original draft, Visualisation, Validation, Software, Methodology, Data curation, Conceptualisation. David Muñoz-Valero: Writing – review & editing, Data curation, Software. Luis Rodriguez-Benitez: Writing – original draft, Resources, Methodology, Formal analysis. Giovanni Montana: Writing – review & editing, Supervision, Methodology, Conceptualisation, Formal analysis. Luis Jimenez-Linares: Supervision, Project administration, Methodology, Conceptualisation, Funding acquisition. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability Data will be made available upon request. Acknowledgments This work was supported by grant PID2020-112967GB-C32 funded by MCIN/AEI/10.13039/501100011033 and by the ERDF A Way of Making 37
Europe. It was completed when Enrique Adrian Villarrubia-Martin was a predoctoral fellow at the Universidad de Castilla-La Mancha funded by the European Social Fund Plus (ESF+) and in a visiting research stay at the University of Warwick funded by a mobility grant from the Universidad de Castilla-La Mancha for predoctoral students. Giovanni Montana acknowledges support from a UKRI AI Turing Acceleration Fellowship (EPSRC EP/V024868/1). Appendix A. Hyperparameters This section provides a comprehensive overview of the hyperparameters used to train the algorithms. For a standardised comparison, the default hyperparameters of each algorithm were used without additional tuning. Shared hyperparameters are presented in Table A.1 and are used unless an algorithmspecific table assigns a different value. Algorithm-specific configurations are detailed in: Table A.2 for RACHE, Table A.3 for MADDPG, Table A.4 for MATD3, Table A.5 for MAAC, and Table A.6 for GA-AC.
38
Table A.1: Shared hyperparameters between algorithms.
Hyperparameter
Value
Number of environments Optimiser Buffer size Training steps Random policy episodes Evaluation episodes Number of hidden layers Hidden units per layer Non-linear activation function Batch size Discount factor (γ) Policy learning rate Critic learning rate Target smoothing coefficient (τ ) Steps per update Seeds Training environment seed Evaluation environment seed Observation normalisation Reward scaling
16 Adam [53] 1,000,000 1,000,000 1,000 10,000 2 256 ReLU 1024 0.99 0.001 0.001 0.005 100 0, 41, 73 {seed + r · 1000 | r ∈ {0, 1, . . . , 15}} {seed + r · 100000 | r ∈ {0, 1, . . . , 15}} Yes 1000
Table A.2: RACHE hyperparameters.
Hyperparameter
Value
Categorical embedding dim Continuous embedding dim R-GCN learning rate R-GCN hidden dim Number of R-GCN layers Dropout Attention score function
4 128 0.001 128 2 0.1 MLP
39
Table A.3: MADDPG hyperparameters.
Hyperparameter Discount factor (γ) Policy learning rate Critic learning rate Exploration noise
Value 0.95 0.01 0.01 Ornstein-Uhlenbeck [54] θ = 0.15 σ = 0.2
Table A.4: MATD3 hyperparameters.
Hyperparameter
Value
Exploration noise Policy noise Policy noise clip Frequency delayed policy updates
N (0, 0.1) N (0, 0.2) 0.5 2
Table A.5: MAAC hyperparameters.
Hyperparameter
Value
Non-linear activation function Number of attention heads Number of updates per update cycle
LeakyReLU 4 4
Table A.6: GA-AC hyperparameters.
Hyperparameter
Value
Entropy regularisation coefficient (α)
0.2
References [1] J. Jiang, C. Dun, T. Huang, Z. Lu, Graph Convolutional Reinforcement Learning, in: 8th International Conference on Learning Representations, ICLR 2020, 2020. [2] A. Goeckner, Y. Sui, N. Martinet, X. Li, Q. Zhu, Graph neural networkbased multi-agent reinforcement learning for resilient distributed coordination of multi-robot systems, in: 2024 IEEE/RSJ International 40
Conference on Intelligent Robots and Systems (IROS), IEEE, 2024, pp. 5732–5739. [3] S. Utke, J. Houssineau, G. Montana, Investigating relational state abstraction in collaborative marl, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, 2025, pp. 20947–20955. [4] E. Ardjmand, E. Izadi, A. Tavasoli, B. Moradi-Jamei, H. Shakeri, Graph-embedded reinforcement learning for dynamic pricing and advertising under network effects, Applied Soft Computing 186 (2026). doi:10.1016/j.asoc.2025.114056. [5] Q. Liu, L. Zhang, M. J. Skibniewski, Network extension planning towards resilient urban critical infrastructures using deep reinforcement learning, Applied Soft Computing 176 (2025). doi:10.1016/j.asoc.2025.113163. [6] T. Li, Y. Qin, M. Xu, Y. Zhou, L. Rong, Spatio-temporal vulnerability of high-speed rail line network in china, Transportation Research Part D: Transport and Environment 134 (2024). doi:10.1016/j.trd.2024.104338. [7] C. Gutiérrez-Hita, O. de la Cruz, R. Ramos-Melero, Infrastructure access charges, service differentiation, and strategic competition in the eu railway passenger market., Transportation Research Part B: Methodological 164 (2022). doi:10.1016/j.trb.2022.08.004. [8] A. Agarwal, S. Kumar, K. Sycara, M. Lewis, Learning transferable cooperative behavior in multi-agent team, in: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, Vol. 2020-May, 2020. doi:10.65109/edcb3795. [9] S. Nayak, K. Choi, W. Ding, S. Dolan, K. Gopalakrishnan, H. Balakrishnan, Scalable multi-agent reinforcement learning through intelligent information aggregation, in: International conference on machine learning, PMLR, 2023, pp. 25817–25833. [10] J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, S. Whiteson, Counterfactual multi-agent policy gradients, in: 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, 2018. doi:10.1609/aaai.v32i1.11794.
41
[11] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems, Vol. 2017-December, 2017. [12] J. Ackermann, V. Gabler, T. Osa, M. Sugiyama, Reducing overestimation bias in multi-agent domains using double centralized critics, arXiv preprint arXiv:1910.01465 (2019). [13] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, M. Shanahan, V. Langston, R. Pascanu, M. Botvinick, O. Vinyals, P. Battaglia, Deep reinforcement learning with relational inductive biases, in: 7th International Conference on Learning Representations, ICLR 2019, 2019. [14] E. A. Villarrubia-Martin, L. Rodriguez-Benitez, D. Muñoz-Valero, G. Montana, L. Jimenez-Linares, Dynamic Pricing in High-Speed Railways Using Multi-Agent Reinforcement Learning, arXiv preprint arXiv:2501.08234 (2025). [15] S. V. Albrecht, F. Christianos, L. Schäfer, Multi-Agent Reinforcement Learning: Foundations and Modern Approaches, MIT Press, 2024. URL https://www.marl-book.com [16] D. Pang, J. Zhu, H. Meng, J. Cao, S. Liu, Data-driven optimal bipartite containment control for heterogeneous multi-agent systems over coopetition networks, Journal of the Franklin Institute 362 (16) (2025) 108057. doi:10.1016/J.JFRANKLIN.2025.108057. [17] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, Y. Wu, The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, in: Advances in Neural Information Processing Systems, Vol. 35, 2022. [18] S. Iqbal, F. Sha, Actor-attention-critic for multi-agent reinforcement learning, in: 36th International Conference on Machine Learning, ICML 2019, Vol. 2019-June, 2019, pp. 5261–5270. [19] L. Hou, Y. Li, J. Yan, C. Wang, L. Wang, B. Wang, Multi-agent reinforcement mechanism design for dynamic pricing-based demand response in charging network, International Journal of Electrical Power and Energy Systems 147 (2023). doi:10.1016/j.ijepes.2022.108843. 42
[20] A. Fraija, N. Henao, K. Agbossou, S. Kelouwani, M. Fournier, S. Nagarsheth, Deep reinforcement learning based dynamic pricing for demand response considering market and supply constraints, Smart Energy 14 (2024). doi:10.1016/j.segy.2024.100139. [21] G. Sun, G. Boateng, L. Luo, H. Chen, D. Mensah, G. Liu, Competitive Pricing for Resource Trading in Sliced Mobile Networks: A Multi-Agent Reinforcement Learning Approach, IEEE Transactions on Mobile Computing (2023). doi:10.1109/TMC.2023.3281203. [22] H. Ma, H. Zhang, D. Tian, D. Yue, G. P. Hancke, Optimal demand response based dynamic pricing strategy via Multi-Agent Federated Twin Delayed Deep Deterministic policy gradient algorithm, Engineering Applications of Artificial Intelligence 133 (2024). doi:10.1016/j.engappai.2024.108012. [23] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, 2017. [24] Y. Liu, W. Wang, Y. Hu, J. Hao, X. Chen, Y. Gao, MultiAgent game abstraction via graph attention neural network, in: AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, 2020. doi:10.1609/aaai.v34i05.6211. [25] W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, in: Advances in neural information processing systems, Vol. 30, 2017. [26] J. Yoon, K. Ahn, J. Park, H. Yeo, Transferable traffic signal control: Reinforcement learning with graph centric state representation, Transportation Research Part C: Emerging Technologies 130 (2021). doi:10.1016/j.trc.2021.103321. [27] S. Yang, Hierarchical graph multi-agent reinforcement learning for traffic signal control, Information Sciences 634 (2023) 55–72. [28] Z. Pang, X. Li, L. Gao, X. Zhang, J. Wu, Multi-relational graph reinforcement learning for dynamic flexible job-shop scheduling under machine breakdowns, Applied Soft Computing 190 (2026). doi:10.1016/j.asoc.2026.114587. 43
[29] L. Ning, M. Zhou, Z. Hou, R. Goverde, F.-Y. Wang, H. Dong, Deep Deterministic Policy Gradient for High-Speed Train Trajectory Optimization, IEEE Transactions on Intelligent Transportation Systems 23 (8) (2022) 11562–11574. doi:10.1109/TITS.2021.3105380. [30] P. Yue, Y. Jin, X. Dai, Z. Feng, D. Cui, Reinforcement Learning for Scalable Train Timetable Rescheduling With Graph Representation, IEEE Transactions on Intelligent Transportation Systems (2024). doi:10.1109/TITS.2023.3344468. [31] S. Mohanty, E. Nygren, F. Laurent, M. Schneider, C. Scheller, N. Bhattacharya, J. Watson, A. Egli, C. Eichenberger, C. Baumberger, others, Flatland-rl: Multi-agent reinforcement learning on trains, arXiv preprint arXiv:2012.05893 (2020). [32] R. Mohammadi, Q. He, A deep reinforcement learning approach for rail renewal and maintenance planning, Reliability Engineering and System Safety 225 (2022). doi:10.1016/j.ress.2022.108615. [33] G. Arcieri, C. Hoelzl, O. Schwery, D. Straub, K. Papakonstantinou, E. Chatzi, POMDP inference and robust solution via deep reinforcement learning: an application to railway optimal maintenance, Machine Learning (2024). doi:10.1007/s10994-024-06559-2. [34] W. Feng, W. Dong, S. Zhai, G. Zhang, X. Sun, Y. Ji, A deep reinforcement learning method for freight train driving based on domain knowledge and mass estimation network, in: ACM International Conference Proceeding Series, 2021, pp. 41–46. doi:10.1145/3468891.3468898. [35] Y. Cui, G. Zhang, W. Dong, X. Sun, W. Yang, Knowledge-based Deep Reinforcement Learning for Train Automatic Stop Control of HighSpeed Railway, in: ACM International Conference Proceeding Series, 2020, pp. 31–36. doi:10.1145/3426826.3426833. [36] J. Lin, X. Qiu, M. Li, Tracking interval control for urban rail trains based on safe reinforcement learning, Engineering Applications of Artificial Intelligence 137 (2024). doi:10.1016/j.engappai.2024.109226. [37] J. Li, Z. Chen, Y. Zheng, S. H. Chan, D. Harabor, P. J. Stuckey, H. Ma, S. Koenig, Scalable Rail Planning and Replanning: Winning the 2020 44
Flatland Challenge, in: 14th International Symposium on Combinatorial Search, SoCS 2021, 2021, pp. 477–485. doi:10.1609/socs.v12i1.18576. [38] M. Littman, Markov games as a framework for multi-agent reinforcement learning, in: Proceedings of the 11th International Conference on Machine Learning, ICML 1994, 1994, pp. 157–163. doi:10.1016/B978-155860-335-6.50027-1. [39] F. A. Oliehoek, C. Amato, A Concise Introduction to Decentralized POMDPs, 1st Edition, Springer Publishing Company, Incorporated, 2016. [40] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, others, Relational inductive biases, deep learning, and graph networks, arXiv preprint arXiv:1806.01261 (2018). [41] L. Ugadiarov, V. Vorobyov, A. Panov, Relational Object-Centric ActorCritic, in: Proceedings of Machine Learning Research, Vol. 275, 2025, p. 1450 – 1476. [42] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: European semantic web conference, Springer, 2018, pp. 593–607. [43] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2016-December, 2016. doi:10.1109/CVPR.2016.90. [44] J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450 (2016). [45] M. Zaheer, S. Kottur, S. Ravanbhakhsh, B. Póczos, R. Salakhutdinov, A. J. Smola, Deep sets, in: Advances in Neural Information Processing Systems, Vol. 2017-December, 2017. [46] S. Fujimoto, H. Van Hoof, D. Meger, Addressing Function Approximation Error in Actor-Critic Methods, in: 35th International Conference on Machine Learning, ICML 2018, Vol. 4, 2018, pp. 1587–1596. 45
[47] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning, in: 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, 2016. [48] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: 35th International Conference on Machine Learning, ICML 2018, Vol. 5, 2018, pp. 2976–2989. [49] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, M. G. Bellemare, Deep reinforcement learning at the edge of the statistical precipice, in: Advances in Neural Information Processing Systems, Vol. 35, 2021. [50] Q. Li, Z. Han, X. M. Wu, Deeper insights into graph convolutional networks for semi-supervised learning, in: 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, 2018. doi:10.1609/aaai.v32i1.11604. [51] L. Van Der Maaten, G. Hinton, Visualizing data using t-SNE, Journal of Machine Learning Research 9 (2008). [52] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, M. Bronstein, Temporal graph networks for deep learning on dynamic graphs, arXiv preprint arXiv:2006.10637 (2020). [53] D. Kingma, J. Ba, Adam: A method for stochastic optimization, in: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015. [54] G. E. Uhlenbeck, L. S. Ornstein, On the theory of the Brownian motion, Physical Review 36 (5) (1930). doi:10.1103/PhysRev.36.823.
46