PREPRINT SUBMITTED TO THE IEEE
1
SatQNet: Satellite-assisted Quantum Network Entanglement Routing Using Directed Line Graph Neural Networks
arXiv:2604.09306v1 [quant-ph] 10 Apr 2026
Tobias Meuser∗ , Jannis Weil∗† , Aninda Lahiri‡ , Marius Paraschiv‡ ∗ Communication Networks Lab, Technical University of Darmstadt, [email protected] † Institute of Communication Technology, Leibniz University Hannover, [email protected] ‡ Quantum Information Group, IMDEA Networks, {aninda.lahiri, marius.paraschiv}@imdea.org Abstract—Quantum networks are expected to become a key enabler for interconnecting quantum devices. In contrast to classical communication networks, however, information transfer in quantum networks is usually restricted to short distances due to physical constraints of entanglement distribution. Satellites can extend entanglement distribution over long distances, but routing in such networks is challenging because satellite motion and stochastic link generation create a highly dynamic quantum topology. Existing routing methods often rely on global topology information that quickly becomes outdated due to delays in the classical control plane, while decentralized methods typically act on incomplete local information. We propose SatQNet, a reinforcement learning approach for entanglement routing in satellite-assisted quantum networks that can be decentralized at runtime. Its key innovation is an edgecentric directed line graph neural network that performs local message passing on directed edge embeddings, enabling it to better capture link properties in high-degree and time-varying topologies. By exchanging messages with neighboring repeaters, SatQNet learns a local graph representation at runtime that supports agents in establishing high-fidelity end-to-end entanglements. Trained on random graphs, SatQNet outperforms heuristic and learning-based approaches across diverse settings, including a real-world European backbone topology, and generalizes to unseen topologies without retraining. Index Terms—Quantum networks, Deep reinforcement learning, Graph neural networks, Satellite communications
I. I NTRODUCTION To unlock the full potential of quantum computing and quantum sensing, it is essential to develop robust quantum communication networks that enable quantum systems to interact and share information over long distances. Quantum communication harnesses phenomena such as entanglement to transmit quantum information securely, for example, via quantum teleportation. Quantum networks [1], [2], [3] are essential not only for scaling up quantum computing capabilities but also for enabling new technologies in quantum sensing and metrology. Entanglement distribution across a network allows for enhanced precision in measurements and synchronization tasks, which can revolutionize fields such as navigation and time measurement [4], [5]. A global quantum network, or quantum internet, would facilitate new forms of secure communication and information processing [6], fundamentally changing the landscape of information technology [7], [8], [9]. Quantum networks can be envisioned as an overlay atop classical communication infrastructures like fiber optics, where entanglement is distributed from initial elementary links (entangled pairs established between neighboring nodes) through entanglement swapping [10]. In this architecture, entanglement
swapping performed by quantum repeaters [11] plays a crucial role in extending the range of entanglement distribution, enabling entangled states to be shared between nodes that are not directly connected. While the literature on entanglement distribution in terrestrial quantum networks with stationary repeaters is extensive [12], [13], [14], [15], few studies address the complexities of satellite-based quantum network routing [11], [16], [17]. Terrestrial fiber-optic networks exhibit attenuation, causing an exponential decay of entangled photon transmission rates over long distances [18], [19]. This limits the entanglement distribution to a few hundred kilometers without repeaters, leading to elementary links that only cover comparatively small distances. Even with repeaters, the maximum achievable distance depends strongly on the length of each elementary link, as each swap operation introduces additional noise to the entangled state. Satellite-based networks offer a promising avenue for global quantum communication by bridging large distances that are impractical for ground-based links. These networks comprise both ground-based repeaters and a fleet of satellites capable of performing entanglement swapping and equipped with local quantum memories for storing qubits. Implementing satellite-based quantum networks introduces unique challenges [20], [21], [22], [23] to both the network itself and entanglement routing. Aspelmeyer et al. [24] provide a detailed list of hardware requirements for satelliteassisted long-distance quantum communication. The effects of atmospheric loss and noise significantly impact the absorption and transmission of photons between satellites and ground stations. Satellite orbital dynamics cause the lengths of links to ground stations to vary continuously, leading to a timedependent network topology. This poses additional challenges to entanglement routing, as it further increases the already high dynamics of quantum networks. In particular, approaches relying on global information about the quantum network, as presented in [25], [26], scale poorly with the size of satelliteassisted quantum networks, as the necessary information can only be gathered with significant latency. In addition, even local approaches often rely on knowledge about the underlying physical topology [27], which is often assumed to be static over time, limiting their applicability to satellite-assisted quantum networks. Although learning-based approaches offer a possible solution to entanglement routing, many approaches are trained and evaluated on a single topology or require a global view of the quantum network, making them unsuitable for satellite-assisted networks [28], [29]. Even approaches that generalize across topologies tend to make assumptions about
This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.
PREPRINT SUBMITTED TO THE IEEE
the structure of the underlying network, e.g., by limiting the maximum degree of quantum repeaters [30] or only generalizing to relatively small networks [31], [32], which also limit their applicability to satellite-assisted quantum networks. In this work, we present SatQNet, a learning-based approach that relies on learned graph representations to make routing decisions in highly dynamic satellite-assisted quantum networks. The key technical novelty of SatQNet is the use of a directed line GNN. In contrast to related work [31], [32], [30], which commonly relies on node embeddings that encode information about adjacent links only indirectly, SatQNet maintains separate embeddings for each link at each node and therefore better captures link-specific dynamics in satelliteassisted quantum networks. Our contributions are as follows: • We model time-varying ground-to-satellite and intersatellite link success probabilities and formulate a satellite-aware entanglement-routing problem. • We propose SatQNet, a learning-based decentralized routing policy that leverages a directed line Graph Neural Network (GNN) and is trained with reinforcement learning. • We perform an extensive evaluation across synthetic and real backbone topologies with statistical testing. The structure of the paper is as follows: Section II reviews the relevant literature in the field, while Section III covers our quantum network model and key concepts in quantum information. In Section IV, we model the link success probabilities for ground-to-satellite and inter-satellite links, followed by a description of our reinforcement learning approach in Section V. Section VI presents the training process and discusses the results, along with a performance comparison of the proposed method against selected heuristics and other learning-based approaches. Finally, the paper concludes with a summary in Section VII. II. R ELATED W ORK Several comprehensive review articles have surveyed the recent progress of quantum networks from various perspectives [1], [2], [3], [7], [8], [9]. Significant advancements have been made in the area of entanglement routing across networks with diverse topologies. Research in entanglement routing has focused on both bipartite entangled states [12], [13], [33], [34], [35] and multipartite entangled states [36], [37], [14], [15]. The literature presents a variety of routing methodologies, including multi-path routing [38], [39], [33], [40], which seeks to establish multiple parallel entanglement links between source and destination nodes concurrently. These links can subsequently undergo entanglement purification procedures [41]. The integration of multi-path routing with timemultiplexed quantum repeaters is explored in [42], while issues related to resource under-utilization and the consequent loss of entangled pairs are addressed in [43]. Alternatively, some approaches utilize virtual graphs constructed from entangled links that are generated on demand [44], [27]. Graph-based techniques also play a role in entanglement distribution, utilizing operations like subgraph complementation [15]. These methods are particularly advantageous in the
2
analysis of graph states [45], where local complementations facilitate the identification of classes of locally equivalent graphs. Moreover, in simple network topologies such as linear chains of repeaters, optimal entanglement routing schemes can be determined [46], [47]. While most research in quantum networking has concentrated on terrestrial quantum networks, the incorporation of satellites has only recently gained attention [11], [48], [49], [50], [51]. Experimentally, recent studies have focused on tracking methods [20], timing precision, and various implementation challenges [21], [22], [23]. Theoretically, there have been developments in secure quantum communications within satellite-based quantum networks [52], as well as the derivation of various theoretical bounds [16]. Reinforcement learning has long been explored for pathfinding in classical networks [53]. Many of these methods rely on centralized control [54], which limit their scalability. Decentralized approaches offer better scalability but suffer from limited network observability, hindering their performance [55]. This challenge is particularly pronounced in dynamic environments like quantum networks, where frequent topology changes necessitate current network information for optimal decision-making. To achieve generalizability in pathfinding tasks, GNNs have been employed in several studies [56], although these typically depend on a centralized network view. Recent advancements have introduced techniques that utilize message passing GNNs to enable generalizability in pathfinding [31], [32], [30]. However, these methods operate based on node information, which is suboptimal for an edgecentric routing task, and their generalizability is constrained to networks with certain properties. III. P RELIMINARIES AND S YSTEM M ODEL We model the quantum communication network based on an underlying optical network, referred to as the physical network, which connects quantum repeaters. Over this physical network, neighboring quantum repeaters establish elementary links, which form the basic resources for entanglement routing. To describe these resources formally, we work with qubits and, more generally, density operators ρ ⪰ 0, Tr(ρ) = 1, which capture both pure states and statistical mixtures. For a bipartite system ρAB ∈ HA ⊗ HB , the reduced state is ρA = TrB (ρAB ) [57]. The four Bell states |Φ± ⟩ = √12 (|00⟩ ± |11⟩) ,
|Ψ± ⟩ = √12 (|01⟩ ± |10⟩) (1)
are the canonical maximally entangled resources that serve as elementary links in our setting. Local Paulis permute them, so many noise processes reduce to random Pauli errors on an ideal Bell pair. Noise and loss are modeled as completely positive and trace preserving (CPTP) maps E. The primary instance used here is the depolarizing channel, whose action on a Bell pair yields a Werner state 1−F − Φ Φ− W F = F Φ+ Φ+ + 3 (2) + Ψ+ Ψ+ + Ψ− Ψ− . In (2), WF is a uniform mixture of all four Bell states weighted by a fidelity parameter F , defined as F (ρ) = ⟨Φ+ | ρ |Φ+ ⟩ ∈ [0, 1].
(3)
PREPRINT SUBMITTED TO THE IEEE
3
After their creation, elementary links are stored in quantum memories and degrade over time due to decoherence [57], [58]. We rely on the decay model developed in [30] based on the measurements of [59], which is shown in (4). " k # t + FB (4) F (t) = (F0 − FB ) · exp − T2,ndec vsi
vdi (a) Initial topology
vsi
vdi
Together, the currently available elementary links define the quantum topology and provide the resources consumed by entanglement swapping.
(b) Topology after movement
Fig. 1: Simplified example of a satellite-assisted quantum network. The quantum repeaters are depicted as nodes, connected via fiber links. Dedicated ground stations (satellite dishes) have both fiber and satellite connections. Link colors represent entanglement generation probability, with blue indicating higher probability (shorter links) transitioning to red for lower probability (longer links).
The fidelity represents the overlap of the considered state ρ with a target state, in this case the pure Bell state |Φ+ ⟩. Throughout the paper, the fidelity value associated with an elementary entangled pair between two repeaters serves as the primary scalar measure of link quality. A. Quantum repeaters To establish end-to-end entanglement over longer distances, elementary links must be combined through entanglement swapping at intermediate quantum repeaters. In our model, we consider two types of repeaters, as shown in Fig. 1: stationary repeaters and mobile repeaters. Stationary repeaters are fixed nodes located at specific sites, such as small data centers, and are interconnected via optical fibers. These fiber connections are stable and remain constant during network operation. A subset of these stationary repeaters is equipped with ground-toair communication interfaces, enabling them to establish connections with mobile repeaters. Mobile repeaters, exemplified by satellites, can communicate both with stationary repeaters and with other mobile repeaters through air-based optical links, whose properties may differ depending on the involved nodes. Because mobile repeaters change their positions over time, the availability and quality of these links vary dynamically. B. Generation and Decoherence of Elementary Links Elementary links are created with a given probability and form the basis of the quantum topology. This probability depends on the distance and connection type between nodes v1 and v2 . For fiber-based links, this probability decays exponentially with distance, while the behavior of air-based links is more complex and is modeled in Sec. IV. The probability of establishing elementary links via fiber is constant over time, whereas the probability for air-based links changes dynamically due to node mobility. This behavior can be observed by comparing Fig. 1a with Fig. 1b.
C. Entanglement Swapping Entanglement swapping enables two nodes without a direct entangled link to become entangled via an intermediate node [10], [60]. Operationally, swapping consumes two input pairs and produces one output pair (possibly only probabilistically, depending on the hardware) [61]. Imperfect local operations can be incorporated as additional local noise that reduces the resulting fidelity. In this work, we simulate the swapping procedure in Qiskit [62]. D. Routing in Quantum Networks In our quantum network model, nodes may request endto-end entanglement with other nodes in the network. We denote such a request by a source-destination pair (vsi , vdi ). As in classical communication networks, a route for a sourcedestination pair is a sequence of neighboring nodes that starts at the source node vsi and ends at the destination node vdi . Using the available elementary links, the goal is to establish end-to-end entanglement for each active source-destination pair. This is achieved by performing a sequence of entanglement swaps along the selected path. The objective is to maximize the quality of the resulting end-to-end entanglement, measured primarily through its fidelity, while also maximizing the number of successful end-to-end entanglements. Once a path is planned, the repeaters along that path perform swap operations, thereby successively extending the entanglement. The quality of each repeater along the path introduces additional noise during the swaps, reducing the fidelity of the resulting entanglement. Consequently, the fidelity of an endto-end entanglement depends not only on the availability of elementary links, but also on their fidelities and on the quality of the repeaters performing the swap operations. Thus, finding high-fidelity routes in satellite-assisted quantum networks requires knowledge of the current quantum topology, i.e., the quality and availability of elementary links. During the routing process, the quantum topology may change frequently, and routing approaches must adapt accordingly. E. Quantum network operation During the operation of our quantum network model, all neighboring repeaters attempt to establish as many elementary links as possible between them. At the same time, paths can be planned to establish end-to-end entanglements between all active source-destination pairs. We adopt a continuous entanglement generation approach, in which elementary links
PREPRINT SUBMITTED TO THE IEEE
4
are generated continuously while the routing is performed. This continuous generation of entangled links ensures that the network maintains available quantum resources throughout the routing operation, enhancing the efficiency and reliability of entanglement distribution. While this increases the number of available elementary links, it also makes the quantum topology more dynamic. Once a new elementary link is created, the maximum fidelity of any elementary link between two repeaters will increase. In addition, links without any available elementary link, which would otherwise be considered unusable, might generate a new elementary link, effectively reactivating the link from the perspective of the quantum topology. IV. O PTICAL L INK M ODEL FOR G ROUND - TO - SATELLITE AND I NTER - SATELLITE L INKS In the time-evolving graph representation of the network, each edge corresponds to a potential elementary quantum link. We associate with each edge a time-dependent single-photon success probability Plink (u, v, t), defined as the probability that a photon emitted by the local source into the spatial mode of the edge (u, v) at time t is successfully detected at the remote node after propagating through the channel. We distinguish two types of links: (i) ground–satellite or satellite–ground links (GS/SG), where one endpoint is a terrestrial station and the other is a satellite; (ii) inter-satellite links (SS), where both endpoints are satellites connected through a vacuum free-space optical (FSO) channel. We model both SS and GS/SG communications based on established physical approaches and environmental constraints. For additional details the reader is also directed to Refs. [63], [64], [65]. These link probabilities are used as edge weights by the higher-level learning and routing modules: edges with large Plink (u, v, t) are usable for entanglement distribution, whereas edges with small Plink (u, v, t) are effectively unavailable. Geometry and airmass for ground–satellite links: Let the Earth be modeled as a sphere of radius RE , and let the ground station be located at geodetic latitude ϕg , longitude λg . A satellite at altitude hs has orbital radius rs = RE + hs and geodetic coordinates (ϕs (t), λs (t)) at time t. The central angle γ(t) between the ground station and the satellite sub-point is cos γ(t) = sin ϕg sin ϕs (t) + cos ϕg cos ϕs (t) cos λs (t) − λg . (5) The corresponding ground–satellite slant range is q 2 − 2R r cos γ(t). (6) LGS (t) = rs2 + RE E s The elevation angle El(t), i.e. the angle between the line-ofsight and the local horizontal at the ground station, is
The atmospheric transmittance for a photon traversing the atmosphere at zenith angle ζ is modeled by a Beer–Lambert law applied to a slant path: sec ζ ηatm (ζ) = exp − τzenith sec ζ = Tzenith , (8) where τzenith is the zenith optical depth at the signal wavelength and Tzenith = e−τzenith is the corresponding vertical transmissivity. For clear mid-latitude nights around λ ≈ 850 nm one finds Tzenith ≈ 0.8 (i.e. τzenith ≈ 0.22) [63], [64], [65]. Diffraction-limited geometric collection: We model the optical beam in both GS/SG and SS links as a fundamental Gaussian mode. Let DT and DR be the transmitter and receiver aperture diameters, and let aR = DR /2 be the receiver radius. Denote by θdiv the (far-field) 1/e2 half-angle beam divergence and by L the distance between transmitter and receiver. In the far-field regime, the beam radius at distance L is w(L) ≃ θdiv L,
and the fraction of power intercepted by the receiver aperture (the diffraction/geometric transmissivity) is 2a2R . (10) ηdiff (L) = 1 − exp − w(L)2 If we additionally assume diffraction-limited transmit optics with Gaussian waist w0 ≈ DT /2, then θdiv = λ/(πw0 ) ≈ 2λ/(πDT ) and (10) can be written explicitly as 2 π 2 DT2 DR . (11) ηdiff (L) = 1 − exp − 8λ2 L2 (11) makes explicit the 1/L2 scaling inside the exponential governing the geometric collection loss. In practice, the effective θdiv may be larger than the diffraction limit because of intentional beam expansion and residual optical aberrations; in that case it is more robust to treat θdiv as an independent hardware parameter and use (10) directly [63], [64], [65], [67]. Pointing statistics: Residual pointing errors due to mechanical jitter and finite-bandwidth tracking cause additional attenuation. For a single terminal we model the instantaneous coupling loss as 8 Λp (θ) ≈ exp − Gθ2 , G ≈ 2 , (12) θdiv where θ is the instantaneous radial pointing error angle, and G is a “pointing stiffness” parameter that depends only on the beam divergence [65], [66]. If the elevation and azimuth pointing errors are independent zero-mean Gaussians with variance σp2 , the radial error θ is Rayleigh-distributed. Averaging Λp (θ) over this distribution yields the mean pointing efficiency of a single terminal η̄point =
cos γ(t) − (RE /rs ) tan El(t) = , sin γ(t)
(7)
and the zenith angle is ζ(t) = π2 − El(t). In simulations we typically impose a minimum elevation El(t) ≥ Elmin (e.g. 20◦ ); below this threshold we set the link probability to zero.
(9)
1 1 = 2 . 2 1 + 2Gσp 1 + 16σp2 /θdiv
(13)
For an inter-satellite link, both transmitter and receiver terminals are subject to independent jitter, so the total mean pointing efficiency is approximately the product of two such factors, 2 η̄point,SS ≈ η̄point . (14)
PREPRINT SUBMITTED TO THE IEEE
5
Parameter
Meaning
Ground-to-satellite
Inter-satellite
RE hs λ DT DR θdiv Tzenith τzenith σp ηopt,tx ηopt,rx ηdet ηsource hmin
Earth radius Satellite altitude Optical wavelength Tx aperture diameter Rx aperture diameter Full beam divergence Zenith transmittance Zenith optical depth RMS pointing jitter per terminal Tx internal optical efficiency Rx internal optical efficiency Detector efficiency Source efficiency (into the mode) Grazing altitude threshold in (18)
6371 km 500–1200 km (LEO) 850 nm (e.g. Micius downlink) 0.30 m (satellite) [63], [64] 1.0 m (ground) [63], [64] ∼ 10 µrad (GS/SG) [63], [64] ≈ 0.8 (clear night at 850 nm) [63], [64] ≈ 0.22 (Tzenith ≈ 0.8) 0.2–0.5 µrad (fine tracking) [64] 0.5–0.8 ≈ 0.16 (ground telescope + coupling) [63] ≈ 0.5 (Si SPADs at 850 nm) [63] 0.3–0.7 (protocol dependent) 20 km
6371 km 500–600 km 850 nm or 1550 nm 0.10 m (typical ISL terminal) [65] 0.10 m 20–40 µrad (SS) [65] 1 0 5–10 µrad (SS) [65], [66] 0.5–0.8 0.5–0.8 0.5–0.9 (e.g. SNSPDs at 1550 nm) 0.3–0.7 (protocol dependent) 20 km
TABLE I: Representative parameter values for ground-to-satellite and inter-satellite links. Values are consistent with reported satellite quantum key distribution (QKD) demonstrations and free-space optical (FSO) link budgets [63], [64], [65], [66], [67].
Satellite–satellite visibility: For a pair of satellites i and j with position vectors ri (t) and rj (t) in an Earth-centered frame, the inter-satellite distance is Lij (t) = ri (t) − rj (t) .
(15)
We introduce a binary visibility function Ivis (i, j, t) that accounts for Earth occultation. Let d(t) = rj (t) − ri (t) and define ri (t) · d(t) ∗ s (t) = clip[0,1] − , (16) ∥d(t)∥2 rmin (t) = ri (t) + s∗ (t) d(t) .
(17)
as the minimum distance of the line segment between the two satellites to the Earth’s center. We use the clip function here to force the s(t) parameter into the [0, 1] range (so s = 0 at satellite i, s = 1 at satellite j, and 0 < s < 1 for intermediary points). The clip function clip[0,1] (x) is 0 below the [0, 1] interval, x inside the interval and 1 above. We then set ( 1, rmin (t) > RE + hmin , Ivis (i, j, t) = (18) 0, otherwise, where hmin is a grazing altitude threshold (we take hmin ≈ 20 km) to ensure that the optical path does not cross the dense lower atmosphere [67], [65]. Ground–satellite (GS/SG) link probability: For a GS/SG channel we group all hardware efficiencies into a single factor ηHW,GS = ηsource ηopt,tx ηopt,rx ηdet η̄point,GS ,
(19)
where ηsource is the source efficiency (probability that an “attempt” injects a photon into the correct spatial mode), ηopt,tx and ηopt,rx are transmit and receive internal optical efficiencies (including telescope and fiber coupling), ηdet is the singlephoton detector efficiency, and η̄point,GS is the mean pointing efficiency of the GS/SG link (often close to unity for wellengineered tracking systems). The per-photon success probability of a GS/SG link at time t is then PGS (t) = ηHW,GS ηatm ζ(t) ηdiff LGS (t) (20)
with ηatm defined in (8) and ηdiff in either (10) or (11). This expression is symmetric between downlink (satellite → ground) and uplink (ground → satellite); in practice one may use different hardware parameters in ηHW,GS for the two directions (e.g. to reflect stronger turbulence and higher background noise in uplinks [63], [64], [65]). For an entanglement-based protocol where, say, one photon of each pair is transmitted along the GS/SG link and the other photon is detected locally at the source station, the elementary entanglement success probability per attempt is obtained by multiplying PGS (t) by the local detection probability at the source node. Satellite–satellite (SS) link probability: For an SS link between satellites i and j, there is no atmospheric extinction (ηatm ≈ 1) and both terminals jitter. We again group all nongeometric efficiencies into ηHW,SS = ηsource ηopt,tx ηopt,rx ηdet η̄point,SS ,
(21)
where η̄point,SS is given by (14). The per-photon SS success probability is then PSS (i, j, t) = Ivis (i, j, t) ηHW,SS ηdiff Lij (t) (22) with Ivis (i, j, t) from (18) and ηdiff from (10) or (11). As before, in entanglement-based protocols the elementary entanglement success probability per attempt is obtained by multiplying PSS (i, j, t) with the local detection probabilities at the relevant nodes. Table I summarizes recommended default values for the main parameters of (20) and (22). The GS/SG values are chosen to be representative of the Micius satellite downlink and similar experimental demonstrations [63], [64], [65], while the SS values are motivated by existing and proposed freespace optical inter-satellite terminals [65], [66], [67]. In the simulations one can either use the detailed decomposition in (20)–(22), or pre-collapse the hardware factors into single scalars ηHW,GS and ηHW,SS calibrated from a more detailed link budget or experimental measurements. The resulting PGS (t) and PSS (i, j, t) then form the time-dependent edge weights for the effective network graph used by the routing and reinforcement-learning components.
PREPRINT SUBMITTED TO THE IEEE
V. S AT QN ET: R EINFORCEMENT L EARNING FOR S ATELLITE - ASSISTED Q UANTUM ROUTING We propose to address the challenges of satellite-based quantum routing like dynamic link properties and high node degrees, as presented in Section III, with deep reinforcement learning. In particular, GNNs allow reinforcement learning agents to generalize over previously unseen topologies and network conditions [32], making them a suitable choice for the highly dynamic topologies of quantum networks. In the following, we provide a short introduction to combining reinforcement learning with GNNs, followed by the modeling of the agents and nodes in the network. A. Generalizable Reinforcement Learning through GNNs Reinforcement learning considers the interaction of an agent with an environment. The agent observes the environment and selects an action based on the received observation. For each action, the agent receives a numerical reward that quantifies the quality of the selected action. The goal is to maximize the long-term rewards and thereby optimally solve a given task. In deep reinforcement learning, the agent’s policy function that defines the mapping from observations to actions is represented by a neural network. Deep reinforcement learning is agnostic to the neural network architecture, but the choice can inherently limit generalizability. In particular, neural network architectures with fixed input and output dimensions cannot process graphs of arbitrary size and order. Our main requirement for satellite-based quantum networks is that the approach should, at least conceptually, be able to generalize over any network topology. As monitoring information in the network can be modeled in graph form, GNNs are a natural choice. However, while technically compatible with any graph, feedforward message-passing GNNs have a fixed number of layers, which limits the extent to which information is propagated through the graph. This limits the maximum supported path length between source and destination nodes. Weil et al. [32] leverage a recurrent GNN to continuously propagate information through the graph and thereby support arbitrary path lengths, but their approach is limited to graphs with a fixed number of nodes and a uniform node degree. The extension of Meuser et al. [30] supports an arbitrary number of nodes, as the observations are independent of the graph order, but is still limited by a maximum node degree. Geyer et al. [31] also use a recurrent GNN and propose modeling network interfaces as nodes in addition to routers. This allows for unified processing of node and interface features and renders the approach applicable to graphs with arbitrary maximum node degree. However, the use of one-hot encoded node identifiers as input features limit their approach to graphs of fixed order. Our proposed approach supports graphs of arbitrary order, size, and maximum node degree. We train a single model on a set of graphs and then apply it to arbitrary unseen graphs. 1) Recurrent GNN fundamentals: Let G = (V, E) be a graph with nodes V , undirected edges E, and time-dependent input features xvt , xet for each node v ∈ V and edge e ∈ E at discrete time t ∈ {0, 1, 2, . . . }. This graph represents
6
the satellite-based quantum network. Analogous to related work [30], we leverage a recurrent GNN where each node v ∈ V manages a node embedding hvt that is updated as ĥvt = encode (hvt , xvt ) n o {v,w} Mtv = aggregate (ĥvt , xt , ĥw ) | {v, w} ∈ E (23) t v v v ht+1 = update ĥt , Mt , where encode, aggregate, and update are arbitrary differentiable functions. The function encode encodes information from the node features into the node embedding, aggregate aggregates features from adjacent edges and the node embeddings of neighboring nodes, and update updates the node embedding with the aggregated information. By repeated execution of (23), node embeddings eventually propagate through the entire graph, which enables distant nodes to communicate. We assume that the edge features x{u,w} are known to both nodes u, w connected by the edge. Note that (23) only requires nodes to exchange node embeddings with their direct neighbors, which implies that its execution can be distributed. 2) Support of dynamic node degrees: Conceptually, the above GNN is applicable to any graph as long as the individual input features xvt , xet do not depend on the order of the graph, the size of the graph, or the maximum node degree. An agent i located at a quantum router v ∈ V at time t leverages local information available at this router to make a routing decision. To select one of the neighbor nodes ait ∈ N (v) := {w | {v, w} ∈ E} at time t independent of the node degree, the agent’s action space must resize dynamically. Inspired by early works on reinforcement learning for routing [53], we achieve this by processing each neighbor separately as {v,w}
ait = argmaxw∈N (v) Q(oit , hvt+1 , xt
, hw t ),
(24)
where Q can be any differentiable function that returns a scalar for each neighbor w based on the information available to agent i and node v at time t. Here, oit refers to the observation of agent i. Details on the concrete observations, node features, and edge features used in this work are provided in Sec. V-B. Note that the invocations of Q can be parallelized, i.e., Q can be evaluated simultaneously for all neighbors. 3) Improving the expressiveness of GNNs: In related works, each repeater is typically modeled as a node in the GNN. However, to make routing decisions, the focus is on edgelevel predictions (see (24)). This may require Q to derive edge information from the node representations hvt+1 and hw t . When assuming node embeddings of fixed size, they cannot encode features from an arbitrary number of incident edges without loss of information. This can harm the predictive performance of the model. Recent works found that edge-level predictions can be improved substantially by leveraging line graphs [68]. A line graph transforms each edge e ∈ E in the base graph to a node. Two different line graph nodes e1 ̸= e2 are connected by a line graph edge iff they represent adjacent edges in the base graph, i.e., ∃v ∈ V : v ∈ e1 ∧ v ∈ e2 . The GNN is then simply applied to the line graph. To make edge-level predictions for an edge e in the base graph, now only the node embedding he of the corresponding node in
PREPRINT SUBMITTED TO THE IEEE
7
1
1
1 vs1
vd1
(a) Start of path planning
vs1
vd1
(b) After first agent decision
vs1
vd1
(c) Agent moves to satellite
vs1
1 vd1
(d) Finished path planning
Fig. 2: Visualization of path-planning in satellite-assisted quantum networks. The quantum repeaters are depicted as the nodes of the graph, which are connected via fiber. Dedicated ground stations have both fiber and satellite connections and serve as hubs between the topologies. If elementary links are available between repeaters, each blue wavy line represents one of these links. The agent is represented as an orange rectangle, which moves across the network during planning. The planned path is depicted as an orange line. Once the path is planned, the elementary links are consumed to create an end-to-end entanglement.
the line graph needs to be considered as input to the scoring function Q. Similarly, it has been shown that transformer models operating on node pairs instead of nodes can achieve higher expressiveness and better empirical performance [69]. The information flow between nodes in GNN models is typically bidirectional, which can lead to an over-homogenization of node embeddings. Pahng and Hormoz [70] propose to explicitly learn the direction of the information flow per edge and show that this improves the expressiveness and empirical performance. 4) Directed Line GNN: To combine the expected benefits of line graph GNNs and a directed information flow, we propose to apply the GNN on the directed line graph, which is constructed as follows. For each undirected edge {v, w} ∈ E in the base graph G, the directed line graph has two nodes a = (v, w), a′ = (w, v), representing the information flow in each direction. We denote the set of line graph nodes that ⃗ Two line graph represent directed edges in the base graph E. ⃗ nodes a, b ∈ E are connected with a directed line graph edge ⃗ if there exists a common node v ∈ V in the base (a, b) ∈ A graph such that ∃u, w ∈ V : a = (u, v) ∧ b = (v, w), i.e., when a is an incoming edge to v and b is an outgoing edge ⃗ = (E, ⃗ A) ⃗ the from v. We call the resulting directed graph G directed line graph of G. When applying the GNN architecture from (23) to the directed line graph, embedding information is now effectively ⃗ edge- and node-level managed for directed edges (u, v) ∈ E, input features are switched, and information is now forwarded backwards along the directed edges of the line graph. Distributed execution of the resulting model is still possible. However, as edges have no computational capacity in practice, the edge embeddings must be managed by nodes in the network. A naive strategy is to let node u manage ⃗ the edge embeddings of all outgoing edges (u, w) ∈ E. Note that, when applying (23) to the base graph, each node manages a node embedding of fixed dimensionality. With the directed line graph model, each node u now manages embedding information of varying dimensionality, depending on the number of incident edges.
B. Integration into Reinforcement Learning An end-to-end entanglement request consists of a unique identifier i, a source-destination pair (vsi , vdi ), and a Time to Live (TTL). This request serves as the basis for the path-planning of our reinforcement learning agent. Similar to related work [30], SatQNet plans paths iteratively. However, in contrast to related work, the path-planning may consider links to satellites to cover large distances, as visualized in Fig. 2. As depicted in Fig. 2a, agent i is instantiated at its source quantum repeater vsi and aims to move towards its destination repeater vdi . At each environment step, the agent is located at exactly one repeater, as depicted in Figs. 2b and 2c. The agent can choose any of the physical connections available as its next step along the path. The operation of the agent is decoupled from the entanglement swap operation, such that unnecessary decays of created entanglements are avoided. Only once a path has been planned, as shown in Fig. 2d, an entanglement swap is performed on all repeaters along the path to create an endto-end entanglement. 1) Observation Space: The observation space contains information about the end-to-end entanglement request associated with the agent and about the neighboring quantum repeaters and the connection to them. For a request, the agent observes the length of its current path to track the number of steps that remain until the TTL expires and path-planning fails. Recall that at each step, the agent is located at a quantum repeater. For each neighboring quantum repeater vj , the agent observes its swap probability and its role in the request. The agent observes a binary feature that indicates whether the neighbor vj is the destination vdi of agent i, i.e., vdi = vj . In addition, it observes whether vj is in the same ground cluster as vdi . This topological information supports the path-planning of the agent in the satellite topology with multiple clusters. For each outgoing connection, the number of available elementary links on the connecting optical link and the highest fidelity of the available elementary links are observed. Note that the agent neither directly observes the global location of the destination nor does it receive any shortest-path heuristics. Instead, these properties are learned by the quantum
PREPRINT SUBMITTED TO THE IEEE
repeaters. In addition to handcrafted observations, the agent receives the learned GNN embedding information available to the quantum repeater it is located on (compare (24)). When using the directed line GNN proposed in Sec. V-A4, the agent observes the edge embeddings of all outgoing links. 2) Node and Edge Features: For each simultaneous sourcedestination pair i, a separate instance of the GNN is required. While this introduces overhead for each additional sourcedestination pair, previous work has shown that the required resources are relatively low [30]. As input to each instance of the GNN for source-destination pair i, node and edge features are required. The node features describe properties of the quantum repeater, specifically its swap probability and its role in a specific request i. While the swap probability is independent of i, the role depends on i and is divided into two aspects, similar to the agent’s observation. The first is a binary feature that indicates whether node v is the destination vdi of agent i, i.e., vdi = v. In addition, we encode whether v is in the same ground cluster as vdi . The edge features describe properties of a connection between two quantum repeaters, specifically the number and quality of available elementary links as well as the probability of a successful establishment of a new elementary link. 3) Action Space: The action space of the agent i depends on the quantum repeater vci that it is currently associated with, as shown in Fig. 2. The size of the action space matches the degree N (vci ) of this quantum repeater vci , and the action ait ∈ N (vci ) corresponds to the next hop. The agent may choose any outgoing connection of the repeater, which will move it to the neighbor associated with this connection as shown in Fig. 2b. This adds a reservation to the connection, which is not bound to any specific elementary link but always reserves the elementary link with the highest fidelity. The reservations are removed once the path-planning is completed and the elementary links are used to create an end-to-end entanglement. Without such reservations, resource contention is a significant issue in multi-agent scenarios. 4) Objective and Evaluation Function: The objective of agent i is to select a path to its destination di that maximizes the expected end-to-end fidelity while limiting resource consumption, thereby supporting a larger total number of established entanglements. As the end-to-end fidelity is only known upon completion of the path, related work [30] utilizes a sparse reward function, where rewards are given upon completion of a path based on its end-to-end fidelity. i The end-to-end fidelity F (τ0:T ) depends on the trajectory i of nodes τ0:T = (v0 , v1 , . . . , vT ) visited by agent i from step t = 0 to step T . It is determined by recursive swap operations between all intermediate neighbor nodes as i if vt = vt+1 , F (τt+1:T ) i F (τt:T ) = f{vt ,vT } if {vt , vT } ∈ E, i swap(f{vt ,vt+1 } , F (τt+1:T )) otherwise, (25) where the function swap models the entanglement swap operation (see Section III-C) and f{u,w} is the fidelity of a direct entanglement from node u to its neighbor w.
8
The goal of reinforcement learning methods is to find an optimal policy PT that maximizes the expected discounted return Git = k=t+1 γ k−t−1 rki . Given a sparse reward rTi = i F (τ0:T ) only at the end of a path if an end-to-end entanglement to the destination could be established, the discounted return for any intermediate step t ∈ {0, 1, . . . , T − 1} reduces i to Git = γ T −t−1 F (τ0:T ). In our quantum routing environment, the discount factor γ ∈ (0, 1] controls the trade-off between entanglement quality and resource consumption. A larger γ results in a prioritization of high-fidelity elementary links, yielding higher-quality end-to-end entanglement at the expense of longer paths and thus increased resource usage, which may reduce the overall entanglement generation rate. In theory, this framework allows learning optimal paths in any network. In practice, however, we found that it is challenging to estimate the quality of long paths, resulting in poor decisions and hence the selection of suboptimal paths. Even if a path from the source to the destination node is found, when elementary links along the path have low fidelity, the end-to-end fidelity may become zero, and agents thus may not receive any meaningful feedback to improve their policy. To address this issue, one would like to define a dense reward function that assigns rewards to each intermediate action. Unfortunately, (25) cannot easily be decomposed into intermediate rewards, as the swap operation does not behave additively (as the rewards in the discounted return) and because the end-to-end fidelity ultimately depends on all elementary links in the chosen path to the destination. In this work, as an alternative to a dense reward function, i of agent we propose to assess the quality of the trajectory τt:T i from step t to step T with the following evaluation function i γ T −1−t F (τt:T ) if vT = di , i i (26) E (τt:T ) = 0 otherwise. Intuitively, it evaluates the quality of the subpath from the current node vt to the final node vT , assuming that the current i , this node was the start node. Given a complete trajectory τ0:T yields qualities for each intermediate step t < T . Note that for t = 0, the evaluation function is equivalent to the discounted return. However, for t > 0, and particularly when vt is close to the destination di , the quality returned by the evaluation function may be positive, even if the entanglement from the source node v0 to the destination could not be established. The evaluation function provides a more fine-grained learning target than the sparse reward function, but deviates from common reinforcement learning frameworks. Reward functions are usually defined on a single state transition, and recent related works investigate non-Markovian rewards that depend on the history of past states and actions [71], [72]. In our case, however, the evaluation function depends on future states and actions and can thus only be computed upon episode completion. While our work focuses on the practical effect of this design decision on quantum routing, this may also provide an interesting opportunity for future basic research. Agents are trained by iteratively approximating the qualities returned by the evaluation function and acting ϵ-greedily upon the predicted qualities. We leverage an experience replay
PREPRINT SUBMITTED TO THE IEEE
9
SatQNetR
SatQNet
RELiQ*
RELiQ
0.7
EDR
50
Fidelity
75
100
EDR
SatQNetu
50
0.6
25
0
0.5
0 0.0
2.5
5.0
7.5
10.0
w/o satellites
with satellites
0
10
20
30
40
50
nth entanglement
Training Step [Millions]
(a) EDR during training
(b) EDR during testing
(c) Fidelity during testing with satellites
Fig. 3: Performance of SatQNet compared to learning-based approaches.
memory, as in deep Q-learning [73], to improve stability. However, instead of the temporal-difference method, we simply minimize the mean squared error between the predicted qualities and those obtained via the evaluation function. VI. E VALUATION
To assess whether observed performance differences are significant, we use one-sided paired permutation tests based on the mean values (H0 : baseline ≥ SatQNet; H1 : SatQNet > baseline), with 100 000 random permutations per test. Such tests are effective in analyzing classifier performance [74], [75]. When we compare SatQNet against multiple baselines, the p-values are adjusted with the Holm–Bonferroni method. Pairwise comparisons (e.g., SatQNet vs. SatQNetu ) are reported without that adjustment.
EDR
We analyze the performance of SatQNet in diverse scenarios and settings using the evaluation framework100 developed in [30], which we extend with satellites and ground stations. 50 The satellite topology is based on the Starlink topology, A. Learning-based Approaches and Ablations 1 which we obtained using CelesTrak . In that version, there 0 We compared SatQNet against other learning-based apare 9307 satellites available, which is slightly lower than the 0.0 2.5 5.0 7.5 which 10.0 proaches, use local observations and message exchange current estimate of approximately 10000 satellites. During Training Step [Millions] to make routing decisions. We utilized ϵdecay = 0.9999, the simulation, the satellites move using predictions from the γ = 0.95, a learning rate of 0.0005, a sequence length of Simplified General Perturbations (SGP4) model. The ground 20, a mini-batch size of 32, and a replay buffer with 100 000 topology and the satellite topology are connected using ground entries. Each approach was trained in episodes of 1000 steps stations, which have both a fiber connection and a wireless link each, up to a total of 10 000 000 steps. During an episode, to exchange data and create elementary links with satellites. an end-to-end entanglement request is created every 100 ms. We evaluate SatQNet against seven representative We trained on a topology of two ground clusters, which are entanglement-routing baselines: Q-PATH and Q-LEAP [25], connected by satellites. MGER, LBER, and NoNLBER [27], AER [26], and Learning approaches that are not able to generalize to RELiQ [30]. These algorithms were selected to cover new topologies are excluded here, as they cannot handle the different routing strategies and information assumptions. dynamically changing topologies in our evaluation. SatQNet MGER, LBER, NoNLBER, and RELiQ are decentralized and is our proposed solution based on directed line graphs and rely only on local quantum topology information. In contrast, the evaluation function described in (26), while SatQNetR Q-LEAP, Q-PATH, and AER rely on centralized coordination u with global topology information, which in our simulation utilizes the reward proposed in RELiQ [30] and SatQNet is collected through a dedicated monitoring framework. utilizes only an undirected line graph. As related learningEspecially for large topologies, centralized methods can based baselines, we compare against RELiQ [30] and a slightly suffer from stale information about distant quantum links. improved version of it, which we label RELiQ*. RELiQ* is RELiQ is also decentralized and learning-based, and is an extension of RELiQ developed to make it applicable to satellite-assisted networks and relies on node observations that therefore the most closely related baseline to our approach. To compare the performance of the approaches, we utilized are independent of the connectivity of quantum repeaters by the Entanglement Distribution Rate (EDR) as main metric, using a similar mechanism as described in Sec. V-A2. It can be which corresponds to the number of end-to-end entanglement seen in Fig. 3a that RELiQ struggles to converge in satellitecreated over the course of one episode. In addition, the fidelity assisted networks due to the high and strongly fluctuating node of the created entanglements is used to assess their usability degree. RELiQ* solves this issue and remains competitive even for quantum applications. If the fidelity is not mentioned in satellite-assisted networks. The three versions of SatQNet explicitly, the fidelity of entanglements created by SatQNet perform similarly during training, with only minor differences in the number of created end-to-end entanglements. However, is similar or higher to those of the other approaches. when evaluating the resulting models in Fig. 3b, we observe 1 https://celestrak.org/ differences in their learned behavior. We compare two settings,
PREPRINT SUBMITTED TO THE IEEE
10
SatQNet RELiQ*
Q-PATH Q-LEAP
NoNLBER LBER
MGER AER
100 80
60
EDR
EDR
80
60
40
40
20
20
0
0 10
30
100
300
1
Size of the quantum network
3
7
10
Number of source-destination pairs
(a) Varying the number of quantum repeaters.
(b) Varying the number of source-destination pairs.
80 80
60
60
EDR
EDR
EDR
80
40
40
60 40
20 20
20 0 0
0% 1
0.1%
1%
0
10%
100%
0.1% 1% 10% 100% Satellite 3fraction (relative to full Starlink 0% constellation) Satellite fraction (relative to full Starlink constellation) Number of ground clusters 2
(c) Varying the number of fully separated network islands.
(d) Varying the share of satellites.
Fig. 4: Entanglement Distribution Rate (EDR) based on the scale and topology of the quantum network.
one in which no satellites are present, and another in which satellites are essential to connect two ground clusters each with 100 ground repeaters. The high node degree of satellite networks leads to performance reductions of RELiQ* in satellite-assisted topologies. Similarly, when utilizing SatQNet based on a undirected line GNN, the performance reduces significantly in the satellite topology (p < 0.001). In Fig. 3c, the fidelity of each approach is shown for the satellite-based topology. While all approaches create a few high-fidelity entanglements, the fidelity of successive entanglements is consistently higher for SatQNet. B. Results Fig. 4a displays the performance of the different routing algorithms for different sizes of the ground cluster. In each plot, the bar height is the mean EDR over 100 episodes per approach. Error bars span the 25th to 75th percentile across those episodes; a marker indicates the median. Permutation tests use the per-episode EDR values as samples. The performance of global-knowledge-based approaches QPATH and Q-LEAP is very high for small network sizes, as information about the underlying topology can still be obtained with almost no latency. When increasing the number of ground nodes from 10 to 30, the performance of most approaches increases, as more resources are available for entanglement routing. As network size increases further, the performance of Q-PATH and Q-LEAP diminishes drastically, such that
even comparatively simple local-knowledge-based approaches become competitive. Similar to SatQNet, RELiQ* performs well across all network sizes. However, SatQNet achieves a higher EDR than RELiQ* for all evaluated network sizes, as its edge-based view of the network adapts better to the varying connectivity of quantum repeaters. For a network with at least 30 ground nodes, SatQNet outperforms every baseline; all seven comparisons are significant (Holm-corrected p < 0.05 for 30 nodes, p < 0.001 for 100 and 300 nodes). Fig. 4b shows the performance per source-destination pair of all approaches when multiple source-destination pairs need to be served simultaneously. While the EDR decreases for all approaches due to resource contention, it can be observed that SatQNet performs well in this multi-agent setting, even though only a single agent was considered during training. Its performance is consistently higher than all baseline approaches (Holm-corrected p < 0.001), but the relative improvement reduces once the network becomes overloaded with requests. This is especially impressive, as the global-knowledge-based approaches have a clear advantage due to the additional coordination that is possible during planning at a central instance. From 7 to 10 simultaneous source-destination pairs, the total number of established entanglements stagnates, as the generation of elementary links becomes the limiting factor. However, we observed that the fidelity of the created entanglements is higher for Q-PATH and Q-LEAP compared to SatQNet when resource contention is high.
PREPRINT SUBMITTED TO THE IEEE
11
SatQNet RELiQ*
Q-PATH Q-LEAP
NoNLBER LBER
MGER AER
60
20
40
EDR
EDR
EDR
40 40
20 0
0 0.5
0.7
0 5.0
0.9
20
7.5
10.0
0.2
RMS jitter [rad × 10−6 ]
Detector efficiency
(a) Varying the detector efficiency.
(b) Varying the antenna jitter.
0.5
0.8
Zenith transmittance
(c) Varying the atmospheric conditions.
Fig. 5: Influence of satellite parameters on the performance of the approaches. 80
EDR
Fig. 4c shows the effect of completely separated ground 60 clusters. If more than one ground cluster is simulated, the 0% 25% 50% 40 in different ground start and destination are always placed Success rate clusters to enforce the usage of satellites. It is evident that the 20 performance of all approaches decreases when satellite-based 0 links must be used, as the relatively high distance between 0%rates. 0.1% 1% 10% 100% satellites causes low elementary link generation Howfraction (relative to full Starlink constellation) ever, the results for 2 and 3 ground clustersSatellite are very similar for all approaches, highlighting that they can correctly identify the destination cluster. In all cases, SatQNet significantly outperforms all other approaches (Holm-corrected p < 0.001). Fig. 4d shows the influence of the share of satellites for a single ground cluster. Thus, entanglements can always be created using ground-only paths. Even when the satellite share is small, the performance of all approaches increases slightly. This shows that ground-based quantum networks already benefit from a comparatively small number of satellites in orbit. In all configurations, SatQNet outperforms all other approaches significantly (Holm-corrected p < 0.001). This highlights the adaptability of SatQNet to different conditions. As already discussed in Sec. IV, the parameters of the satellite model have a strong influence on the success probability for establishing elementary links, thus limiting the number of end-to-end entanglements that can be created. Fig. 5 shows the influence of three important parameters, the efficiency of the detector, the jitter of the antenna, and the zenith transmittance. The results are generated in a setting with two ground clusters to enforce the utilization of satellite links. As shown in Fig. 5a, the quality of the detector has a strong influence on the results. All approaches adapt well to the increasing efficiency and create more end-to-end entanglements. The jitter of the antenna has the opposite effect. As shown in Fig. 5b, high antenna jitter leads to a drastic drop in the performance of all approaches, highlighting the need for precise antenna control in satellite-assisted quantum networks. Zenith transmittance, as a measure of photon absorption in the atmosphere, also has a significant influence, with all approaches adapting to changing conditions. In all satellite scenarios, SatQNet adapts well to changing conditions and maintains a high EDR, outperforming all other approaches by a significant margin (Holm-corrected p < 0.001).
(a) Q-PATH
75%
100%
(b) SatQNet
1 1 Fig. 6: Connectivity of Klagenfurt to other European cities.
C. Application to real-world backbone networks In this section, we analyze the performance of SatQNet on real-world topologies. Even though SatQNet is trained on random topologies, applying it to real-world backbones without retraining provides a practical test of its adaptability to unseen graph structures. Fig. 6 shows the achieved endto-end entanglement rate from Klagenfurt (Austria) to major European cities when satellite assistance is enabled. For this evaluation, up to 10 entanglements are created between Klagenfurt and any other city in Europe, and the results are reported as a success rate. Note that performing episodes of 1000 steps as for the other experiments was not desirable due to the high number of European cities (> 500), so we used only 100 steps to reduce the computational load. Klagenfurt is marked with a black pentagon. The node colors indicate the percentage of successfully created end-to-end entanglements, ranging from 0% (white) to 100% (dark blue). We observe that SatQNet achieves a higher success rate than Q-PATH for 97% of destination cities. For the permutation test, we use the success rate to all destination cities as samples. When comparing the overall success rates between SatQNet and QPATH, with each city contributing one sample, the Europewide difference is statistically significant (p < 0.001). While both approaches perform similarly in parts of central Europe, their performance diverges with increasing distance, where
PREPRINT SUBMITTED TO THE IEEE
SatQNet shows clear gains, including in northern Germany, Denmark, France, and Turkey. These results highlight that SatQNet generalizes well beyond its training distribution and remains effective on realistic large-scale backbone topologies. VII. C ONCLUSION We have presented SatQNet, a decentralized reinforcement learning-based approach relying on a directed line GNN for entanglement routing in satellite-assisted quantum networks. Existing approaches often rely on global information about the quantum topology or assume a static physical topology, which limits their scalability and their ability to route efficiently in large-scale, highly dynamic quantum networks. SatQNet addresses this challenge by learning graph representations from local observations. In contrast to existing learning-based approaches that rely on node embeddings, SatQNet maintains edge embeddings at each node, allowing it to adapt dynamically to the varying connectivity of each repeater. The evaluation on random graphs shows that SatQNet significantly outperforms learning-based approaches relying on node observations in all evaluated settings. Additionally, SatQNet outperforms non-learning-based approaches in large networks, where the latency of collected global information becomes prohibitive. It also handles resource scarcity when multiple source-destination pairs compete for resources, and achieves clear gains when the end-to-end entanglement requires satellite-assisted paths. The advantage also persists under changes in satellite parameters, indicating that the learned policy adapts well to varying physical conditions. Beyond synthetic topologies, SatQNet generalizes to a realworld European backbone without retraining, where it achieves a higher success rate than Q-PATH for 97% of destination cities. Together, these results support the central claim of this work: decentralized, edge-centric learning is a scalable and effective approach to entanglement routing in highly dynamic satellite-assisted quantum networks. In future work, we plan to improve the resource efficiency of SatQNet by adaptively choosing the message size depending on the expected impact on the reinforcement learning agent and investigate strategies for asynchronous environments. ACKNOWLEDGMENT During the preparation of this manuscript, the authors used Generative AI to improve the spelling, grammar, and readability of the text. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the final version of the manuscript. R EFERENCES [1] Z. Li, K. Xue, J. Li, L. Chen et al., “Entanglement-Assisted Quantum Networks: Mechanics, Enabling Technologies, Challenges, and Research Directions,” Communications Surveys & Tutorials, vol. 25, no. 4, pp. 2133–2189, 2023. [2] A. Singh, K. Dev, H. Siljak, H. D. Joshi, and M. Magarini, “Quantum Internet—Applications, Functionalities, Enabling Technologies, Challenges, and Research Directions,” Communications Surveys & Tutorials, vol. 23, no. 4, pp. 2218–2247, 2021. [3] Z. Li, K. Xue, J. Li, N. Yu et al., “Building a large-scale and wide-area quantum Internet based on an OSI-alike model,” China Communications, vol. 18, no. 10, pp. 1–14, 2021.
12
[4] J. Luo, D. Wu, Q. Miao, C. Yang, and T. Wei, “Navigation ranging scheme based on microwave-optical entanglement prepared by electroopto-mechanical converters,” IEEE Photonics Journal, vol. 12, no. 2, pp. 1–15, 2020. [5] P. Kómár, E. M. Kessler, M. Bishof, L. Jiang et al., “A quantum network of clocks,” Nature Physics, vol. 10, no. 8, p. 582–587, Jun. 2014. [6] K. Azuma, S. Economou, D. Elkouss, P. Hilaire et al., “Quantum repeaters: From quantum networks to the quantum internet,” Reviews of Modern Physics, vol. 95, 12 2023. [7] N. Sandilya and A. K. Sharma, “Quantum internet: An approach towards global communication,” in International Conference on Reliability, Infocom Technologies and Optimization (ICRITO), 2021, pp. 1–5. [8] J. Illiano, M. Caleffi, A. Manzalini, and A. S. Cacciapuoti, “Quantum internet protocol stack: a comprehensive survey,” Computer Networks, p. 109092, 2022. [9] M. Chehimi and W. Saad, “Physics-informed quantum communication networks: A vision toward the quantum internet,” Network, vol. 36, no. 5, pp. 32–38, 2022. [10] M. Zukowski, A. Zeilinger, M. Horne, and A. Ekert, ““Event-readydetectors” Bell experiment via entanglement swapping,” Physical Review Letters, vol. 71, pp. 4287–4290, 01 1994. [11] S. Biswas, R. Bassoli, J. Nötzel, C. Deppe et al., “Quantum satellite communications,” in A Roadmap to Future Space Connectivity: Satellite and Interplanetary Networks. Springer, 2023, pp. 85–104. [12] S. Shi and C. Qian, “Concurrent Entanglement Routing for Quantum Networks: Model and Designs,” in SIGCOMM. ACM, 2020, p. 62–75. [13] Y. Zeng, J. Zhang, J. Liu, Z. Liu, and Y. Yang, “Multi-Entanglement Routing Design over Quantum Networks,” in International Conference on Computer Communications (INFOCOM). IEEE, 2022, pp. 510–519. [14] G. Avis, F. Rozpedek, and S. Wehner, “Analysis of Multipartite Entanglement Distribution Using a Central Quantum-Network Node,” Physical Review A, vol. 107, p. 012609, Jan. 2023. [15] A. Sen, K. Goodenough, and D. Towsley, “Multipartite Entanglement in Quantum Networks Using Subgraph Complementations,” in International Conference on Quantum Computing and Engineering (QCE). IEEE, Sep. 2023, pp. 252–253. [16] S. Pirandola, “Satellite Quantum Communications : Fundamental Bounds and Practical Security,” Physical Review Research, vol. 3, 05 2021. [17] A. Chang, Y. Wan, G. Xue, and A. Sen, “Entanglement Distribution in Satellite-Based Dynamic Quantum Networks,” Network, vol. 38, no. 1, pp. 79–86, 2024. [18] V. Krutyanskiy, M. Meraner, J. Schupp, V. Krcmarsky et al., “Lightmatter entanglement over 50 km of optical fibre,” npj Quantum Information, vol. 5, no. 1, Aug. 2019. [19] Y. Yu, F. Ma, X.-Y. Luo, B. Jing et al., “Entanglement of two quantum memories via fibres over dozens of kilometres,” Nature, vol. 578, no. 7794, p. 240–245, Feb. 2020. [20] G. Wang, D. Shen, G. Chen, K. Pham, and E. Blasch, “Polarization tracking for quantum satellite communications,” in Sensors and Systems for Space Applications VII, K. D. Pham and J. L. Cox, Eds., vol. 9085, International Society for Optics and Photonics. SPIE, 2014, p. 90850T. [21] G. Vallone, D. Bacco, D. Dequal, S. Gaiarin et al., “Experimental Satellite Quantum Communications,” Physical Review Letters, vol. 115, 06 2014. [22] C. Agnesi, L. Calderaro, D. Dequal, F. Vedovato et al., “Sub-ns timing accuracy for satellite quantum communications,” Journal of the Optical Society of America B, vol. 36, p. B59, 01 2019. [23] Z. Wang, R. Malaney, and R. Aguinaldo, “Temporal Modes of Light in Satellite-to-Earth Quantum Communications,” Communications Letters, vol. 26, pp. 311–315, 02 2022. [24] M. Aspelmeyer, T. Jennewein, M. Pfennigbauer, W. Leeb, and A. Zeilinger, “Long-distance quantum communication with entangled photons using satellites,” Journal of Selected Topics in Quantum Electronics, vol. 9, no. 6, p. 1263786, 2003. [25] J. Li, M. Wang, K. Xue, R. Li et al., “Fidelity-Guaranteed Entanglement Routing in Quantum Networks,” Transactions on Communications, vol. PP, pp. 1–1, 10 2022. [26] J. Xiong, Q. Zhang, A. Gatto, F. Musumeci et al., “Adaptive Entanglement Routing for Quantum Networks with Cutoff,” in International Conference on Network and Service Management (CNSM). IEEE, 2023, pp. 1–5. [27] K. Chakraborty, F. Rozpedek, A. Dahlberg, and S. Wehner, “Distributed Routing in a Quantum Internet,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11630
PREPRINT SUBMITTED TO THE IEEE
[28] L. Le and T. N. Nguyen, “DQRA: Deep Quantum Routing Agent for Entanglement Routing in Quantum Networks,” Transactions on Quantum Engineering, vol. 3, pp. 1–12, 2022. [29] D. Abreu and A. Abelém, “qRL: Reinforcement Learning Routing for Quantum Entanglement Networks,” in Symposium on Computers and Communications (ISCC). IEEE, 2024, pp. 1–6. [30] T. Meuser, J. Weil, A. Lahiri, and M. Paraschiv, “RELiQ: Scalable Entanglement Routing via Reinforcement Learning in Quantum Networks,” Transactions on Communications, pp. 1–16, 2025. [31] F. Geyer and G. Carle, “Learning and Generating Distributed Routing Protocols Using Graph-Based Deep Learning,” in Workshop on Big Data Analytics and Machine Learning for Data Communication Networks (Big-DAMA). ACM, 2018, p. 40–45. [32] J. Weil, Z. Bao, O. Abboud, and T. Meuser, “Towards Generalizability of Multi-Agent Reinforcement Learning in Graphs with Recurrent Message Passing,” in International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2024, p. 1919–1927. [33] L. Zhang, S.-X. Ye, Q. Liu, and H. Chen, “Multipath Concurrent Entanglement Routing in Quantum Networks Based on Virtual Circuit,” in International Conference on Advances in Computer Technology, Information Science and Communications (CTISC). IEEE, Apr. 2022, pp. 1–5. [34] F. Hahn, A. Pappa, and J. Eisert, “Quantum network routing and local complementation,” npj Quantum Information, vol. 5, p. 76, 2019. [35] Á. G. Iñesta, G. Vardoyan, L. Scavuzzo et al., “Optimal entanglement distribution policies in homogeneous repeater chains with cutoffs,” npj Quantum Information, vol. 9, p. 46, 2023. [36] V. Mannalath and A. Pathak, “Multiparty entanglement routing in quantum networks,” Physical Review A, vol. 108, 12 2023. [37] E. Sutcliffe and A. Beghelli, “Multi-User Entanglement Distribution in Quantum Networks Using Multipath Routing,” Transactions on Quantum Engineering, vol. PP, pp. 1–16, 01 2023. [38] M. Pant, H. Krovi, D. Towsley, L. Tassiulas et al., “Routing Entanglement in the Quantum Internet,” npj Quantum Information, vol. 5, no. 1, pp. 1–9, Mar. 2019. [39] T. N. Nguyen, K. J. Ambarani, L. Le, I. Djordjevic, and Z.-L. Zhang, “A Multiple-Entanglement Routing Framework for Quantum Networks,” Jul. 2022. [Online]. Available: http://arxiv.org/abs/2207.11817 [40] C. Li, T. Li, Y.-X. Liu, and P. Cappellaro, “Effective Routing Design for Remote Entanglement Generation on Quantum Networks,” npj Quantum Information, vol. 7, no. 1, pp. 1–12, Jan. 2021. [41] M. Victora, S. Tserkis, S. Krastanov, A. S. de la Cerda et al., “Entanglement Purification on Quantum Networks,” Physical Review Research, vol. 5, p. 033171, Sep. 2023. [42] E. A. Van Milligen, E. Jacobson, A. Patil, G. Vardoyan et al., “Entanglement routing over networks with time multiplexed repeaters,” in International Conference on Quantum Computing and Engineering (QCE), vol. 01. IEEE, 2025, pp. 1170–1178. [43] S. Pouryousef, N. K. Panigrahy, and D. Towsley, “A Quantum Overlay Network for Efficient Entanglement Distribution,” in International Conference on Computer Communications (INFOCOM). IEEE, 2023, pp. 1–10. [44] E. Schoute, L. Mancinska, T. Islam, I. Kerenidis, and S. Wehner, “Shortcuts to Quantum Network Routing,” Oct. 2016. [Online]. Available: http://arxiv.org/abs/1610.05238 [45] J. C. Adcock, S. Morley-Short, A. Dahlberg, and J. W. Silverstone, “Mapping Graph State Orbits Under Local Complementation,” Quantum, vol. 4, p. 305, Aug. 2020. [46] W. Dai, T. Peng, and M. Z. Win, “Optimal Remote Entanglement Distribution,” Journal on Selected Areas in Communications, vol. 38, no. 3, pp. 540–556, 2020. [47] H. Gu, Z. Li, R. Yu, X. Wang et al., “Fendi: Toward high-fidelity entanglement distribution in the quantum internet,” IEEE/ACM Transactions on Networking, vol. 32, no. 6, pp. 5033–5048, 2024. [48] L. De Forges de Parny, O. Alibart, J. Debaud, S. Gressani et al., “Satellite-based quantum information networks: use cases, architecture, and roadmap,” Communications Physics, vol. 6, p. 12, 01 2023. [49] C. Agnesi, F. Vedovato, M. Schiavon, D. Dequal et al., “Exploring the boundaries of quantum mechanics: advances in satellite quantum communications,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 376, no. 2123, p. 20170461, 05 2018. [50] H. Gu, R. Yu, Z. Li, X. Wang, and G. Xue, “QuESat: Satellite-Assisted Quantum Internet for Global-Scale Entanglement Distribution,” in International Conference on Computer Communications (INFOCOM). IEEE, 2025, pp. 1–10.
13
[51] X. Wei, J. Liu, L. Fan, Y. Guo et al., “Optimal Entanglement Distribution Problem in Satellite-Based Quantum Networks,” Network, vol. 39, no. 1, pp. 97–103, 2025. [52] D. Mishra, P. Rewal, and K. Pursharthi, “Efficient and quantum-secure authenticated key exchange scheme for mobile satellite communication networks,” International Journal of Satellite Communications and Networking, vol. 42, 04 2024. [53] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: a reinforcement learning approach,” in Advances in Neural Information Processing Systems (NeurIPS). Morgan Kaufmann Publishers Inc., 1993, p. 671–678. [54] P. Almasan, J. Suárez-Varela, K. Rusek, P. Barlet-Ros, and A. CabellosAparicio, “Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case,” Computer Communications, vol. 196, pp. 184–194, 2022. [55] S. Schneider, H. Qarawlus, and H. Karl, “Distributed Online Service Coordination Using Deep Reinforcement Learning,” in International Conference on Distributed Computing Systems (ICDCS). IEEE, 2021, pp. 539–549. [56] K. Rusek, J. Suárez-Varela, P. Almasan, P. Barlet-Ros, and A. CabellosAparicio, “RouteNet: Leveraging Graph Neural Networks for Network Modeling and Optimization in SDN,” Journal on Selected Areas in Communications, vol. 38, no. 10, pp. 2260–2270, 2020. [57] M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information, 10th ed. Cambridge University Press, 2010. [58] H.-P. Breuer and F. Petruccione, The Theory of Open Quantum Systems. Oxford University Press, 2002. [59] P.-J. Stas, Y. Q. Huan, B. Machielse, E. N. Knall et al., “Robust multiqubit quantum network node with integrated error detection,” Science, vol. 378, no. 6619, pp. 557–560, 2022. [60] J.-W. Pan, D. Bouwmeester, H. Weinfurter, and A. Zeilinger, “Experimental Entanglement Swapping: Entangling Photons That Never Interacted,” Physical Review Letters, vol. 80, no. 18, pp. 3891–3894, 1998. [61] Q.-C. Sun, Y.-F. Jiang, Y.-L. Mao, L. You et al., “Entanglement swapping over 100 km optical fiber with independent entangled photonpair sources,” Optica, vol. 4, p. 1214, 10 2017. [62] A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood et al., “Quantum computing with Qiskit,” 2024. [Online]. Available: https: //arxiv.org/abs/2405.08810 [63] S.-K. Liao, W.-Q. Cai, W.-Y. Liu, L. Zhang et al., “Satellite-to-ground quantum key distribution,” Nature, vol. 549, pp. 43–47, 2017. [64] C.-Y. Lu, Y. Cao, C.-Z. Peng, and J.-W. Pan, “Micius quantum experiments in space,” Reviews of Modern Physics, vol. 94, p. 035001, 2022. [65] H. Kaushal and G. Kaddoum, “Optical communication in space: challenges and mitigation techniques,” Communications Surveys & Tutorials, vol. 19, no. 1, pp. 57–96, 2017. [66] A. A. Farid and S. Hranilovic, “Outage capacity optimization for free-space optical links with pointing errors,” Journal of Lightwave Technology, vol. 25, no. 7, pp. 1702–1710, 2007. [67] S. Khatri, A. J. Brady, R. A. Desporte, M. P. Bart, and J. P. Dowling, “Spooky action at a global distance: analysis of space-based entanglement distribution for the quantum internet,” npj Quantum Information, vol. 7, p. 4, 2021. [68] L. Cai, J. Li, J. Wang, and S. Ji, “Line Graph Neural Networks for Link Prediction,” Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021. [69] L. Müller, D. Kusuma, B. Bonet, and C. Morris, “Towards principled graph transformers,” in Advances in Neural Information Processing Systems (NeurIPS). Curran Associates Inc., 2024. [70] S. H. Pahng and S. Hormoz, “Improving Graph Neural Networks by Learning Continuous Edge Directions,” in International Conference on Learning Representations (ICLR). OpenReview, 2025. [71] M. Gaon and R. Brafman, “Reinforcement Learning with NonMarkovian Rewards,” AAAI Conference on Artificial Intelligence, vol. 34, no. 04, pp. 3980–3987, Apr. 2020. [72] X. Lin and X. Zhang, “Efficient Reinforcement Learning in Probabilistic Reward Machines,” AAAI Conference on Artificial Intelligence, vol. 39, no. 18, pp. 18 728–18 736, Apr. 2025. [73] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015. [74] M. Ojala and G. C. Garriga, “Permutation Tests for Studying Classifier Performance,” Journal of Machine Learning Research, vol. 11, no. 62, pp. 1833–1863, 2010. [75] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2009.