Scalable Multi-Controller Coordination in Periplus via Border-Switch Forwarding Graphs E. M. Castro Barbero1 , P. de las Heras Quirós1 , F. J. Simó Reigadas1 1 Universidad Rey Juan Carlos
arXiv:2607.19508v1 [cs.NI] 21 Jul 2026
July 2026 Abstract In-band SDN control planes, where control traffic shares the data-plane infrastructure, suit wide-area, resource-constrained deployments—such as rural backbones—that cannot afford a dedicated control network. Partitioning such a network across multiple controllers improves scalability but raises a coordination challenge that in-band designs have largely ignored: controllers must discover one another and exchange state in-band, and switches must recover when their controller fails, all without forwarding state that grows with the number of controllers. This paper presents the multi-controller coordination plane of Periplus, an in-band control plane whose single-controller design is developed in a companion paper. Periplus controllers discover their neighbors through Controller Advertisement (C-Adv) messages and build inter-controller routes incrementally: each border switch inserts a partial forwarding graph covering only the next domain, so per-controller forwarding state is confined to border switches and never distributed across the interior of an intermediate domain. The same C-Adv mechanism reattaches a switch to a surviving controller after a controller failure. We evaluate a Ryu-based implementation in Mininet, including a 96switch, 5-controller scenario. Per-switch flow-table state is set by a switch’s role rather than by network size—interior occupancy stays constant as controllers are added—partitioning scales bootstrap to networks of around a hundred switches, and inter-controller discovery converges within seconds. The design needs no switch-firmware modifications: it runs on stock Open vSwitch, using only its built-in Nicira extensions for Network Service Header (NSH) encapsulation.
1
Introduction
Despite significant advances in telecommunications infrastructure, many rural and underserved regions still lack reliable Internet connectivity [13]. These regions combine low population density, limited income, a shortage of local 1
technical personnel, and scarce complementary power and communications infrastructure. Their service requirements, however, are often similar to those of dense urban areas, so they call for technologies that are simultaneously low-cost, robust, flexible, and easy to deploy and operate. Software-defined networking (SDN) [11] can bring clear benefits to these networks, but only if some of their specific constraints are met. Because they lack a parallel infrastructure for control or signaling, the control plane must run in-band, sharing the data-plane links it manages [3,9]. SDN’s centralized control simplifies operation, yet it must be reinforced with robustness and resilience mechanisms so that the network keeps serving users despite the unstable links and frequent topology changes typical of these environments, and continues to operate with little to no human intervention. An in-band control plane must overcome four challenges that a dedicated control network would not face. First, a switch must attach to the control plane before any forwarding rule exists to carry its own control traffic (bootstrapping). Second, once attached, its control messages must travel over efficient paths that keep per-switch state low (routing). Third, switches must survive link and node failures without waiting for the controller to intervene (failure recovery). Fourth, once the network is large enough to require several controllers, those controllers must stay coordinated over the same in-band substrate they manage (multi-controller coordination). Earlier proposals for these environments strike different balances: some prioritize adaptability, low cost, and quality at the expense of high complexity [14], while others trade some quality for easier deployment and operation [16]. Periplus was recently proposed as an SDN architecture tailored to these settings. Its single-controller design, presented in [4], addresses the first three challenges: a bootstrap that installs flow rules in only two switches when a switch joins, graph-based source routing with minimal per-switch state, and sub-50 ms failure recovery performed locally at the switch, combining prompt link-failure detection with forwarding alternatives precomputed for each hop. That design, however, assumes a single controller and does not address the fourth challenge. The present paper builds on Periplus to tackle this fourth and least-explored challenge: coordinating multiple controllers over the in-band network. Partitioning a wide-area network across several controllers improves scalability and keeps control traffic local, but coordinating those controllers in-band is difficult. Controllers must discover one another and exchange switch state over the same network they manage, and a switch must keep reaching a controller when its own fails, all without flooding the network or storing forwarding state that grows with the number of controllers. Existing in-band designs that admit multiple controllers (Section 2) give each controller its own spanning tree or a set of disjoint control paths, so per-switch forwarding state scales with the controller count, eroding the scalability that motivated partitioning in the first place. This paper contributes a multi-controller coordination mechanism whose per-controller forwarding state is confined to border switches. Periplus controllers discover their neighbors through Controller Advertisement (C-Adv) mes2
sages and establish inter-controller routes across domains incrementally: each border switch inserts a partial forwarding graph covering only the next domain, so the routing state needed to reach a remote controller is never distributed across the interior switches of any intermediate domain. The same C-Adv substrate lets a switch recover after a controller failure by re-attaching through a neighboring domain. Because coordination state is confined to domain boundaries, the architecture remains scalable as the number of controllers and switches increases. Periplus is implemented in Python on the Ryu SDN framework [29, 30] and evaluated through network emulation in Mininet across several multi-controller topologies, including a large-scale scenario with 96 switches and 5 controllers. The results confirm that per-switch forwarding state stays bounded: interior switches hold a constant number of rules regardless of the number of controllers, and additional state appears only at border switches. The remainder of this paper is organized as follows. Section 2 reviews related work on multi-controller coordination in in-band SDN. Section 3 recaps the single-controller design of Periplus that the multi-controller plane builds on. Section 4 presents the multi-controller coordination mechanism, covering inter-controller discovery, communication, and switch recovery after a controller failure. Section 5 reports an experimental evaluation in Mininet across multicontroller topologies, including the 96-switch, 5-controller scenario. Finally, Section 6 concludes the paper.
2
Related Work
This paper extends Periplus, an in-band SDN control plane, to operate across multiple controllers. The single-controller design of Periplus (automatic bootstrapping, graph-based source routing, and sub-50 ms failure recovery) and the related work on those three challenges are presented in [4]. This section concentrates on the fourth and least-explored challenge of in-band SDN control planes: coordination among multiple controllers. For context, we briefly situate the three single-controller challenges. Bootstrapping proposals such as Medieval and Renaissance [2, 23], Sakic et al. [22], FASIC [28], and Wong and Lee [31] achieve plug-and-play attachment but at the cost of network-wide flooding or per-switch state that grows with network size. For routing, source-based approaches such as Amaru [12] reduce forwarding state yet cannot guarantee controller reachability after a failure. For failure recovery, protection schemes such as those of Sharma et al. [25–27] reach sub50 ms recovery but preinstall backup paths on every switch and rely on STP for bootstrap. Multi-controller coordination. Multi-controller coordination has received comparatively little attention in the in-band control plane literature. Medieval and Renaissance support multiple controllers by having each controller build an additional spanning tree covering all switches in the network, which imposes per-controller forwarding state on every switch. Sakic et al. [22] deploy multi-
3
ple controller replicas synchronized via RAFT consensus, provisioning disjoint control paths per switch-controller pair; per-switch forwarding state likewise grows with the number of controllers. Holzmann et al. [8] extend Izzy to elastic controller clusters via the Seedling algorithm, which divides controllers into proximity-based groups and assigns a separate Izzy spanning tree per group; however, per-switch forwarding state still scales with the number of controllers. Chan et al. [5] target fast failure recovery in in-band multi-controller OpenFlow networks: a Main Controller and several Standby Controllers collaborate to detect failures through monitoring cycles and reroute traffic by installing new flow entries, provisioning K disjoint control paths per switch (one from each controller) to guarantee reachability after any single device failure, with average recovery times below 50 ms. Unlike Periplus, recovery there requires explicit coordination between controllers and is not performed locally at the switch. Across these proposals, none jointly addresses fast inter-controller communication and bounded per-switch state overhead in an in-band setting. Schiff et al. [24] explore an orthogonal approach to controller coordination: using part of the data-plane configuration space as transactional shared memory, implementing atomic compare-and-swap operations via OpenFlow bundling to synchronize distributed controllers. Panda et al. [19] argue that strong consensus protocols are conceptually inappropriate for network state in distributed SDN controllers, since the physical network is inherently uncertain; they propose eventual correctness instead. Periplus is aligned with this view: it provides the inter-controller communication infrastructure without imposing any consistency model, leaving coordination semantics to the applications above. Multi-controller coordination thus remains the least explored challenge of in-band SDN control planes, and existing solutions impose forwarding state proportional to the number of controllers on every switch. Periplus departs from this pattern: it uses Controller Advertisement messages and border switches to confine per-controller forwarding state to the switches at domain boundaries, and it provides the inter-controller communication infrastructure without mandating any consistency model. The following sections describe this mechanism and evaluate how it scales Periplus across multiple controllers.
3
Background: Single-Controller Periplus
This section recaps the single-controller design of Periplus to the extent needed to follow its multi-controller extension; the full design, with its protocols, tables, and message-overhead analysis, is presented in [4]. Periplus rests on three core design choices for the single-controller case: anchor-relayed bootstrap, graphbased source routing, and packet-encoded failure recovery. A fourth mechanism, Controller Advertisement (C-Adv) messages, serves as a cross-cutting discovery substrate. Multi-controller operation (Section 4) extends these choices through C-Adv messages and border switches. We assume that node communication interfaces, wired or wireless, are based
4
on IEEE 802 link technologies, so that each interface is identified by a unique 48-bit MAC address, and that links are bidirectional. Switches are denoted si with cj reserved for the OVS switch co-located with the controller (the root switch).
3.1
Bootstrap
Periplus bootstraps the control plane incrementally: switches attach one by one, each relying on a neighboring managed switch to relay its connection requests. Before it can connect, an unmanaged switch must learn the port toward the controller, which it caches in a soft-state register (reg9) with a short hard timeout. It learns this port from whichever controller-attributable signal arrives first: a proxy-ARP reply from a managed neighbor (the anchor) answering on the controller’s behalf, or a Controller Advertisement re-forwarded by a managed neighbor (Section 3.4). Both can only reach an unmanaged switch from the controller side, so the ingress port on which they arrive is the upstream port. The two triggers cover complementary cases: the ARP reply handles a cold start, and the C-Adv handles a warm restart, when a previously managed switch still caches the controller’s MAC address, issues no ARP request, and can relearn its upstream port only from a neighbor’s C-Adv. The switch sends its TCP SYN through the learned port; if none has been learned the SYN is dropped, and if the signals stop the cached entry expires and the switch reverts to the unmanaged state. The controller identifies the new switch from the Ethernet source address of its SYN and installs the forwarding flows that place it in the managed state, updating only two switches rather than every switch on the path to the controller; the full preconfigured rule set is given in [4]. In a multi-controller deployment this anchor relay operates within each controller domain, and border switches extend it across domain boundaries (Section 4).
3.2
Forwarding based on graphs
Periplus routes control traffic with the Slick Packets approach [15]: a forwarding graph, an acyclic subgraph of the topology embedded in the packet headers between the L2 and L3 layers using NSH [20] (realized with OVS’s Nicira extensions [18]), encodes a primary path P and, at each hop i, an ordered list of up to Ki alternative paths as output-port sequences. The controller computes P with Dijkstra and derives source-disjoint alternatives by iteratively pruning and re-running it. A switch that cannot forward through its primary port falls over locally to the first reachable alternative, with no controller involvement. Only the root switch co-located with the controller maintains routing state for all switches in its domain; every other switch stores only the path to its own controller. This domain-local property is what the multi-controller plane builds on: inter-domain routes are assembled from partial graphs inserted by border switches, so no interior switch ever holds routing state for a remote controller (Section 4).
5
3.3
Monitoring ports
Each switch monitors its links with two complementary detectors: Bidirectional Forwarding Detection (BFD) [10], used together with OpenFlow select group tables [17], and Input Traffic Detection (ITD), based on OpenFlow Echo Request/Reply messages and learning flows. On detecting a link or node failure, a switch immediately reroutes along an encoded alternative from the packet’s graph, meeting carrier-grade (sub-50 ms) recovery without contacting the controller.
3.4
Link Discovery Protocol
Once switches are managed, the controller knows the tree of links that connect it to them, but not the redundant links that can serve as alternatives. Periplus discovers these through Controller Advertisement (C-Adv) messages. The controller periodically issues a single Packet-Out to the root switch, which floods it; each switch floods a C-Adv only on first reception, suppressing duplicates with a short-lived learning flow whose hard timeout is shorter than the C-Adv period, and reports the reception to the controller with a Packet-In. Each managed switch encodes its own identity and egress port into the header of every C-Adv it forwards, so the controller can identify both endpoints of each discovered link. This controlled flooding avoids broadcast storms and reduces the controller to a single Packet-Out per discovery round; its message-overhead advantage over standard link-discovery protocols is analyzed in [4]. C-Adv is the substrate the multi-controller plane reuses: it underpins both inter-controller discovery and the recovery of switches after a controller failure (Section 4).
4
Multiple Controllers
Periplus is designed to support multiple controller instances. Each controller manages a disjoint set of switches, known as its domain; a switch with a link into another controller’s domain is a border switch, through which inter-controller traffic transits from one domain to the next. A switch is assigned to a controller as a direct consequence of the bootstrap mechanism of Section 3.1: the nearest managed neighbor relays the booting switch’s TCP SYN toward its own controller, yielding a first-come, first-served (FCFS) assignment to typically the nearest controller. Disjointness is enforced implicitly, since once the routing flows are installed the switch’s control traffic is directed exclusively through its anchor to a single controller; if the switch later reboots, it re-runs the bootstrap procedure and may attach to a different controller if the topology has changed. This assignment rests on how controllers are addressed. All controllers share a single anycast address, so an unmanaged switch attaches to whichever controller its bootstrap signals reach first—typically the nearest—without needing to know which controller that is. In addition, each controller owns a distinct unicast address that identifies it to the other controllers; the inter-controller discovery and communication mechanisms described below rely on these addresses 6
to tell controllers apart and route between them. These unicast addresses are used only among controllers: a switch never needs to know the unicast address of any controller, and reaches the control plane solely through the shared anycast address. This section describes how controllers discover and communicate with neighbor controllers, how this mechanism is extended to non-neighbor controllers, and how switches recover from a controller failure.
4.1
Communication between neighbor controllers
C-Adv messages are used to discover other controllers. The propagation of C-Adv messages follows the controlled flooding mechanism described in Section 3.4. Since multiple controllers coexist within the same network, when a switch forwards a C-Adv through an interface connected to a switch managed by another controller, the receiving switch generates a Packet-In message to its own controller and does not forward the C-Adv further. This mechanism enables a controller to detect the presence of another controller in the network. Figure 1 illustrates a network with two controllers. Switches managed by controller 1 are shown in yellow, while those managed by controller 2 are shown in blue. As depicted in Fig. 1a, switch s1 receives a C-Adv message generated by controller 2 and generates a Packet-In to controller 1. Upon receiving this message, controller 1 becomes aware of controller 2’s presence, and then controller 1 installs a new flow in c1 (the root switch of controller 1) to reach controller 2 via s1. This flow instructs c1 to embed in packets destined for c2 a forwarding graph whose primary path terminates at port 2 of s1. Figure 1b shows how packets from controller 1 to controller 2 are forwarded through controller 1’s domain using the embedded graph until reaching s1. Between s1 and s2, packets do not carry the graph; however, s2 is a border switch with a flow installed to embed the forwarding graph for its own domain, and therefore inserts the graph that enables the packet to reach controller 2. Analogously, when s2 receives a C-Adv from controller 1 and generates a Packet-In to its own controller, controller 2 installs a new flow rule in c2 to reach controller 1.
4.2
Communication between non-neighbor controllers
The mechanism described above operates directly when controller domains are neighbors. It generalizes to non-neighboring controllers through Algorithm 1. Each C-Adv carries a list of all controllers known to the sender, each entry accompanied by its learnt from path vector. Upon reception, the local controller processes the sending controller as a direct neighbor, then iterates over the payload to discover non-neighboring controllers. Three conditions guard each entry: the controller’s own IP is skipped; a direct neighbor cannot be downgraded to non-neighbor status; and any entry whose learnt from list already contains the local controller’s IP is discarded. This last condition prevents routing loops by the same principle as the AS PATH attribute in BGP [21]: just as a BGP 7
Algorithm 1 Inter-Controller Discovery (at controller c) 1: State: known(c): set of entries (cx , is neighbor , learnt from, alive), where: 2: is neighbor : true if cx discovered directly via Packet-In 3: learnt from: path vector (controllers that reported cx ) 4: alive: liveness counter, reset to T on each C-Adv refresh 5: upon C-Adv from cn carrying P = {(cx , learnt from x )} do 6: add or refresh cn in known(c): 7: is neighbor ← true, learnt from ← [cn ], alive ← T 8: update ARP cache for cn 9: install forwarding graph to reach cn 10: for each (cx , learnt from x ) ∈ P do 11: ▷ skip ourselves 12: if cx = c: continue 13: ▷ neighbor takes precedence 14: if is neighbor x = true: continue 15: ▷ loop detected 16: if c ∈ learnt from x : continue 17: add or refresh cx in known(c): 18: is neighbor ← false, 19: learnt from ← learnt from x + [cn ], 20: alive ← T 21: update ARP cache for cx 22: install forwarding graph to reach cx 23: upon periodic timer do 24: for each cx ∈ known(c): decrement alive x 25: if alive x = 0: remove cx from known(c) and its flows 26: refresh forwarding graph to each surviving cx ∈ known(c) 27: install border routes between all neighbor controller pairs 28: ▷ see state distribution analysis below 29: C-Adv sent by c carries {(cx , learnt from x )} for all cx ∈ known(c)
8
(a) s1 receives C-Adv from controller 2.
(b) controller 1 sends a message to controller 2. Figure 1: Inter-controller discovery via C-Adv on a two-controller topology: (a) s1 receives a C-Adv from controller 2 and reports the link to controller 1; (b) controller 1 forwards a packet to controller 2 using the embedded forwarding graph. router rejects a route whose AS PATH already contains its own AS number, a controller rejects an entry that has already passed through it. Forwarding graphs are computed using Dijkstra’s algorithm on the local topology graph: the primary path is the shortest path within the local domain to the border switch adjacent to the destination controller’s domain, and for each hop, an alternative path is computed where one exists. These flows are installed when a new peer controller is discovered, following the same soft-state principle as the alive counter. This locality has a direct consequence for state distribution. Non-border switches require no state beyond the path to their own controller—the same property established in Section 3.2 for the single-controller case. Additional state is installed only at border switches that serve as entry points for transit between pairs of neighbor controllers (Algorithm 1, line 27): each such switch stores a single forwarding graph per transit destination, encoding only the short, local path within its own domain from the entry border to the exit border switch facing the next domain. This guarantees transit without distributing state across the domain interior. Liveness is maintained through a soft-state alive counter, following the same
9
design principle as RSVP [1]: entries are periodically refreshed by incoming CAdv messages and expire silently when refreshes stop. When a controller fails, its C-Adv messages cease, all entries referencing it are no longer refreshed, and they eventually expire across the network. Fig. 2 illustrates Algorithm 1 for a three-controller topology. Controller 1 is a direct neighbor of both controller 2 and controller 3. It advertises both in its C-Adv. Controllers 2 and 3 learn of each other from controller 1’s payload and install the required flows; the loop-detection condition prevents each from re-accepting the other’s entry via controller 1.
Figure 2: Three-controller topology where controllers 2 and 3 are non-neighbors and discover each other via controller 1’s C-Adv advertisements. Fig. 3 uses the same three-controller topology as Fig. 2, showing the forwarding state each controller holds after discovery completes; in each subfigure, switches belonging to other domains are collapsed into black segments. As shown in Fig. 3a and Fig. 3b, controllers 2 and 3 each install a new flow in their root switches (c2 and c3) to embed the connectivity graph required to reach the newly discovered non-neighboring controller. Controller 1 also installs new flows in its border domain switches: as illustrated in Fig. 3c, a new flow in switch s12 to reach controller 3, and new flows in switches s23 and s22 to reach controller 2. This mechanism resembles a segment-routing approach [7], but the forwarding graph is installed incrementally rather than at the ingress: each border switch adds the segment crossing its own domain. For example, a message from controller 2 to controller 3 in the topology of Fig. 2 carries P = [2, 1] from c2 to border switch s12, then P = [3, 1] across controller 1’s domain to the exit border switch s33, where the local segment within controller 3’s domain is added. The full end-to-end path is never encoded at any single point, which scales more effectively than ingress-only encoding across multiple domains containing large numbers of switches.
10
(a) New flow rule in c2 to reach controller 3.
(b) New flow rule in c3 to reach controller 2.
(c) New flow rules in border domain switches to reach controllers. Figure 3: Non-neighbor controller discovery (same topology as Fig. 2); switches in other domains are collapsed into black segments. 11
4.3
Controller Failure Recovery
Unlike the link and node failures of Section 3.3, which a switch survives locally by rerouting to an alternative path toward the same controller, a controller failure leaves the switch unable to reach its controller at all, so it must attach to a surviving one. The affected switches recover through the event-driven procedure of Algorithm 2, which reuses the preconfigured bootstrap rules of Section 3.1. Algorithm 2 Controller Failure Recovery (at switch s) 1: upon controller-installed routing flows expiring do 2: s reverts to preconfigured initial flow rules 3: upon C-Adv from neighboring managed switch do 4: reg9 ← ingress port of C-Adv 5: send TCP SYN via reg9 6: upon reg9 = 0 and ARP cache empty do 7: broadcast ARP request through all ports 8: upon ARP reply received do 9: reg9 ← ingress port of ARP reply 10: send TCP SYN via reg9 11: upon TCP session established with new controller do 12: complete OpenFlow handshake 13: new controller installs routing flows 14: s transitions to managed state under new domain
The controller installs the routing flows with a hard timeout and refreshes them periodically. Recovery therefore requires no explicit failure detection on the switch’s part: it is triggered implicitly when the controller fails and stops refreshing these flows, so they expire and return the switch to the unmanaged state of Section 3.1 with no upstream port cached (reg9 = 0). The two learning signals that drive initial attachment then reattach the switch, now to a surviving controller. As at bootstrap, the C-Adv path takes priority: a Controller Advertisement from a managed neighbor sets reg9 and suppresses the ARP path, which activates only when no C-Adv has arrived and the ARP cache holds no controller entry (the warm-restart case of Section 3.1). Both paths converge at the TCP SYN, re-running the attachment of Section 3.1 against the surviving controller, which installs routing flows and brings the switch into its domain.
5
Evaluation
This section evaluates the multi-controller plane of Periplus along the two properties its design promises (Section 1): that per-controller forwarding state stays confined to border switches, and that partitioning a network across controllers scales the control plane to networks of around a hundred switches, with each
12
controller managing only a small domain. We first measure how bootstrap time scales as a network is partitioned across an increasing number of controllers (Section 5.1). We then evaluate the coordination mechanism in action: the time for controllers to discover one another (Section 5.2) and the time for switches to recover after a controller fails (Section 5.3). Finally, we measure the perswitch flow-table footprint as a function of each switch’s role, confirming that coordination state is bounded at domain boundaries (Section 5.4). The experimental setup follows that of [4]. Topologies are emulated with Mininet 2.3 [6], switches are OVS 2.15, and Periplus runs on the Ryu 4.32 SDN framework, on an Ubuntu 24.04 system with an AMD Ryzen 7 4800H CPU and 64 GB of RAM. As in the single-controller evaluation, each OVS instance runs in its own network namespace and in secure mode (fail-mode=secure), so that a switch cannot reach any controller until a neighbouring switch is itself managed, reproducing a pure in-band control plane. Unless otherwise noted, each experiment is repeated twenty times.
Figure 4: Multi-controller topology variants: (i) 2c Linear, (ii) 2c Simple, (iii) 4c Mesh, (iv) 2c B4, and (v) 2c Clos. The prefix indicates the number of controllers. We evaluate the multi-controller variants of Fig. 4, derived from the singlecontroller topologies of [4] by placing controllers at selected switches. Two additional Mesh variants (2c Mesh and 3c Mesh) are obtained from the 4c Mesh
13
Figure 5: Large-scale network topology (Large) with 96 switches and 5 controllers placed at geographically distributed positions. topology by removing controllers. The large-scale topology of Fig. 5 (Large, 96 switches) is tested with 2, 3, 4, and 5 controllers.
5.1
Bootstrap scaling across controllers
Periplus bootstraps incrementally, so a switch can begin attaching only once a neighbouring switch is managed: bootstrap time is governed by the diameter of the region a controller must reach, not by the total number of switches [4]. Partitioning a network across several controllers exploits this directly, since each controller bootstraps only its own domain. Adding controllers shrinks the percontroller diameter and shortens the critical path, letting the control plane scale to networks far larger than a single controller could bootstrap quickly. This scaling is what the bounded per-controller state of Section 5.4 pays for: because coordination state stays at domain boundaries, partitioning adds controllers without inflating per-switch state. We measure the wall-clock bootstrap time, the interval from the first TCP SYN received at any controller to the moment the last switch is marked managed across all controller logs, on the Large topology of Fig. 5 (96 switches) partitioned across 2, 3, 4, and 5 controllers. We use twenty repetitions per configuration on the patched OVS build (Section 5.3 describes the patch). Fig. 6 shows the result. The median wall-clock falls from 14.8 s with two controllers to 10.2 s with three, and then flattens (9.7 s at four, 10.4 s at five): once the per-controller diameter reaches its minimum, adding controllers yields no further gain. A 96-switch network thus bootstraps in around ten seconds once it is partitioned into three or more domains, confirming that partitioning keeps each controller’s domain small—a dozen switches here—while the network as a whole scales to nearly a hundred. The within-configuration spread reflects the first-come-first-served variability in how many switches each controller ends
14
Wall-clock bootstrap (s)
16 12 8 4 0
2c
3c
4c
5c
Figure 6: Wall-clock bootstrap for the Large topology (96 switches) with 2, 3, 4, and 5 controllers. up managing (Section 4).
5.2
Inter-controller discovery time
In a multi-controller deployment, no inter-controller coordination can begin until every controller has discovered every other controller. The convergence time of this initial discovery phase is therefore the floor on how quickly the multicontroller plane becomes fully operational. The C-Adv-based mechanism of Section 4 performs this discovery implicitly: each controller learns about a peer the first time one of its managed switches forwards a C-Adv originated by that peer. We measure how long that learning takes in practice. We run three multi-controller scenarios (2c Simple, 4c Mesh, and the 5controller large-scale topology), and for each run we record the interval between the moment the last controller is launched and the latest “first-time-learned” event across all controllers. We use twenty repetitions per scenario. Fig. 7 shows the distributions. Patched-build medians are 4.2 s (2c Simple), 5.9 s (4c Mesh), and 3.6 s (5c Large); stock-build medians are 10.2 s, 11.8 s, and 8.0 s, 2.0–2.4× slower because the cascade-bound bootstrap of Section 5.1 delays the first C-Adv from reaching a peer’s domain. The 5c Large topology discovers faster than 4c Mesh despite having more controllers, because its richer connectivity offers more paths for C-Adv propagation. Discovery cost depends on the inter-domain diameter, not on the number of alternative paths K: each controller needs only one C-Adv per peer, and the number of C-Adv messages per beacon period stays constant per managed switch. Each C-Adv’s payload does grow with the number of controllers—it carries one entry, a controller and its path vector, per known controller—but for the handful of controllers these deployments use, this list fits comfortably within a single packet, so the added overhead is negligible.
5.3
Controller failure recovery
When a controller fails, the switches it managed are momentarily orphaned and must rejoin the control plane through a surviving controller. The controller15
Inter-controller discovery time (s)
35 stock OVS patched OVS
30 25 20 15 10 5 0
2c Simple
4c Mesh
5c Large
Topology
Figure 7: Inter-controller discovery time, measured from the launch of the last controller until every controller knows about every other controller. Twenty repetitions per scenario. failure recovery procedure of Section 4.3 keeps this outage short: a switch whose flow entries expire reverts to its preconfigured initial rules, learns its new upstream port from the next C-Adv that arrives, and re-runs the bootstrap procedure against a surviving controller. We quantify how long that sequence takes end to end. Data-plane recovery from link and switch failures under a single controller is evaluated in [4]; here the failure is the controller itself. The experiment uses the 4c Mesh topology. The controller colocated in c4 is killed, and the four switches it managed (s41–s44) must reattach via C-Adv to a surviving controller (colocated in c1, c2, or c3), see Fig. 8. Because s42, s43, and s44 reach a surviving controller only through s41 or each other, recovery proceeds as a daisy chain in which each switch completes its own OpenFlow handshake only after its upstream neighbour on the new path is itself managed. The metric is the wall-clock recovery time: the interval between the kill of the controller colocated in c4 and the moment the last affected switch is marked managed on its new controller. We use twenty repetitions per OVS variant. Fig. 9 shows a median recovery of 10.2 s on the patched OVS build: this is the Periplus reattach cost, the time to install the flows for the four-switch daisy chain plus the BFD and C-Adv detection latency. The default build is roughly three times slower (median 35.5 s), but the gap is entirely OVSside. Stock OVS keeps a switch convinced its dead controller is still alive for several seconds, through hardcoded floors on its inactivity-probe and reconnectbackoff timers and, dominant here, a definition of connection “activity” that 16
Figure 8: 4c Mesh topology where the controller colocated in c4 is killed. counts queued-but-unacknowledged messages as peer liveness. The optional rconn patch, described in [4], removes these obstacles.
5.4
Flow-table occupancy by switch role
The amount of per-switch forwarding state a controller must install bounds how large a network can grow before a single switch’s flow table becomes the limit. Periplus’s central claim is that this state is local: an interior switch holds only the forwarding graph toward its own controller, while a border switch additionally holds one transit graph per peer controller it relays for. This is the property that makes the bootstrap scaling of Section 5.1 possible: because coordination state is confined to domain boundaries, a network can be partitioned across more controllers without inflating the state on any interior switch. We test the claim by counting the distinct (table, priority, match) rules the controller installs on each switch, grouped by role: root (the controller’s co-located switch), border (a switch that received at least one transit-graph install), and interior. We sweep the Mesh family from one to four controllers, with five repetitions per configuration. Fig. 10 confirms the locality claim. Interior occupancy is constant at ≈42 rules per switch regardless of the number of controllers. Border switches sit at ≈52 rules, the interior baseline plus a single peer’s transit subgraph, and appear only once a domain must relay for a non-adjacent peer (with two controllers placed at opposite ends of the Mesh, no switch lands on a transit path, so border switches first appear at three controllers). The root switch holds ≈27–37 rules, dropping slightly as controllers are added and routing work is shared among peers. None of the three counts depends on the absolute number of switches in the network: per-switch state is set by a switch’s role, not by network size, which is precisely what lets Periplus partition a network across controllers and scale to networks of around a hundred switches. By contrast,
17
Wall-clock recovery time (s)
40 35 30 25 20 15 10 5 0
stock OVS
patched OVS
Figure 9: Wall-clock controller-failure recovery time after killing the controller colocated in c4 in the 4c Mesh topology, for the default and patched OVS builds. 20 repetitions per variant. the spanning-tree-per-controller designs of Section 2 install one tree on every switch per controller, so their interior state grows linearly with the number of controllers; Periplus keeps it constant.
6
Conclusions and Future Work
This paper presented the multi-controller coordination plane of Periplus, an in-band SDN control plane whose single-controller design is developed in [4]. Periplus partitions a wide-area network across several controllers while keeping the coordination cost in-band and local: controllers discover one another through Controller Advertisement messages, and inter-controller routes are assembled incrementally from partial forwarding graphs inserted by border switches, so that per-controller forwarding state is confined to the switches at domain boundaries and never distributed across the interior of an intermediate domain. The Mininet evaluation, including a large-scale scenario with 96 switches and up to five controllers, confirms the two properties this design targets and characterises the coordination mechanism in action. Partitioning scales bootstrap: a 96-switch network is brought under control in around ten seconds once it is split across three or more controllers, since each controller bootstraps only its own domain. Per-switch flow-table occupancy is set by a switch’s role rather than by network size: interior occupancy is constant regardless of the number of controllers, and grows only at border switches, by one transit subgraph per peer
18
Root Border Interior
Installed flow rules per switch
50
40
30
20
10
0
1
2 3 Number of controllers K
4
Figure 10: Per-switch flow-table occupancy by role as the Mesh family is partitioned across one to four controllers. Markers are medians, whiskers the interquartile range. controller. Inter-controller discovery converges in 3.6–5.9 s, and a failed controller’s switches reattach to a surviving controller in a median of 10.2 s. The bootstrap, discovery, and recovery figures above are all reported on an optional patched OVS build; the patch addresses stock-OVS reconnect heuristics and is not required by the Periplus design, which otherwise runs unmodified on stock OVS. Several directions remain open. The evaluation so far relies on Mininet emulation; validating Periplus on a physical testbed—in particular over the resource-constrained links of the rural deployments that motivate it—is an important next step. The current first-come, first-served switch assignment leaves load uneven across controllers; using the switch-count and topology information already exchanged through C-Adv to inform attachment decisions is a natural step toward load-aware controller assignment. Building coordination and consistency protocols, such as eventual-correctness approaches [19], on top of Periplus’s inter-controller communication infrastructure is a further open direction.
References [1] R. Braden, L. Zhang, S. Berson, S. Herzog, and S. Jamin. Resource ReSerVation Protocol (RSVP) – Version 1 Functional Specification. RFC 2205,
19
IETF, September 1997. [2] Marco Canini, Iosif Salem, Liron Schiff, Elad M Schiller, and Stefan Schmid. Renaissance: A self-stabilizing distributed sdn control plane using in-band communications. Journal of Computer and System Sciences, 127:91–121, 2022. [3] David Carrascal, Elisa Rojas, Jose M. Arco, Diego Lopez-Pajares, Joaquin Alvarez-Horcajo, and Juan Antonio Carral. A comprehensive survey of in-band control in sdn: Challenges and opportunities. Electronics, 12(6), 2023. [4] E. M. Castro Barbero, P. de las Heras Quirós, and F. J. Simó Reigadas. Periplus: A resilient in-band sdn control plane via embedded forwarding graphs, 2026. arXiv preprint arXiv:2607.14869 [cs.NI]. [5] Kwan-Yee Chan, Chen-Hua Chen, Yi-Huei Chen, Yun-Ju Tsai, Steven SW Lee, and Cheng-Shong Wu. Fast failure recovery for in-band controlled multi-controller OpenFlow networks. In 2018 International Conference on Information and Communication Technology Convergence (ICTC), pages 396–401. IEEE, 2018. [6] Rogério Leão Santos de Oliveira, Christiane Marie Schweitzer, Ailton Akira Shinoda, and Ligia Rodrigues Prete. Using Mininet for emulation and prototyping software-defined networks. In 2014 IEEE Colombian Conference on Communications and Computing (COLCOM), pages 1–6, 2014. [7] Clarence Filsfils, Stefano Previdi, Les Ginsberg, Bruno Decraene, Stephane Litkowski, and Rob Shakir. Segment Routing Architecture. RFC 8402, IETF, July 2018. [8] Polina Holzmann, Artur Hecker, and Martina Zitterbart. Towards a distributed routing protocol for in-band control channel with elastic controller clusters. In 2018 IEEE Globecom Workshops (GC Wkshps), pages 1–6. IEEE, 2018. [9] Ahmad Jalili, Hamed Nazari, Sahar Namvarasl, and Manijeh Keshtgari. A comprehensive analysis on control plane deployment in sdn: In-band versus out-of-band solutions. In 2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI), pages 1025–1031. IEEE, 2017. [10] D. Katz and D. Ward. Bidirectional forwarding detection (bfd). RFC 5880, RFC Editor, June 2010. [11] Diego Kreutz, Fernando MV Ramos, Paulo Esteves Verissimo, Christian Esteve Rothenberg, Siamak Azodolmolky, and Steve Uhlig. Softwaredefined networking: A comprehensive survey. Proceedings of the IEEE, 103(1):14–76, 2014.
20
[12] Diego Lopez-Pajares, Joaquin Alvarez-Horcajo, Elisa Rojas, ASM Asadujjaman, and Isaias Martinez-Yelmo. Amaru: Plug&play resilient in-band control for sdn. IEEE Access, 7:123202–123218, 2019. [13] Maria Teresa Venegas Lorenti, Yue Su, Yang Xiang, Kien Nguyen, and Hiroo Sekiya. Combining in-band sdn and wireless mesh networks for internet provisioning in rural areas. In 2024 15th International Conference on Information and Communication Technology Convergence (ICTC), pages 1828–1832, 2024. [14] Andres Martinez-Fernandez, Josep Vidal, Javier Simo-Reigadas, Ignacio Prieto-Egido, Adrian Agustin, Juan Paco, and Alvaro Rendon. The tucan3g project: wireless technologies for isolated rural communities in developing countries based on 3g small cell deployments. IEEE Communications Magazine, 54(7):36–43, 2016. [15] Giang TK Nguyen, Rachit Agarwal, Junda Liu, Matthew Caesar, P Brighten Godfrey, and Scott Shenker. Slick packets. ACM SIGMETRICS Performance Evaluation Review, 39(1):205–216, 2011. [16] Christian Niephaus, Osianoh Glenn Aliu, Mathias Kretschmer, Senka Hadzic, and Gheorghita Ghinea. Wiback: A back-haul network architecture for 5g networks. In International Conference on Frontiers of Communications, Networks and Applications (ICFCNA), 2014. [17] Open Networking Foundation. OpenFlow Switch Specification, Version 1.5.1. ONF Technical Specification TS-025, March 2015. Accessed: 202604-09. [18] Open vSwitch Contributors. OVS Extensions — open vswitch documentation. https://docs.openvswitch.org/en/latest/topics/ ovs-extensions/, 2025. Accessed: 2026-04-09. [19] Aurojit Panda, Kangwook Lee, Kostas Argyraki, Mooly Sagiv, Michael Schapira, and Scott Shenker. SCL: Simplifying distributed SDN control planes. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 329–345, Boston, MA, March 2017. USENIX Association. [20] Paul Quinn, Uri Elzur, and Carlos Pignataro. Network Service Header (NSH). RFC 8300, IETF, January 2018. [21] Y. Rekhter, T. Li, and S. Hares. A Border Gateway Protocol 4 (BGP-4). RFC 4271, IETF, January 2006. [22] Ermin Sakic, Mirza Avdic, Amaury Van Bemten, and Wolfgang Kellerer. Automated bootstrapping of a fault-resilient in-band control plane. In Proceedings of the Symposium on SDN Research, pages 1–13, 2020.
21
[23] Liron Schiff, Stefan Schmid, and Marco Canini. Ground control to major faults: Towards a fault tolerant and adaptive sdn control network. In 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshop (DSN-W), pages 90–96. IEEE, 2016. [24] Liron Schiff, Stefan Schmid, and Petr Kuznetsov. In-band synchronization for distributed sdn control planes. ACM SIGCOMM Computer Communication Review, 46(1):37–43, 2016. [25] Sachin Sharma, Dimitri Staessens, Didier Colle, Mario Pickavet, and Piet Demeester. Automatic bootstrapping of openflow networks. In 2013 19th IEEE Workshop on Local & Metropolitan Area Networks (LANMAN), pages 1–6. IEEE, 2013. [26] Sachin Sharma, Dimitri Staessens, Didier Colle, Mario Pickavet, and Piet Demeester. Fast failure recovery for in-band openflow networks. In 2013 9th International Conference on the Design of Reliable Communication Networks (DRCN), pages 52–59, 2013. [27] Sachin Sharma, Dimitri Staessens, Didier Colle, Mario Pickavet, and Piet Demeester. In-band control, queuing, and failure recovery functionalities for openflow. IEEE Network, 30(1):106–112, 2016. [28] Yu-Lun Su, I-Chih Wang, Yao-Tsung Hsu, and Charles H-P Wen. FASIC: A fast-recovery, adaptively spanning in-band control plane in software-defined network. In GLOBECOM 2017-2017 IEEE Global Communications Conference, pages 1–6. IEEE, 2017. [29] RYU Project Team. Ryu sdn framework. https://book.ryu-sdn.org/ en/Ryubook.pdf, 2014. Accessed: 2026-04-09. [30] RYU Project Team. Welcome to ryu the network operating system (nos). https://ryu.readthedocs.io/en/latest, 2014. Accessed: 2026-04-09. [31] Ting-Shan Wong and Steven SW Lee. Design of an in-band control plane for automatic bootstrapping and fast failure recovery in p4 networks. IEEE Transactions on Network and Service Management, 20(3):3612–3629, 2023.
22