Resource versus Responsiveness: Benchmarking SDN Controller Runtimes for a Moving-Target-Defense Control Plane at Scale Souhail Chakkour
Umesh Biswas
Charan Gudla
arXiv:2609.35585v1 [cs.NI] 28 Sep 2026
Mississippi State University, Mississippi State, USA [email protected] [email protected] [email protected]
Abstract Network Moving Target Defense (MTD) built on Software-Defined Networking (SDN) continuously rotates host-facing addresses to invalidate an attacker’s reconnaissance. Such a defense is only as good as the control plane that drives it: every rotation is a burst of flow mutations, and every new connection is a reactive flow install that must complete before the address moves again. Yet the choice of SDN controller runtime—the framework that schedules these mutations—is treated as an implementation detail in the MTD literature. We show it is not. We port a single, identical MTD controller (CPAM) to three widely used runtimes—Ryu (single-threaded cooperative Python), OpenDaylight, and ONOS (both multi-threaded JVM)—and benchmark them under an identical 500-host campus fabric using an RFC 8456-aligned methodology with ten runs per controller. All three runtimes deliver the same data-plane correctness (near-zero loss, sub-millisecond jitter) but diverge sharply on the control plane: the two JVM controllers keep reactive round-trip time low and flat and preserve all sessions across rotations, whereas Ryu’s cooperative scheduler serializes reactive flow installs behind the periodic rotation burst, inflating reactive RTT by roughly 100× and dropping a small fraction of connections at setup time. Ryu, in turn, is markedly lighter, with roughly an order-of-magnitude smaller memory footprint. Crucially, the two JVM runtimes are not interchangeable: ONOS attains OpenDaylight-class reactive latency at the lowest CPU utilization of the three and a smaller live heap than OpenDaylight, showing that low reactive latency need not carry OpenDaylight’s full resource cost. The controller runtime thus imposes a concrete resource-versus-responsiveness trade-off on the MTD control plane. We root-cause each difference to the runtime’s concurrency and flow-programming model and argue that controller selection should be a first-class, measured decision in MTD system design.
Keywords: Moving Target Defense, Software-Defined Networking, address mutation, SDN controller benchmarking, control-plane performance, session continuity, Ryu, OpenDaylight, ONOS
1
Introduction
Moving Target Defense (MTD) reduces the value of adversarial reconnaissance by repeatedly changing the system attributes on which attackers depend [1–3]. In network-based MTD, a prominent approach is to periodically change the virtual IP addresses exposed by protected hosts. Consequently, information obtained through host discovery, port scanning, and topology inference becomes stale before it can be reliably used in an attack [4–8]. Software-Defined Networking (SDN) provides a natural platform for implementing address mutation. A logically centralized controller can maintain virtual-to-real address mappings, translate packets, and reprogram the forwarding fabric without modifying end hosts [9, 10]. However, the 1
security benefit of address mutation depends partly on how frequently addresses change. More frequent mutation shortens the useful lifetime of reconnaissance information, but also places greater pressure on the SDN control plane. Each mutation interval —the fixed period between two consecutive address rotations—ends with a burst of control-plane operations. During this burst, the controller updates virtual-to-real address mappings and modifies the corresponding forwarding rules in the switches. Meanwhile, a newly arriving connection may trigger reactive flow installation, in which the controller processes a PacketIn event and installs the rules needed to forward the connection’s packets. Established connections may also require retained session state, meaning per-session bindings and forwarding rules associated with the virtual address under which the session was created, so that the session remains valid after that address rotates. The controller must therefore handle address mutation, new-flow setup, and session-state maintenance concurrently. If these operations are delayed or improperly ordered, a new connection may use an address or rule that is being replaced, creating a rotation race that increases connection-establishment latency or causes the connection attempt to fail. Although SDN-based address mutation has been studied extensively, existing work generally treats the controller framework as an implementation choice rather than as an experimental variable [11– 13]. This assumption can be misleading because controller runtimes differ substantially in their concurrency, event-dispatch, state-management, and flow-programming architectures. A singlethreaded cooperative runtime may serialize reactive processing behind a mutation burst, whereas a multi-threaded runtime may execute these activities concurrently at the cost of greater CPU and memory consumption. General SDN controller benchmarks measure latency, throughput, and scalability [14], but do not reproduce the bursty and session-stateful workload created by address mutation. This raises the central question of this paper: How does the SDN controller runtime affect the responsiveness, continuity, and resource cost of an address-shuffling MTD when the underlying MTD logic is held constant? To answer this question, we implement Continuity-Preserving Address Mutation (CPAM) [15] on Ryu [16], a cooperative single-threaded Python controller, and on two multi-threaded JVM platforms—OpenDaylight (ODL) [17], which programs flows through its MD-SAL datastore, and ONOS [18], which uses a lighter in-memory flow subsystem. All three implementations use the same VIP lifecycle, session bindings, translation placement, rotation parameters, and reclamation rules; the controller runtime is the principal independent variable. We evaluate all three controllers over ten runs on an identical three-tier fabric with 500 hosts and 19 OpenFlow switches using RFC 8456-aligned metrics. All three provide near-zero UDP loss, sub-millisecond jitter, and preserve established sessions. The JVM controllers achieve low, flat reactive latency (0.131 ± 0.014 ms RTT and a 1.4 ms p99 processing time for OpenDaylight; 0.159 ± 0.014 ms and 1.2 ms for ONOS), compared with 15.98 ± 8.41 ms and 15.4 ms for Ryu. Ryu instead uses substantially less memory and achieves a higher aggregate flow-setup rate. The two JVM runtimes are themselves distinct: ONOS matches OpenDaylight’s reactive latency at the lowest CPU of the three and a smaller heap, so the low-latency operating point does not carry a single fixed resource cost. Our Contributions. This paper makes the following contributions: • We present, to our knowledge, the first controlled comparison of SDN controller runtimes for address-shuffling MTD in which the MTD logic, topology, workloads, and configuration are held constant. • We develop an RFC 8456-aligned benchmarking methodology tailored to address mutation, including rotation-aware reactive latency, session continuity, and fair Python-versus-JVM resource 2
measurements. • We quantify the resource-versus-responsiveness trade-off across Ryu, OpenDaylight, and ONOS on a 500-host campus fabric over ten runs per controller, showing that the two JVM runtimes occupy distinct operating points rather than a single one. • We provide a root-cause analysis connecting the observed latency, continuity, throughput, and resource differences to the controllers’ cooperative and thread-pooled concurrency models.
2
Background and Motivation
This section introduces SDN-based address-shuffling MTD, explains the control-plane workload created by address mutation, and reviews the prior work that motivates our controller-runtime comparison.
2.1
SDN-Based Address-Shuffling MTD
Prior work establishes address mutation as a practical MTD mechanism for reducing the lifetime of reconnaissance information and increasing adversarial uncertainty [1–3, 6–8]. In an SDN-based realization, each protected host retains a stable real IP address while communicating peers use a short-lived virtual IP address (VIP). The controller maintains the virtual-to-real mappings and installs the corresponding forwarding and translation rules [10, 19, 20]. We define the mutation interval as the time between two consecutive VIP rotations. At each rotation, the controller assigns new primary VIPs and updates the associated control and forwarding state. Shorter intervals therefore increase the frequency of control-plane updates, creating the runtime challenge examined in Section 2.2. Our evaluation adopts the Continuity-Preserving Address Mutation (CPAM) design introduced by Chakkour et al. [15]. CPAM pins the VIPs used when a session is established, allowing that session to continue after newer VIPs are assigned. Both implementations use the same VIP lifecycle, session bindings, translation placement, rotation parameters, and state-reclamation rules. The address-mutation mechanism is therefore fixed while the controller platform is varied.
2.2
The Controller-Runtime Challenge
Address-shuffling MTD creates a control-plane workload that differs from conventional reactive SDN forwarding in three important ways. First, each address rotation creates a periodic mutation burst. During this burst, the controller generates new address mappings, updates translation state, modifies forwarding rules, publishes the new mappings, and eventually reclaims retired addresses. These operations are concentrated around each rotation epoch rather than distributed uniformly over time [8, 15, 19]. Second, newly arriving connections may require reactive flow installation while a mutation burst is in progress. When a packet does not match an existing switch rule, the switch sends a PacketIn event to the controller. The controller must resolve the destination VIP, determine the forwarding path, and install the required rules before the connection can proceed [14]. If reactive processing is delayed behind mutation work, connection establishment may experience increased latency. Third, continuity-preserving mutation requires retained session state. The controller must maintain the address bindings and forwarding rules associated with established sessions until those sessions terminate. It must therefore create, access, and reclaim session state while simultaneously processing new connections and periodic rotations [15, 21, 22].
3
These activities can also create a rotation race. Such a race occurs when a new connection is being established while the mapping or forwarding state associated with its destination VIP is being replaced. Depending on the ordering and duration of these operations, the connection may be delayed, may use stale state, or may fail during setup. How a controller handles these competing activities depends on its runtime architecture. A cooperative single-threaded runtime may serialize reactive flow installation behind mutation work. A multi-threaded runtime may overlap the two activities through separate worker threads, but may require additional CPU, memory, synchronization, and state-management overhead. Thus, two controllers executing identical MTD logic may provide the same forwarding correctness but differ substantially in reactive latency, connection setup, flow-installation throughput, and resource consumption. Section 3 examines the relevant architectural differences in detail.
2.3
Existing Practice and Research Gap
SDN-based address-shuffling MTD has been studied extensively, including mutation design, adversaryaware address randomization, and transparent deployment [6–8, 11, 12, 20]. Jafarian et al. [8, 19] established foundational OpenFlow-based host-mutation mechanisms. Later work has considered session continuity [21, 22], recognition of active MTD mechanisms [23], and deployment in IoT environments [24]. A recent systematic review [13] identifies more than 80 SDN-based MTD systems. This literature primarily evaluates mutation strategies, security effectiveness, session preservation, and deployment mechanisms. The controller platform is typically selected as an implementation choice rather than treated as an experimental variable. Consequently, reported latency, throughput, or resource overhead may reflect both the MTD mechanism and the architecture of the controller that executes it. SDN controller performance has also been studied independently of MTD [25, 26]. RFC 8456 [14] defines metrics including reactive latency, flow-setup throughput, scalability, and resource consumption. However, conventional controller benchmarks generally model reactive forwarding without the periodic mutation bursts, retained session state, and rotation-overlapping connection setup introduced by address-shuffling MTD. The two research areas therefore leave a clear gap: MTD evaluations rarely isolate controllerplatform effects, while controller benchmarks do not reproduce an MTD-specific control-plane workload. We address this gap through a controlled comparison of Ryu, OpenDaylight, and ONOS in which the CPAM mechanism, topology, traffic, configuration, and measurement procedure are held constant. This design isolates how controller architecture affects reactive responsiveness, session continuity, flow-programming behavior, and resource consumption under the same address-mutation workload.
3
Controller Architecture
All three implementations execute the same CPAM control logic under the same configuration and workload. The controller runtime is therefore the principal independent variable in our study. This section compares Ryu, OpenDaylight, and ONOS along the architectural dimensions most relevant to periodic address mutation: event scheduling, state management, and flow programming. Ryu is a single-threaded Python controller; OpenDaylight and ONOS are both multi-threaded JVM platforms, but they differ in how flows are programmed—OpenDaylight through a YANG/MD-SAL transactional datastore, ONOS through a lighter in-memory flow subsystem—which, as the results show, separates them on resource cost. Figure 1 summarizes these differences and their expected effects. 4
Figure 1: Controller architectures under identical CPAM control logic. Ryu executes cooperatively on a single thread, while OpenDaylight and ONOS are both multi-threaded JVM runtimes that differ chiefly in their control-state and flow-programming paths (MD-SAL datastore vs. in-memory FlowRuleService).
3.1
Execution and Event Dispatch
Ryu [16] is a lightweight Python controller that uses cooperative green threads. Packet-in processing, periodic timers, address rotation, and statistics collection share one operating-system thread. A handler continues until it completes or explicitly yields at an I/O or sleep operation. This execution model simplifies state access because controller handlers generally do not modify the mapping tables concurrently. However, a long-running handler can delay other controller activities. This behavior is particularly relevant to CPAM. At each mutation interval, the rotation handler updates the VIP assignments and associated forwarding state for many hosts. A PacketIn arriving during this work may remain queued until the rotation handler yields. Ryu cannot use additional processor cores to execute the reactive handler in parallel with the rotation task. OpenDaylight [17] runs on a multi-threaded JVM-based platform. Its OpenFlowPlugin dispatches southbound events through worker threads, while application timers and service calls are handled through controller scheduler and service threads. Periodic rotation and reactive flow installation may therefore proceed concurrently on different cores. This parallel execution can reduce interference between mutation and connection setup. It also introduces costs absent from Ryu, including thread scheduling, synchronization, service abstraction, datastore processing, JVM heap management, and garbage collection. ONOS [18] is likewise a multi-threaded JVM platform, built on an OSGi (Apache Karaf) runtime. Southbound PacketIn events are delivered to a pipeline of packet processors running on a shared thread pool, while periodic timers (rotation, reclamation, statistics) execute on separate scheduler threads. Like OpenDaylight, ONOS can therefore overlap reactive flow installation with an ongoing rotation on different cores; unlike OpenDaylight, it does not interpose a transactional model-driven 5
datastore on the flow-programming path (below), which bears directly on its resource profile.
3.2
State Management and Flow Programming
Ryu stores CPAM’s host mappings and session bindings in ordinary in-process Python dictionaries. Because the application executes cooperatively on one thread, these structures require no applicationlevel synchronization during normal event handling. Ryu installs forwarding rules by constructing OpenFlow FlowMod messages and sending them directly through the switch datapath connection. OpenDaylight represents the corresponding state through its YANG-based Model-Driven Service Abstraction Layer (MD-SAL). The OpenDaylight implementation programs forwarding rules through the salFlowService interface, which the OpenFlowPlugin translates into southbound OpenFlow messages. This path provides structured state management and transactional controller services, but adds software layers between the application and the switch. ONOS keeps CPAM’s mapping and session state in ordinary concurrent in-memory structures and programs forwarding rules through its FlowRuleService, whose flow store is applied to the switch by the OpenFlow provider. This path is multi-threaded like OpenDaylight’s, but avoids the YANG datastore round-trip, so it sits between Ryu’s direct FlowMod path and OpenDaylight’s model-driven one in software depth. The three approaches consequently suggest different per-operation costs. Ryu’s direct state and flow-programming path requires the least memory and software overhead but serializes on one core. The two JVM controllers can overlap independent control-plane operations; between them, ONOS’s lighter in-memory flow path is expected to consume less CPU and heap than OpenDaylight’s model-driven datastore. These architectural differences motivate our measurements of reactive latency, flow-setup throughput, session continuity, and controller resource consumption in Sections 4–6.2.
4
Evaluation Methodology
We compare CPAM running on Ryu, OpenDaylight, and ONOS while holding the MTD logic, topology, workloads, configuration, and measurement procedure constant. The evaluation follows metrics aligned with the RFC 8456 controller-benchmarking methodology [14], extended with rotation-aware latency and session-continuity measurements for address-shuffling MTD.
4.1
Common CPAM Implementation
All three controllers implement CPAM [15]. Each protected host has a stable real IP address and a rotating VIP exposed to peers. Every 60 s, the controller assigns new primary VIPs and atomically publishes the updated mapping. Translation occurs at the destination access switch, so packets cross the fabric using VIPs before last-hop translation. A bidirectional session binding pins the VIPs used when a connection is established. New sessions use current primary VIPs, while established sessions retain their pinned mappings across rotations. Retired VIPs and session-scoped rules are reclaimed only after their session and flow references expire. A packet without a matching rule triggers a PacketIn; the controller resolves the VIP, computes the path, and installs forwarding and translation rules. Algorithm 1 summarizes the common control logic, while Table 1 lists the shared configuration. Mapping publication is atomic, preventing traffic generators from observing a partially updated mapping.
6
Algorithm 1 CPAM rotation and reactive-processing logic. State: primary[h]: current VIP of host h; SBT: bidirectional session bindings, keyed by flow; each VIP in state Primary, Grace, or Quarantine. 1: procedure OnRotationTick( ) ▷ every rotation interval (60 s) 2: for all protected hosts h do 3: vold ← primary[h]; vnew ← AllocVIP( ) 4: primary[h] ← vnew ; mark vnew Primary 5: mark vold Grace ▷ still valid for pinned sessions 6: end for 7: publish the virtual-to-real mapping atomically 8: for all VIPs v in state Grace do 9: if flowRefs(v)=0 ∧ sessionRefs(v)=0 beyond the reclaim threshold then 10: delete v’s forwarding rules; mark v Quarantine 11: end if 12: end for 13: return quarantined VIPs to the pool after the quarantine period 14: end procedure 15: procedure OnPacketIn(pkt) 16: classify pkt by source/destination namespace (real vs. VIP) 17: f ← normalized flow key of pkt 18: if f ∈ / SBT then 19: vc ← primary[src]; vs ← current VIP of the destination 20: SBT[f ], SBT[f¯] ← binding that pins (vc , vs ) 21: increment session and flow references on vc , vs 22: else 23: reuse the pinned binding SBT[f ] 24: end if 25: compute the forwarding path to the destination access switch 26: install path flows, with last-hop VIP→real translation 27: forward pkt 28: end procedure
4.2
▷ packet with no matching flow
▷ new protected session
▷ survives rotation
Control-Plane Workload and Complexity
We characterize the controller work generated by CPAM. Let N denote the number of protected hosts, G the number of VIPs currently in the Grace state, H the number of switches on a forwarding path, and λ the arrival rate of new reactive connections. Let T denote the mutation interval. During a rotation, the controller updates the primary VIP of every protected host and examines the Grace-state VIPs for reclamation. Assuming constant-time VIP allocation and reference-count access, the controller-side processing complexity of a rotation is Crot = O(N + G). This bound describes state-processing work; the actual completion time also depends on the number of generated OpenFlow operations and how the runtime schedules them. For a new connection, the controller performs an expected O(1) session-table lookup, computes a forwarding path, and installs rules along that path. If Cpath denotes the path-computation cost, the reactive-handler complexity is Creact = O(Cpath + H). With cached paths, Cpath = O(1) and the reactive cost is O(H). Without caching, a graph traversal 7
Table 1: Configuration shared by the Ryu, OpenDaylight, and ONOS implementations. Parameter
Value
Rotation interval VIP pool size Rotation batch size Rotation batch delay TCP session idle timeout UDP active timeout Grace-state reclaim threshold VIP quarantine period VIP-rule priority Table-miss priority OpenFlow version Translation placement Mapping publication
60 s 6,000 20 hosts 150 ms 15 s 30 s 5s 30 s 100 0 1.3 Last hop Atomic
requires O(|Vs | + |Es |) time, where Vs and Es are the switch vertices and links in the topology. The resulting asymptotic control-plane work rate is N +G Φ(T, λ) = O + λ(Cpath + H) . T Thus, reducing the mutation interval increases periodic controller work as 1/T , while connection churn increases reactive work linearly with λ. The runtime determines whether these two components are serialized or processed concurrently. In Ryu, the latency of a reactive event that overlaps a non-yielding rotation segment can be expressed as LRyu = Lbase + Rrot , where Rrot is the residual rotation work before the cooperative scheduler yields. For OpenDaylight, LODL = Lbase + Qpool , where Qpool captures worker-pool queuing and synchronization overhead. ONOS is likewise a thread-pool JVM runtime and takes the same form, LONOS = Lbase + Q′pool , where Q′pool is the corresponding queuing and coordination term for its in-memory flow subsystem. Because that path avoids OpenDaylight’s model-driven datastore round-trip, Q′pool is expected to be smaller than Qpool , consistent with ONOS’s lower measured resource cost at comparable reactive latency (Section 5). The model thus separates the single cooperative runtime (Ryu), whose delay is dominated by the rotation residual Rrot , from the two thread-pool runtimes, whose additional delay is instead worker-pool queuing. These expressions do not assume a particular measured latency; rather, they identify the scheduling components evaluated in Sections 5 and 6.2.
4.3
Testbed and Topology
Mininet and Open vSwitch emulate the same three-tier campus fabric for all three controllers: one core, six aggregation, and twelve edge switches connecting 500 hosts in 250 communicating pairs (Figure 2). Host assignments and traffic sequences are identical across controllers. 8
Figure 2: Three-tier evaluation fabric with 19 switches and 500 hosts in 250 communicating pairs.
4.4
Performance Metrics
We evaluate three aspects of controller behavior: data-plane correctness, control-plane responsiveness, and resource cost. Data-plane correctness. UDP packet loss and jitter measure whether continuous mutation affects packet delivery. TCP session continuity measures the fraction of long-lived sessions that remain operational across at least one complete address-rotation cycle. Control-plane responsiveness. Reactive round-trip time (RTT) measures the end-to-end latency of a packet that triggers a PacketIn, including controller processing, rule installation, and the first reply. Reactive per-packet processing time (RPPT) isolates controller-side processing from PacketIn receipt to FlowMod transmission. We also report flow-setup rate (FSR), measured as the number of new flows installed per second during connection churn. Resource cost. We measure controller CPU utilization and memory consumption. Because the JVM may reserve substantially more memory than it actively uses, the JVM controllers (OpenDaylight and ONOS) report memory using both resident set size and post-garbage-collection live heap. Ryu memory is reported as resident set size.
4.5
Workload Generation
Each run executes the same sequence of traffic phases on all three controllers. A settling period separates consecutive phases and allows forwarding state to stabilize. 1. UDP QoS: Each host pair generates a constant-bit-rate UDP stream to measure jitter and packet loss. 2. Reactive ICMP: Each pair sends ICMP requests to a newly selected VIP, forcing reactive rule installation and providing the reactive RTT measurement. 3. TCP throughput: Each pair performs a bulk TCP transfer to exercise the forwarding path. 4. Session continuity: Each pair opens a long-lived TCP connection that spans multiple rotation 9
intervals. 5. Connection churn: Each pair repeatedly creates short-lived TCP connections. Each unmatched connection generates reactive controller work and is used to measure FSR and RPPT under load. Each connection resolves its destination VIP immediately before connection establishment using the controller’s published mapping. Resolution uses bounded retry if a rotation occurs during the lookup. This prevents a workload launched before a rotation from using a stale VIP and incorrectly attributing the resulting failure to the controller.
4.6
Measurement Instrumentation
Reactive RTT, UDP jitter and loss, and TCP behavior are collected from ping and iperf client logs. RPPT is measured using controller-generated timestamps around each PacketIn-to-FlowMod processing interval. This measurement excludes network-channel and data-plane delay. FSR and forwarding-table occupancy are derived from periodic ovs-ofctl snapshots collected during the connection-churn phase. Controller CPU utilization is sampled with pidstat and restricted to the controller process. Memory is measured from process resident size; for the JVM controllers (OpenDaylight and ONOS), a forced garbage collection followed by a heap query additionally provides the live JVM heap.
5
Results
Table 3 summarizes the ten-run results. Data-plane correctness is similar across controllers, whereas control-plane latency and resource use differ sharply.
5.1
Reactive Round-Trip Time
Reactive RTT is the user-visible cost of the control plane: the round-trip time of a connection whose first packet must trigger a flow installation before it can be forwarded. It is 0.131 ± 0.014 ms on OpenDaylight and 0.159 ± 0.014 ms on ONOS, against 15.98 ± 8.41 ms on Ryu (Figure 3). The two JVM controllers land within 0.03 ms of each other—roughly two orders of magnitude below Ryu. Ryu’s minimum remains sub-millisecond, but its run-level values are much more widely distributed. Section 6.2 relates this variation to the timing of reactive events relative to address rotation.
10
Table 2: RPPT percentiles (ms), pooled over all per-event samples in the ten-run set. Ryu’s p99≫p50 is the signature of installations serialized behind the rotation burst. Controller
Load
p50
p90
p95
p99
OpenDaylight OpenDaylight ONOS ONOS Ryu Ryu
idle churn load idle churn load idle churn load
0.34 0.32 0.35 0.32 3.48 3.86
0.74 0.72 0.68 0.74 7.66 7.35
0.93 0.91 0.83 1.01 9.03 11.83
1.43 1.97 1.22 2.09 15.39 20.92
(a) Mean ± std
(b) Per-run distribution
Figure 3: Reactive RTT across ten runs. OpenDaylight and ONOS both remain tightly clustered and sub-millisecond, whereas Ryu exhibits a substantially larger mean and run-to-run spread.
5.2
Reactive Processing Time
Reactive per-packet processing time (RPPT) isolates the controller-side component of the reactive path—the interval from PacketIn receipt to FlowMod departure—and therefore removes channel and data-plane latency. The two JVM controllers cluster an order of magnitude below Ryu in the mean (0.44 and 0.42 vs 4.86 ms), but the mean understates the effect; the full distributions (Figure 4, percentiles in Table 2) reveal a divergence in shape. Pooling all per-event RPPT samples across the ten runs, OpenDaylight and ONOS are both tight from median to tail (p50 to p99 spans 0.34–1.4 and 0.35–1.2 ms respectively, a factor of about four), whereas Ryu is heavy-tailed: a 3.5 ms median but a p99 of 15.4 ms and a maximum of 909 ms, a single installation stalled for nearly one second behind a rotation burst. A concurrent connection-churn load (“under load” in Table 2) barely moves the JVM controllers (p99 1.4 → 2.0 ms for OpenDaylight, 1.2 → 2.1 ms for ONOS), confirming that their thread pools absorb the additional PacketIn rate, whereas Ryu’s tail rises further (p99 15.4 → 20.9 ms, maximum 1084 ms). The heavy tail, not the median, is what a latency-sensitive application experiences during rotation.
11
Figure 4: Cumulative distribution of RPPT, pooled over the ten-run set (log-scale x). OpenDaylight and ONOS are concentrated below 2 ms; Ryu is shifted an order of magnitude higher with a tail extending to ∼1 second, the same shape at idle and under churn load.
5.3
Flow-Setup Throughput
Flow-setup rate (FSR) measures aggregate rule-installation throughput under the connection-churn workload, reported here as the net Open vSwitch flow-count delta so that the metric is identical across controllers. Ryu achieves 441 ± 115 flows/s, ONOS 393 ± 56, and OpenDaylight 355 ± 54 (Figure 5)—all within a comparable band. Thus, the controller with the highest reactive latency (Ryu) nevertheless provides the greatest aggregate flow-setup throughput. This result highlights the distinction between throughput and latency. FSR measures the total number of installations completed per second under sustained load, whereas RTT and RPPT measure the delay experienced by individual reactive events. Consequently, aggregate throughput and per-installation latency favor different controllers. Section 6.2 relates this difference to their flow-programming paths and concurrency models.
5.4
Forwarding-Table Occupancy
Peak forwarding-table occupancy measures the transient data-plane state induced during the connection-churn burst. Because every controller runs identical CPAM logic, the per-session forwarding state is the same by construction; the differences in peak count—10 829 ± 2 793 entries for OpenDaylight, 13 048 ± 3 070 for Ryu, and 26 717 ± 10 479 for ONOS (Figure 6)—therefore reflect how many concurrent sessions’ flows co-reside at the peak instant, which scales with how quickly the host datapath completes the churn burst. This is the same host-bound effect observed 12
(a) Mean ± std
(b) Per-run distribution
Figure 5: Flow-setup rate under saturating churn across the three controllers. Ryu’s thinner flowprogramming path yields the highest aggregate throughput despite its worse per-installation latency. for TCP throughput (§5.6), not a structural divergence in switch state. Occupancy stays well within a commodity switch’s flow-table capacity at 500 hosts.
(a) Mean ± std
(b) Per-run distribution
Figure 6: Peak forwarding-table occupancy during churn. The per-session footprint is identical by construction; the peak-count differences reflect host-bound churn-completion timing (cf. TCP throughput), not controller structure.
5.5
Controller Resource Overhead
The low reactive latency of the JVM controllers comes at a resource cost, though the two JVM runtimes differ sharply. Average CPU utilization is highest on OpenDaylight (28.2 ± 1.5%), lowest on ONOS (12.1 ± 1.6%), with Ryu in between (18.8 ± 4.3%, confined to a single core). Memory 13
tracks the runtime: OpenDaylight commits ∼2.4 GB of resident size (fair post-garbage-collection live heap ∼1 GB) and ONOS ∼1.8 GB (∼0.4 GB live heap), both an order of magnitude or more above Ryu’s 107 ± 17 MB (Figure 7). ONOS thus attains OpenDaylight-class reactive latency at markedly lower CPU and a smaller live heap, while Ryu remains by far the most memory-frugal at the cost of its reactive-path latency and tail.
(a) Controller CPU
(b) Controller memory
Figure 7: Controller resource overhead across Ryu, OpenDaylight, and ONOS (mean ± std over 10 runs). Memory is shown as committed resident size; the fair post-garbage-collection live heap is ≈1 GB for OpenDaylight and ≈0.4 GB for ONOS.
5.6
Data-Plane Correctness
Once flows are installed, all three controllers forward identically. UDP loss is near-zero everywhere (≤ 0.011%) and mean jitter is sub-millisecond (0.63, 0.73, and 0.93 ms for OpenDaylight, Ryu, and ONOS), as shown in Figure 8. This is expected: the per-packet translation and forwarding rules are identical, and every controller programs the same Open vSwitch datapath. Bulk TCP throughput is bounded by the host software datapath rather than by the controller—once a flow is installed the controller is no longer on the packet path, and the emulation links are uncapped—so it is not a controller-distinguishing metric; measured in a single session on the same host, all three forward at the same rate (617–673 Mbit/s). The data plane therefore does not distinguish the runtimes; the entire difference between them is in the control plane.
14
(a) UDP jitter
(b) UDP loss
Figure 8: Data-plane correctness across the three controllers (mean ± std over 10 runs). UDP loss and jitter are near-identical, confirming an identical data plane. TCP throughput is host-datapath bound (uncapped emulation links) rather than controller-distinguishing and is reported in the text (§5.6).
5.7
Session Continuity
All three controllers keep near-all long-lived sessions alive across rotations, and—the key qualitative result—no controller ever tore down an established, data-carrying session; every observed loss occurs at connection setup or is an emulator-saturation event (§6). Over ten back-to-back continuity runs each, OpenDaylight and ONOS each preserve 100% of sessions and Ryu preserves 99.9% (3 of 2500 sessions broke, range 99.2–100%). Each Ryu loss was a setup-time race with a rotation—two mid-establishment resets within the first sub-second and one connection that never established. When continuity is instead measured within the full mixed-traffic workload, where a burst of 250 simultaneous connection setups competes with concurrent UDP, ICMP, and TCP traffic, Ryu’s continuity decreases to 96.8%, again entirely setup-time contention rather than mid-session teardown. Continuity is thus best read as a mechanism result (§6): the aggregate percentages are near-identical, and the substantive finding is that established sessions are never broken by a rotation on any of the three controllers.
15
Table 3: Results on the 500-host fabric (mean ± sd over ten runs). The JVM controllers’ memory gives committed RSS with fair post-GC live heap in parentheses. ‡: dedicated continuity workload. †: TCP throughput is bounded by the host software datapath (uncapped emulation links), not the controller; values are a same-session measurement (§5.6). §: peak concurrent flow count during churn; the per-session footprint is identical by construction, so the difference reflects host-bound churn-completion timing (cf. TCP), not controller structure.
Metric
OpenDaylight
Ryu
ONOS
Data plane UDP loss (%) UDP jitter (ms) TCP throughput (Mbit/s)†
0.011 ± 0.001 0.63 ± 0.32 617
0.000 ± 0.000 0.73 ± 0.24 639
0.002 ± 0.001 0.93 ± 0.41 673
Control plane Reactive RTT (ms) RPPT mean (ms) Session continuity (%) Flow-setup rate (fl/s) Forwarding-table peak§
0.131 ± 0.014 0.44 ± 0.06 100.0‡ 355 ± 54 10 829 ± 2 793
15.98 ± 8.41 4.86 ± 1.36 99.9 441 ± 115 13 048 ± 3 070
0.159 ± 0.014 0.42 ± 0.04 100.0‡ 393 ± 56 26 717 ± 10 479
28.2 ± 1.5 2.4 GB (≈ 1 GB)
18.8 ± 4.3 107 ± 17 MB
12.1 ± 1.6 1.8 GB (≈ 0.4 GB)
Resource cost Controller CPU (%) Memory
6
Analysis and Discussion
The results reveal an observed resource-versus-responsiveness trade-off. Because the CPAM mechanism, topology, workload, configuration, and measurement procedure are controlled across the three implementations, the differences are consistent with the controllers’ execution, state-management, and flow-programming architectures. We interpret these results using the control-plane workload model in Section 4.2.
6.1
Runtime-Level Mechanisms
Rotation scheduling. Ryu multiplexes address rotation, PacketIn handling, and statistics collection through a cooperative event loop. A reactive event arriving during a non-yielding rotation segment cannot be processed until that segment yields. It therefore incurs the residual rotation delay Rrot identified in Section 4.2. This behavior is consistent with the RPPT distribution in Figure 4: Ryu remains fast outside rotation epochs, but events overlapping rotation form a substantially heavier tail. OpenDaylight distributes rotation-related work and reactive OpenFlow processing across worker and service threads. Reactive processing can therefore proceed without waiting for the active rotation segment to finish. Its additional delay is instead associated with worker-pool queuing and synchronization, represented by Qpool in Section 4.2. This execution model is consistent with OpenDaylight’s tighter reactive-latency distribution.
16
ONOS provides an independent confirmation of this mechanism. As a second, architecturally distinct JVM controller, it dispatches reactive PacketIn processing on a thread pool separate from its rotation and statistics timers, and it reproduces the same low, flat reactive-latency distribution as OpenDaylight (Table 2). That two unrelated JVM runtimes both avoid the heavy tail indicates the effect stems from the concurrency model—parallel dispatch of reactive and rotation work— rather than any implementation detail specific to OpenDaylight. Where the two JVM controllers diverge is resource cost: ONOS programs flows through an in-memory FlowRuleService, whereas OpenDaylight serializes each installation through its MD-SAL datastore, and this heavier peroperation path is consistent with OpenDaylight’s larger CPU utilization and live heap at equal reactive latency. As a diagnostic check, explicitly yielding inside the Ryu rotation loop using hub.sleep(0) substantially reduced its reactive tail. This observation further supports cooperative serialization, rather than address translation itself, as the principal source of rotation-correlated delay. Setup-time races. Once CPAM creates a session binding and installs its forwarding state, the session remains pinned to the VIPs under which it was established. Those VIPs and their associated rules are retained until their session and flow references expire. Consequently, a subsequent rotation does not invalidate the forwarding state of an established session, consistent with the absence of mid-session failures in Section 5. The observed failures instead occur before a connection becomes a stable, data-carrying session. A new connection may resolve or begin using a VIP while the corresponding mapping and forwarding state are being updated. Depending on the ordering of these operations, its initial packets may be delayed, reset, or fail to reach the destination. The duration of this vulnerable setup window depends partly on controller scheduling. Ryu’s serialized execution can keep reactive processing waiting while rotation work remains active, increasing the opportunity for overlap. OpenDaylight can process reactive events concurrently with rotation, reducing that window. The relevant distinction is therefore between setup-time reliability and preservation of established sessions. Throughput and resources. Ryu’s direct flow-programming path and lightweight single-process runtime impose relatively little per-flow framework overhead. When the rotation handler is inactive, this path can complete many rule installations, which is consistent with its higher aggregate flowsetup rate. The same cooperative execution model, however, cannot overlap a reactive installation with a non-yielding rotation segment. OpenDaylight incurs additional coordination through its worker pools, MD-SAL services, managed state, and JVM runtime. These components increase CPU and memory consumption and introduce per-flow processing overhead, but they also isolate reactive work from periodic mutation processing. Consequently, higher aggregate throughput and lower per-event latency need not belong to the same controller: FSR measures completed installations per unit time, whereas RTT and RPPT measure the delay experienced by individual events.
6.2
Implications for MTD Design
Joint configuration. The workload model in Section 4.2 shows that periodic controller work increases as the mutation interval T decreases, while reactive work increases with the new-connection rate λ. A controller that performs well under infrequent rotations may therefore behave differently when rotations become more frequent or connection arrivals become more bursty. Under the tested configuration, the three controllers trace out distinct operating points rather than a single trade-off curve. Ryu is attractive when controller resources are constrained and occasional reactive tail latency is acceptable. The JVM controllers provide stronger isolation 17
between rotation and connection establishment—lower, flatter reactive latency and better setup-time reliability—at higher resource cost; but they are not equivalent. ONOS reaches this low-latency operating point at markedly lower CPU and a smaller live heap than OpenDaylight, making it the more resource-efficient choice when JVM-class responsiveness is required, whereas OpenDaylight’s model-driven services incur the highest resource cost of the three. These results do not establish one platform as universally superior; they identify three operating points—resource-frugal but tail-prone (Ryu), responsive but heavy (OpenDaylight), and responsive-and-leaner (ONOS)—whose suitability depends on deployment requirements. Evaluation metrics. Aggregate throughput alone does not capture the delay experienced by individual connections. Likewise, an aggregate continuity percentage does not distinguish a connection that fails during setup from an established session disrupted by rotation. Evaluations of SDN-based MTD should therefore report tail-latency metrics such as p95 and p99 RPPT, separate setup-time failures from mid-session failures, and report both throughput and per-event latency. Resource measurements should also account for runtime architecture. For a JVM-based controller, resident set size and post-garbage-collection live heap describe different aspects of memory use and should be reported separately when comparing against a lightweight controller runtime. Incremental rotation. The Ryu results do not imply that cooperative controllers are inherently unsuitable for address mutation. Their blocking interval can be shortened by dividing rotation work into smaller batches, yielding between batches, incrementally publishing updates, or proactively installing forwarding state for the next VIP assignment. These techniques reduce the residual rotation work faced by an arriving reactive event. Thread-pooled controllers may instead benefit from tuning worker pools, JVM memory, datastore transactions, and flow-programming services. More generally, high-frequency mutation should avoid a monolithic run-to-completion update and be implemented incrementally so that reactive connection setup can continue.
6.3
Limitations
The absolute measurements are specific to the three tested controller platforms, the 500-host emulated topology, and the fixed 60 s mutation interval. The controllers were also evaluated through backto-back runs without a restart between runs, and all 19 switches shared one software datapath. Sustained execution in this environment may increase run-to-run variance and does not reproduce the behavior of physical or distributed switching hardware. The matched experimental configuration supports the relative comparison reported in this paper, but the measured values should not be generalized to all controllers, mutation frequencies, workloads, or deployment platforms. Restart-isolated experiments, additional controllers, varied mutation intervals, and physical or distributed testbeds are left for future work.
7
Conclusion and Future Work
This paper shows that the SDN controller platform materially affects the performance of addressshuffling Moving Target Defense. By implementing identical CPAM logic on Ryu, OpenDaylight, and ONOS and evaluating all three on the same 500-host topology, we isolate the effect of controller architecture. All three implementations preserve data-plane correctness and established sessions across address rotations, confirming that CPAM’s continuity mechanism operates consistently across the platforms. The controllers nevertheless expose different resource-versus-responsiveness operating points. The two JVM controllers achieve substantially lower reactive latency and stronger connection18
establishment reliability during rotations, whereas Ryu uses considerably less memory and provides higher aggregate flow-setup throughput. The observed differences are consistent with their execution models: Ryu serializes rotation and reactive work through cooperative scheduling, while the JVM controllers process them concurrently through worker threads. Crucially, the two JVM runtimes are not interchangeable: ONOS attains OpenDaylight-class reactive latency at the lowest CPU utilization of the three and a smaller live heap than OpenDaylight, tracing to its lighter in-memory flow path versus OpenDaylight’s model-driven datastore—low reactive latency therefore need not carry OpenDaylight’s full resource cost. Controller selection should be considered together with mutation frequency, workload intensity, latency requirements, and available resources. Our evaluation is limited to three controllers, one emulated topology, a fixed 60 s mutation interval, and back-to-back runs on a shared software datapath. Future work will validate the results using restart-isolated runs and physical or distributed testbeds, and vary the mutation interval and workload intensity. We will also explore incremental rotation, proactive rule installation, and programmable data planes such as P4 [27] and eBPF [28, 29] to reduce control-plane contention and support higher-frequency mutation.
References [1] Sushil Jajodia, Anup K. Ghosh, Vipin Swarup, Cliff Wang, and X. Sean Wang. Moving Target Defense: Creating Asymmetric Uncertainty for Cyber Threats. Springer, 2011. [2] Hamed Okhravi, Michael Rabe, Thomas Mayberry, William Leonard, Thomas Hobson, Dave Bigelow, and William Streilein. Survey of cyber moving targets. Technical Report TR-1166, MIT Lincoln Laboratory, 2013. [3] Rui Zhuang, Scott A. DeLoach, and Xinming Ou. Towards a theory of moving target defense. In Proc. ACM Workshop on Moving Target Defense (MTD), pages 31–40, 2014. [4] Gordon Fyodor Lyon. Nmap Network Scanning: The Official Nmap Project Guide to Network Discovery and Security Scanning. Insecure.Com LLC, 2009. [5] Zakir Durumeric, Eric Wustrow, and J. Alex Halderman. ZMap: Fast internet-wide scanning and its security applications. In Proc. USENIX Security Symposium, pages 605–620, 2013. [6] Thomas E. Carroll, Michael Crouse, Errin W. Fulp, and Kenneth S. Berenhaut. Analysis of network address shuffling as a moving target defense. In Proc. IEEE International Conference on Communications (ICC), pages 701–706, 2014. [7] Ehab Al-Shaer, Qi Duan, and Jafar Haadi Jafarian. Random host mutation for moving target defense. In Proc. International Conference on Security and Privacy in Communication Networks (SecureComm), pages 310–327, 2012. [8] Jafar Haadi Jafarian, Ehab Al-Shaer, and Qi Duan. Adversary-aware ip address randomization for proactive agility against sophisticated attackers. In Proc. IEEE Conference on Computer Communications (INFOCOM), pages 738–746, 2015. [9] Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. Openflow: Enabling innovation in campus networks. volume 38, pages 69–74, 2008.
19
[10] Diego Kreutz, Fernando M. V. Ramos, Paulo Verissimo, Christian Esteve Rothenberg, Siamak Azodolmolky, and Steve Uhlig. Software-defined networking: A comprehensive survey. Proceedings of the IEEE, 103(1):14–76, 2015. [11] Jin-Hee Cho, Dilli P. Sharma, Hooman Alavizadeh, Seunghyun Yoon, Noam Ben-Asher, Terrence J. Moore, Dong Seong Kim, Hyuk Lim, and Frederica F. Nelson. Toward proactive, adaptive defense: A survey on moving target defense. IEEE Communications Surveys & Tutorials, 22(1):709–745, 2020. [12] Rongbo Sun, Yuefei Zhu, Jinlong Fei, and Xiangyu Chen. A survey on moving target defense: Intelligently affordable, optimized and self-adaptive. Applied Sciences, 13(9):5367, 2023. [13] L. Souto, P. Eisenkraemer, R. Shrestha, M. Chamana, O. Adeyanju, and S. Bayne. Moving target defense in software-defined networks: A systematic review. IEEE Access, 14:32252–32272, 2026. doi: 10.1109/ACCESS.2026.3668537. [14] Vishwas Bhuvaneswaran, Anton Basil, Mark Tassinari, Vishwas Manral, and Sarah Banks. Benchmarking methodology for SDN controller performance. Technical Report RFC 8456, IETF, 2018. [15] Souhail Chakkour, Umesh Biswas, and Charan Gudla. Balancing mutation entropy and session continuity in sdn-based moving target defense. In 2026 56th Annual IEEE International Conference on Dependable Systems and Networks-Supplemental Volume (DSN-S), pages 233–235. IEEE, 2026. [16] Ryu SDN Framework Community. Ryu: A component-based software defined networking framework. https://ryu-sdn.org, 2014. [17] Jan Medved, Robert Varga, Anton Tkacik, and Ken Gray. OpenDaylight: Towards a modeldriven SDN controller architecture. In Proc. IEEE WoWMoM, pages 1–6, 2014. [18] Pankaj Berde, Matteo Gerola, Jonathan Hart, Yuta Higuchi, Masayoshi Kobayashi, Toshio Koide, Bob Lantz, Brian O’Connor, Pavlin Radoslavov, William Snow, and Guru Parulkar. ONOS: Towards an open, distributed SDN operating system. In Proc. ACM HotSDN, pages 1–6, 2014. [19] Jafar Haadi Jafarian, Ehab Al-Shaer, and Qi Duan. Openflow random host mutation: Transparent moving target defense using software defined networking. In Proc. ACM Workshop on Hot Topics in Software Defined Networks (HotSDN), pages 127–132, 2012. [20] Douglas C. MacFarland and Craig A. Shue. The SDN shuffle: Creating a moving-target defense using host-based software-defined networking. In Proc. ACM Workshop on Moving Target Defense (MTD), pages 37–41, 2015. [21] Marek Żal, Marcin Michałski, and Piotr Zwierzykowski. Implementation of a lossless moving target defense mechanism. Electronics, 13(5):918, 2024. [22] Mateusz Fronczyk and Mariusz Rawski. Hybrid mutation MTD solution with dedicated SDN agents. In Proc. Federated Conference on Computer Science and Information Systems (FedCSIS), volume 43 of Annals of Computer Science and Information Systems, 2025. [23] Tina Moghaddam et al. MTDSense: AI-based fingerprinting of moving target defense techniques in software-defined networking, August 2024. URL https://arxiv.org/abs/2408.03758. 20
[24] Dilli Prasad Sharma. Evaluating moving target defense methods using time to compromise and security risk metrics in IoT networks. Electronics, 14(11):2205, 2025. doi: 10.3390/ electronics14112205. [25] Júlio Mendonça, Jin-Hee Cho, Terrence J. Moore, Frederica F. Nelson, Hyuk Lim, and Dong Seong Kim. Performance impact analysis of services under a time-based moving target defense mechanism. The Journal of Defense Modeling and Simulation, 20(1):41–56, 2023. [26] Jargalsaikhan Narantuya, Seunghyun Yoon, Hyuk Lim, Jin-Hee Cho, Dong Seong Kim, Terrence J. Moore, and Frederica Nelson. SDN-based IP shuffling moving target defense with multiple SDN controllers. In Proc. IEEE/IFIP DSN – Supplemental Volume, pages 15–16, 2019. [27] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, and David Walker. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review, 44(3):87–95, 2014. [28] Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. The express data path: Fast programmable packet processing in the operating system kernel. In Proc. ACM CoNEXT, pages 54–66, 2018. [29] Cilium Authors. Cilium: eBPF-based networking, observability, and security. https://cilium. io, 2024.
21