Escape from Callback Hell! A New Programming Paradigm for Network Simulation Yuanyi Zhu∗ , Zijian Li∗ , Xin Ai, Zixuan Chen, Sen Liu, Yang Xu†
arXiv:2605.16897v1 [cs.NI] 16 May 2026
Fudan University
events in the order of their scheduled times, thereby simulating the evolution of the system over time [4, 5]. Therefore, there is a need in DES for frequent event triggers. To support event triggering, modern DES network simulators widely adopt callbacks [6, 7, 8, 9]. This design aims to optimize network simulation development, aiding in module decoupling and enhancing the extensibility of the network simulation framework. Callbacks simplify the introduction and modification of inter-module interactions, enabling individual modules to be added, updated, and compiled independently [6]. In this way, callback-based designs conform to the Open/Closed Principle [10] and improve the overall efficiency of network simulation development. However, as the complexity of network design and optimization functions increases, the pervasive use of callbacks in network simulators has revealed significant limitations. In certain scenarios, callbacks deviate from their original intent and fail to comply with the Open/Closed Principle. Consider two network components in a simulation where Procedure A invokes Procedure B (Figure 1 and Section II-B). As the system evolves, blocking statements, which are fundamental network operations, may be introduced into Procedure B. To preserve correct execution, Procedure B must extend its interface with an additional callback to notify Procedure A upon completion, which in turn requires Procedure A to update its interface and implementation. Although this example involves only two procedures, greater complexity and tighter coupling can lead to cascading modifications. In addition, callbacks also impose substantial burdens in both development and debugging. Inverted control flow often forces developers to implement later callbacks first to satisfy earlier interfaces, while also requiring explicit management of callback state, which distracts from core logic. As systems scale, callback chains become deeply nested, and more than 10 levels within a simulated node is common, leading to issues such as stack ripping [11] and callback hell [12]. The fundamental reason lies in the inability of callbacks to naturally simulate network events, particularly blocking operations. In real-world networks, blocking operations can be suspended for user-friendliness or be staged via event callbacks to preserve system performance. In contrast, in DES, blocking operations have to be expressed entirely as discrete events, i.e., callback invocations. A single callback can represent only one point in a blocking operation, such as its start or completion, but it cannot identify which operation it belongs to or whether it denotes the start, suspension, or
Abstract—Network simulation plays a crucial role in both networking research and industry. Existing commonly-used Discrete Event Simulations (DES) are based on callback mechanisms for discrete event (DE). However, due to the inability of callbacks to naturally simulate network events, programs in network simulation cannot be written in a sequential workflow. This leads to inherent complexity and poor maintainability, resulting in stack ripping and callback hell. These problems significantly increase simulation development workloads and introduce substantial cognitive loads associated with programming and debugging. To enable more efficient development of network simulation and facilitate the rapid evaluation and evolution of network functions, we propose a novel development paradigm for network simulation named “CoDES” (Coroutine-based DES). To the best of our knowledge, we are the first to focus on optimizing the network simulation development process rather than performance based on the coroutine mechanism. We implement a new network simulation framework based on CoDES that is capable of naturally simulating network events and effectively address key system challenges related to correctness, functionality, compatibility, and overhead. It enables developers to create sequential workflows for network programs and simplifies the code structure, thus reducing development workloads while enhancing code readability and maintainability. We apply this paradigm to a commonly used network simulator, NS-3 to implement Message Passing Interface (MPI), High Precision Congestion Control (HPCC), and Routing Information Protocol (RIP), achieving up to 62.3% and 82.6% reduction in code volume and structure complexity without sacrificing simulation accuracy, extending execution time or increasing runtime memory of simulation. Index Terms—network simulation, data center network, networking
I. I NTRODUCTION Network simulation plays a pivotal role in facilitating the rapid advancement of network optimization and design. It serves as an essential tool for verification and evaluation of network design during the prototype design phase, or in scenarios with limited experimental resources [1]. Recent work on network simulation includes Unison [2] and DONS [3], which accelerate execution of network simulation through parallel technologies, and M3 [1], a flow-level network simulator that leverages machine learning to accelerate simulation. The core of network simulation lies in simulating network events and their corresponding effects to evaluate network performance. Current network simulators are mainly based on Discrete Event Simulation (DES), where events are scheduled along a timeline, and the event scheduler processes these ∗ Both authors contributed equally to this research. † Yang Xu is the corresponding author.
1
Data obtained successfully
Procedure A
Connect
The subsequent send can correctly transmit data
Get data
Procedure B
Send data
Fetch data Data returns immediately
Close
Return data Process B returns immediately
(a) Procedure A successfully gets data when data can be fetched immediately, e.g., local data fetch. Wrong program behavior
Procedure A
Connect
Procedure B
Get data
Incorrect data may be transmitted
Send data
Fetch data Data fetch involves blocking operations
Close
Return But no callback interface ensures that Procedure A proceeds only after data retrieval completes.
(b) Procedure A fails to get data when data fetch involves blocking operations, e.g., remote data fetch.
Fig. 1: Data fetch and send example with network function extension. end of that operation. As a result, callbacks provide only a fragmented view of blocking behavior, creating mismatches between the sequential logic of real network programs and their simulations, and increasing the burden for programmers familiar with conventional network programming.
evolution in network research. In summary, we outline three main contributions: • We point out that one key cause of the heavy development workloads in DES is callback’s inability to naturally simulate network events (Section II). We design CoDES (Coroutine-based DES), a novel programming paradigm for network simulation (Section III), and implement a corresponding simulation framework leveraging the C++ coroutine mechanism [13]. CoDES enables more accurate and developer-friendly representations of network events, substantially reducing learning cost and development effort. • We demonstrate, both theoretically and empirically (Sections IV and V-C), that callbacks and coroutines are interconvertible, yet differ markedly in how conveniently they express blocking-style network events. We further address key system challenges—coroutine resource conflicts, operation-level computation requirements, compatibility with existing callback-based simulators, and overhead (Section V)—and open-source our framework1 to promote adoption. • We implement three representative network functions across layers—MPI [14], HPCC [15], and RIP [16]— in NS-3 based on CoDES (Section VI). Compared with callback-based implementations, CoDES reduces LOC by up to 62.3% and function-call nesting depth by up to 82.6%, with negligible accuracy loss and negligible runtime and memory overhead (Section VII).
To reduce development effort and facilitate progress in network design and algorithms, a user-friendly network simulator should follow the sequential control flow of real programs and provide a blocking-style interface for modeling blocking network operations. This, in turn, requires a simulation framework that supports lightweight and explicit suspension and resumption of network operations, together with automatic state management. By transparently saving execution context and scheduling resumptions across discrete-event boundaries, the framework removes the need for manual state maintenance, reducing errors and improving programming productivity. We propose a novel programming paradigm, CoDES, for network simulation that enables user-friendly development of simulation programs based on lightweight coroutines [13]. To the best of our knowledge, we are the first to explicitly target development productivity through a coroutine-based programming paradigm in network simulation.We show that callbacks and coroutines are interconvertible, yet exhibit a distinct asymmetry in how naturally and succinctly they express blockingstyle network behaviors. We further implement a CoDESbased network simulation framework in NS-3, addressing key system challenges, including resource conflicts among concurrent coroutines, support for operation-level computation, compatibility with existing callback-based simulation components, and runtime overhead. We also implement representative network functions to illustrate the practical benefits of CoDES. Results show that CoDES reduces code volume by up to 62.3% and nesting depth by up to 82.6%, while incurring negligible overhead compared with widely used DES simulators, including NS-3 and SST/Macro. Overall, CoDES facilitates more efficient implementation and maintenance of network simulations, thereby accelerating evaluation, refinement, and
II. BACKGROUND & M OTIVATION A. Why are Callbacks Used Network simulation evaluates network performance by modeling network events and their effects. Most existing network simulators are based on DES, in which events are scheduled and processed in chronological order [17]. In DES, an event 1 https://github.com/CoDES-Development/CoDES
2
1 2 3 4
void get(data_t& value){ value = local_get(); return; }
5 6 7
void connect_and_send(addr_t peer_address, func_t data_provider){ socket = ...; // initialize socket connect(socket, peer_address, ...);
1 void get_new(data_t& value){ 2 value = remote_get(); 3 return; 4 }
“ t ” means “ type ”
5 6 7
data source changes
void connect_and_send(addr_t peer_address, func_t data_provider){ socket = ...; // initialize socket connect(socket, peer_address, ...);
8 9 10
data_t buffer; data_provider(buffer); // local data send(socket, buffer, ...);
8 9 10
data_t buffer; data_provider(buffer); // remote data send(socket, buffer, ...);
11 12 }
close(socket);
11 12 }
close(socket);
// USAGE: // connect_and_send(peer_address, get_new);
// USAGE: // connect_and_send(peer_address, get);
(a) Real-world program of local data fetch and update to remote data fetch 1 2 3 4
void get(data_t& value){ value = local_get(); return; }
1 2 3 4
void get_new(data_t& value){ value = remote_get(); return; }
5 6 7 8 9
void connect_and_send(addr_t peer_address, func_t data_provider){ socket = ...; // initialize socket socket.connect(peer_address, [&](socket){ data_t buffer; data_provider(buffer); // local data // data obtained immediately 10 socket.send(peer_address, buffer, [&](socket){ 11 socket.close(); 12 }); 13 }); 14 }
5 6 7 8 9
// USAGE: // connect_and_send(peer_address, get);
// USAGE: // connect_and_send(peer_address, get_new);
cannot be aligned with real-world modification
void connect_and_send(addr_t peer_address, func_t data_provider){ socket = ...; // initialize socket socket.connect(peer_address, [&](socket){ data_t buffer; data_provider(buffer); // remote data // !! Failure!! Buffer data may be inconsistent, causing send failures. 10 socket.send(peer_address, buffer, [&](socket){ 11 socket.close(); 12 }); 13 }); 14 }
(b) Simulation implementation of local data fetch and direct update to remote data fetch
Fig. 2: Real-world and callback-based simulation pseudocode of data fetching and sending under data source extension. 1 2 3 4 5 6
// !! get_new() has to change the interface to add callback parameter !! void get_new(data_t& value, callback c) { remote_get(value, c); // c(value) is called in remote_get // to return data } // !! connect_and_send() has to change the data_provider type to fit get_new() !! void connect_and_send(addr_t peer_address, func_new_t data_provider){ socket = ...; // initialize socket socket.connect(peer_address, [&](socket){ // !! socket.send() has to be a callback in data_provider() // instead of socket.connect() !!
7 8 9 10 11 12 13 14 }
modularity, and compilation efficiency. B. Problems of Callbacks Despite their widespread use and apparent suitability for modeling network events, callbacks exhibit substantial limitations as code complexity grows. Callbacks were originally introduced to improve development efficiency and enhance extensibility and modularity. However, when module functionality is extended or new modules are added, callbacks make it difficult to support blocking-style network operations without violating standard software engineering principles such as the Open/Closed Principle [10]. We provide a basic networking scenario as an example in Figure 1 and its corresponding implementation with callback in Figure 2 to explain the problem of using callbacks during network function extension. Procedure A, a typical network function connect and send(), first fetches required data and then sends it; Procedure B is a get() function that returns the data. In the initial scenario, as shown in Figure 1(a), the data source responds immediately, so Procedure B returns at once, enabling Procedure A to obtain the data and proceed. However, in network simulation development, new requirements frequently emerge. As in Figure 1(b), the data source may need to support both immediate retrieval (e.g., local storage) and delayed retrieval (e.g., remote cloud storage), which may block. Under the Open/Closed Principle, the interface between Procedures A and B should remain unchanged. However, once Procedure B becomes blocking, the original callback structure provides no mechanism to deliver the result back
data_t buffer;
data_provider([&](buffer){ // remote data socket.send(peer_address, buffer, [&](socket){ socket.close(); }); Violation of Open-Closed Principle }); });
// USAGE: // connect_and_send(peer_address, get_new);
Fig. 3: Simulation implementation of update to remote data fetch. typically combines a future operation, commonly used in realworld network programming to represent operations that take time to complete [6, 18, 19], with a timestamp indicating when the operation should be executed. Consequently, the callback mechanism is well suited for representing DEs [18]. By utilizing callbacks, the simulator can effectively manage control flow: once a scheduled operation concludes, the associated callback function is invoked to process the outcome. Callbacks also reduce direct inter-module dependencies compared to earlier tightly coupled designs, where simulation models directly invoked each other’s methods [6, 18]. By mediating interactions through callbacks, modules can be added, updated, and compiled independently, improving extensibility,
3
connect
to Procedure A upon completion, preventing the subsequent send in Procedure A from executing correctly. To further illustrate this issue, Figure 2 compares concrete implementations in real systems and in simulation. In real systems, as shown in Figure 2(a), blocking behavior can be introduced without changing function interfaces because the operating system manages blocking events. In DES, by contrast, blocking must be modeled explicitly through callbacks. Therefore, as shown in Figure 2(b), the same requirement change cannot be addressed by modifying Procedure B alone; instead, it necessitates changing the interface of Procedure B to accept an additional callback and updating Procedure A accordingly, as shown in Figure 3. This change increases callback nesting and structural complexity, and it compromises the original benefits of callbacks by breaking interface stability, increasing inter-module coupling, and violating the Open/Closed Principle. As the codebase grows, such callback dependencies become increasingly tangled. The two procedures shown here form a minimal example; real network simulators typically involve much more complex interaction patterns. Besides the problem of handling function extension, the use of nested callbacks makes the code structure chaotic and difficult to manage. This issue, commonly referred to as “callback hell”, exists not only in network simulation but also in various fields of software development involving asynchronous programming [11, 12, 20]. In network simulation, the problem is particularly severe due to the requirements of managing and maintaining numerous complex network states, such as packet loss handling, timeout retransmissions, and multi-node communication [21]. As shown in Figure 2(b), we can observe that the code structure becomes more nested in the simulation implementation with callbacks. Because the network function in the example is relatively simple, there is no obvious difference in LOC. In more complex scenarios, the code structure becomes even harder to develop and maintain. For instance, we find that MPI [22], a typical protocol used for communication in parallel systems, is implemented with about 10K LOC and numerous nested code blocks with more than 15 nesting depth by callback functions in a commonly used network simulator SST/Macro [23, 24], resulting in difficult future code extension and maintenance. Callback-based development also disrupts the natural sequential workflow of network programs. Using the simulation code for data fetching and sending as an example, Figure 4 illustrates the step-by-step implementation with callbacks. Developers must first implement the callback for the final close operation so that the preceding send operation can invoke it, and the same pattern continues recursively. This inverted development order differs substantially from realworld programming practice and imposes additional cognitive overhead. Moreover, the nested structure induced by callbacks complicates software maintenance tasks such as debugging. Figure 5 shows an example debugging stack for a basic applicationlevel function in the simulator. When a breakpoint is set at the application-layer handler for received packets, the stack re-
get data
send
close
(a) Real-world procedure of data fetch and send close
send
get data
connect Callback Hell !
getCallback
sendCallback closeCallback ( function body… )
connectCallback
sendCallback
closeCallback ( function body… )
closeCallback ( function body… )
getCallback sendCallback closeCallback ( function body… )
(b) Callback implementation procedure for data fetch and send
Fig. 4: “Callback Hell” problem in existing network simulation. Application Layer
ns3::PacketSink::HandleAccept ns3::Socket::NotifyNewConnectionCreated ns3::TcpSocketBase::DoForwardUp
Transport Layer
ns3::TcpSocketBase::ForwardUp ns3::Ipv4EndPoint::ForwardUp ns3::TcpL4Protocol::Receive
Network Layer
ns3::Ipv4L3Protocol::LocalDeliver ns3::Ipv4ListRouting::RouteInput ns3::Ipv4L3Protocol::Receive ns3::TrafficControlLayer::Receive
Data Link Layer
ns3::Node::ReceiveFromDevice ns3::Node::NonPromiscReceiveFromDevice ns3::PointToPointNetDevice::Receive ns3::MakeEvent::EventMemberImpl1::Notify
Simulator Event Scheduler Layer
ns3::EventImpl::Invoke ns3::DefaultSimulatorImpl::ProcessOneEvent ns3::DefaultSimulatorImpl::Run ns3::Simulator::Run main
Fig. 5: Debugging stack of handling received packets on the application layer under callback-based simulation. veals multiple network layers traversed by the packet. Extracting useful information from such a deep and entangled stack is difficult. In addition, call frames from different layers are often interleaved (e.g., transport and network layers), exposing low-level details that are largely irrelevant to application-layer developers. Conversely, the application-level context of the breakpoint that developers actually need is obscured. In summary, the fundamental reason why so many drawbacks of callbacks are exposed after extensive use, lies in the callbacks’ inability to naturally simulate network events, particularly blocking operations. We design a survey to investigate the use of current network simulators in research with the target group of over 100 people including PhD students, engineers, and researchers in the fields of networking. The survey results highlight the difficulties and workloads encountered during the development process of network simulation, consistent with the callback-induced limitations discussed above. C. What are Coroutines The core idea of coroutines is to allow a function to suspend execution, yield control, and later resume from the suspension point, thereby enabling asynchronous operations. This mechanism is fully supported in C++20 [13], the language upon which most network simulators are built, and is applied in complex environments like IoT and embedded systems [25]. Notably, to address the limited resources of embedded
4
1 2 3 4
systems, some approaches [25, 26] abandon the storage of local coroutine states in favor of using global variables. It is important to distinguish coroutines from threads. Coroutines are cooperatively scheduled in user space and typically rely on an event loop provided by the runtime or libraries, with relatively low memory overhead of only tens to hundreds of bytes per instance [13, 27] and low context-switch overhead. Threads, in contrast, are preemptively scheduled by the kernel and typically incur a much heavier burden, with individual sizes reaching several megabytes [28]. Their overhead grows significantly as the number of concurrent tasks increases. A key distinction is that coroutines allow for the explicit transfer of control to a designated routine, a capability that threads lack. This characteristic makes coroutines well suited to DES, a domain requiring precise event ordering. Furthermore, it is essential to distinguish the coroutines in CoDES from fibers [28, 29, 30]. While both function as lightweight, cooperatively scheduled user-space execution units, they differ fundamentally in implementation. Fibers utilize a dedicated scheduler and maintain a full, pre-allocated call stack (often several kilobytes) [28], which introduces architectural redundancy when integrated with the native event loop of DES. In contrast, coroutines are stackless and managed directly via language constructs like co await, storing only minimal state. Consequently, coroutines exhibit significantly lower memory usage and context-switching overheads—approximately 20× lower than fibers [28]—making them better suited to the resource constraints and scalability demands of network simulation.
Task<void> get_new(data_t& value){ value = co_await remote_get(); co_return; }
5 void connect_and_send(addr_t peer_address, func_t_ data_provider){ 6 socket = ...; // initialize socket 7 co_await socket.connect(peer_address); 8 data_t buffer; 9 co_await data_provider(buffer); 10 co_await socket.send(peer_address, buffer); 11 co_await socket.close(); 12 }
// USAGE: // connect_and_send(peer_address, get_new);
Fig. 6: Simulation implementation of CoDES-based data fetch and send example. ns3::CoDESPacketSink::HandleAccept Application Layer
ns3::CoDESPacketSink::connect ns3::CoDESPacketSink::Initialize ns3::CoDESPacketSink::StartApplication ns3::MakeEvent::EventMemberImpl0::Notify
Simulator Event Scheduler Layer
ns3::EventImpl::Invoke ns3::DefaultSimulatorImpl::ProcessOneEvent ns3::DefaultSimulatorImpl::Run ns3::Simulator::Run main
Fig. 7: Debugging stack of handling received packets on the application layer under coroutine-based simulation. in the usual manner before being sent. Second, CoDES substantially reduces structural complexity compared with the callback-based implementation shown in Figure 2(b). Third, the process of current simulation program corresponds well to the real-world workflow. Finally, we examine the debugging stack when handling received packets in a CoDES-based simulation as shown in Figure 7. In contrast to the convoluted stack trace produced by the callback-based simulation as shown in Figure 5, CoDES yields a clearer, application-oriented view that avoids irrelevant low-level details and preserves the application-layer context at the breakpoint. This provides more actionable debugging information, reduces debugging effort, and improves development efficiency. In the aforementioned survey, callback-based and CoDES-based examples are presented. The results show that 85.7% of respondents believe that the CoDES-based implementation reduces the development burden.
D. A New Paradigm Based on the above analysis of callback-related issues, network simulation development would become easier and more efficient if the implementation process better aligned with the sequential workflow of real-world network programs. This motivates a new programming paradigm for network simulation that supports lightweight, explicit suspension and resumption of network operations, together with automatic state management. A conventional approach to state maintenance is to use threads, relying on operating-system scheduling [31]. However, DES does not provide an operating system to manage such state in the first place. Moreover, threads introduce additional limitations [20], including substantial memory and context-switch overhead, as well as limited control over scheduling points—developers cannot explicitly determine which task regains control next, since thread scheduling is fundamentally preemptive. Based on the coroutine mechanism, we propose a new programming paradigm, CoDES, which addresses all the drawbacks of callbacks mentioned above. First, CoDES improves extensibility for module upgrades and feature additions. Using CoDES, we rewrite the data-fetching and sending simulation program, as shown in Figure 6. The resulting code remains natural and sequential: irrespective of whether data retrieval involves blocking operations, the fetched data can be returned
III. C O DES OVERVIEW Compared with existing network simulators that implement blocking network operations based on callbacks, we propose a novel programming paradigm, CoDES, and implement a framework for network simulations based on coroutines to enhance the modeling of network events in network simulation and the efficiency of network simulation development. In DES, existing event triggers are implemented based on callback. However, callbacks can only emulate the start or the completion of an individual blocking operation, making it difficult to capture the operation as a coherent whole during simulation. Therefore, DES requires a higher-level abstraction for network events.
5
Coroutines can provide such an abstraction for network events in DES. We show both theoretically (Section IV) and practically (Section V-C) that callbacks and coroutines are interconvertible, yet exhibit a distinct asymmetry in how naturally and succinctly they express blocking-style network behaviors. CoDES leverages coroutines to represent and simulate blocking network events within DES. Specifically, CoDES adopts proactive suspension and explicit resumption to more faithfully reproduce blocking behavior. It also provides automatic state management that hides intricate control logic and state transitions from developers. In addition, CoDES is lightweight, with low memory footprint and computational overhead. For users, all that is required is to encapsulate the network operation, whether blocking or non-blocking, as a CoDES operation. The creation of a CoDES operation signifies the initialization and commencement of the encapsulated network operation. One can use co await to wait for the completion of the network operation and obtain the corresponding result. One can actively suspend the current task using co yield, which will be resumed from the suspended position when resumed. Moreover, one can also actively terminate the current task using co return, and return the corresponding results. The management of the network operation state information is handled by CoDES, providing a seamless experience for users. The framework we implement for DES network simulation primarily consists of two components: coroutine frame structure and CoDES operation. A coroutine frame structure represents the information that a network operation needs to save. It manages the current state, result and exception of the network operation. A CoDES operation symbolizes a network operation, whether blocking or non-blocking, and handles the complete process of the network operation. This process includes creation, execution, suspension, and termination of the network operation. The primary challenges are as follows: 1) Given that multiple coroutines can lead to resource conflicts and deadlock issues, CoDES needs to ensure the correctness of coroutine implementation. 2) Given that network optimization design and algorithms in simulation require complex operations on network operations, CoDES operations need to support operationlevel calculations and chain operations. 3) As all existing DES network simulations are based on callback implementation, CoDES needs to adapt to the already implemented callbackbased simulation environment. 4) As simulations have constraints on resources and time, CoDES needs to reduce the overhead of state maintenance during the suspension and reentry of coroutines.
A. Definition of Callback and Coroutine We model the simulation runtime state as a tuple (Σ, ∆), where Σ represents the global state (e.g., the simulation event queue), and ∆ represents the local execution context (e.g., local variables). Definition 1 (callback). A callback-based event handler is a standard subroutine. Let Fcb be the set of callback functions. An execution of a callback f ∈ Fcb is a transition: call
⟨f, Σ, ∆in ⟩ − −− → ⟨Σ′ , ⊥⟩
(1)
Upon completion of f , the local context ∆in is destroyed (denoted by ⊥). Consequently, a callback cannot preserve local state across distinct simulation events without externalizing that state into Σ. Definition 2 (coroutine). Following [32], we define a coroutine as a state machine equipped with resume and yield primitives. A coroutine instance c encapsulates a suspended computation. Its execution is modeled as a sequence of transitions. The yield operation suspends execution and returns control to the caller: yield
⟨c, Σ, ∆⟩ −−−→ ⟨c′ , Σ′ , ∆⟩suspended
(2)
The resume operation continues a suspended coroutine: resume
⟨c′ , Σ′ , ∆⟩suspended −−−−−→ ⟨Σ′′ , ∆′ ⟩
(3)
Crucially, the local context ∆ is preserved across the yield transition. When the coroutine is resumed, it continues execution with ∆ intact. B. Expressiveness Comparison While callbacks and coroutines are interconvertible (Section V-C), this equivalence is asymmetric. Leveraging macro elimination (Theory 1), we prove that coroutines are more expressive, as a single coroutine can naturally replace a callback, whereas simulating a coroutine typically requires multiple callbacks. Theory 1. Language feature A is more expressive than feature B if translating A into B requires a global restructuring of the program, whereas translating B into A requires only a local transformation. [33] Let Lcb be a language supporting only callbacks, and Lco be a language supporting coroutines. We assert that Lco is strictly more expressive than Lcb in the context of network simulation. Proof. Lcb → Lco (Local Transformation): Any callback f can be trivially wrapped as a coroutine that executes f and never yields.
IV. T HEORETICAL A NALYSIS OF THE E XPRESSIVENESS : C ALLBACK VS . C OROUTINE In this section, we provide an analysis of the expressiveness of callbacks versus coroutines in the context of network simulation. We ground our definitions in the operational semantics of coroutines [32] and evaluate their relative expressiveness using Felleisen’s theory of macro elimination [33].
Wrap(f ) = coroutine{f (); return; } This transformation is local because it does not require changing the internal structure of f or the global state definitions.
6
Proof. Lco → Lcb (Global Restructuring): Consider a coroutine C that performs a network operation, yields for time t, and then continues:
Moreover, there are instances where multiple callers interact with the same network operation, thus posing a race condition issue for the coroutine frame structure that stores the information of this network operation. To deal with this challenge, it is crucial to guarantee that the coroutine frame structure can be properly released—specifically, by releasing the frame as early as possible and ensuring it is released only once. We clarify the ownership of the coroutine frame structure to ensure that only one caller holds the exclusive ownership [36] of the coroutine frame structure (this solution corresponds to challenge 1). This approach ensures that even when multiple callers simultaneously hold the coroutine frame structure, only one caller controls its lifecycle, while the other callers can freely access the state data structure within its lifecycle. Additionally, we introduce a counter in the coroutine frame structure to ensure that it is accurately released when the last holder finishes using it (this solution corresponds to challenge 1). Considering simulations have constraints on resources and time, we design CoDES to employ lightweight stackless coroutines [13], with the aim of reducing the overhead (this solution corresponds to challenge 4). In contrast to stackful coroutines [11, 37], which are adopted in DepFast [20], the stackless coroutines used in CoDES saves only the minimal state required to resume execution from the point where it yielded. This approach results in lower memory overhead, as there is no need for a full stack for each coroutine, and faster context switching, as less data needs to be saved and restored [13]. Of course, stackless coroutines are more complex to implement due to the need for careful management of the execution state [13]. However, the complexity of this implementation is encapsulated within CoDES, making it invisible for developers when using the framework of CoDES.
C = {op1 ; yield(t); op2 (local var); } Eliminating the yield operator requires stack ripping, which necessitates: 1) Partitioning C into disjoint functions f1 (pre-yield) and f2 (post-yield). 2) Defining a new global data structure S to externalize the local state of v. T −1 (C) ⇒ {S, f1 (Σ) → S, f2 (Σ, S)} Since this transformation introduces new global types (S) and fragments the control flow across disjoint syntactic units, it constitutes a global restructuring. Thus, Lco is strictly more expressive than Lcb , meaning that coroutines is more expressive than callbacks. V. C O DES D ESIGN CoDES comprises two core components: state maintenance and CoDES operations. State maintenance stores and manages the states of network operations, while CoDES operations implement specific suspendable network operations. The following sections detail these components and demonstrate compatibility of CoDES with existing callback-based network simulators. A. State Maintenance In contrast to callbacks that require users to manually save the context, CoDES automatically manages state maintenance. CoDES utilizes the coroutine frame structure to store all state information pertaining to network operations. In CoDES, the states of network operations are automatically preserved within coroutine frame structures upon suspension and subsequently restored upon resumption. The coroutine frame structure maintains the local variables, the continuations, and the references to certain resources such as file handles and network connections. This ensures the accurate restoration when network operations are resumed. As mentioned above, the coroutine frame structure preserves references to the resources required in a network operation. A deadlock issue may arise when the same resource is referenced by multiple coroutine frame structures. To handle this challenge, we transform the resource reference graph [34] in state management into a directed acyclic graph (this solution corresponds to challenge 1), preventing cyclic references [34] within the coroutine frame structure that could otherwise lead to memory leak [35] issues. This approach is feasible because, in a single-process network simulator [6, 7, 8], events are triggered sequentially, with only one event requiring access to a resource at any given time. Therefore, we can assign exclusive ownership [36] of the resource to the event currently referencing it, while other events maintain a nonowning relationship, effectively breaking any potential cycles in resource ownership.
B. CoDES Operation To provide a better abstraction for network operations, we designed CoDES Operation. As such, any network operation can be encapsulated as an Operation, allowing execution to be paused and resumed. A CoDES Operation comprises several components: the coroutine frame structure, the coroutine handle, the execution logic, the suspension points, the return values, and exception handling. These components enable effective management of the entire network operation process. Existing network designs and algorithms necessitate computation of network operations, hence we introduce a higher-level abstraction, OperationCalculation, designed to facilitate the calculation of CoDES operations (this solution corresponds to challenge 2). In many typical network scenarios, such as anycast and load balancing, one sender sends requests to multiple receivers and can continue execution once the first response is received. To address this requirement, OperationCalculation supports operations of the form op1 ∥ op2 ∥ op3 , where it unifies the result types of the operations. If any of the operations completes, the others are terminated, and a result, based on the previously unified types, is returned.
7
void function(callback_t callback);
coroutine_t c;
// USAGE: Add one more layer to the callback nesting level // function(callback);
// USAGE: Add new coroutine-based functionality // co_await c;
coroutine_t function_wrapper(){ coroutine_t c = makeCoroutine(); function([](){ c.terminate(); }); return c; }
// if there exits nested_callback callback_t c_callback = [](callback_t nested_callback){ c.onComplete(nested_callback); c.resume(); }; // if there doesn’t exit nested_callback callback_t c_callback = [](){ c.resume(); };
// USAGE: The nesting level of callbacks remains unchanged // co_await function_wrapper(); // callback();
// USAGE: No need to modify existing functions that accept callback interfaces. // function(c_callback);
(a). Callbacks upgrade to coroutines in general case
(b). Coroutines downgrade to callbacks in general case
Fig. 8: Adapting CoDES to existing callback-based network simulation. To address the issue of callback hell [21] resulting from multi-level nesting, we design ChainOperation, inspired by the concepts of lazy evaluation [38] and chaining [39] from functional programming [40] (this solution corresponds to challenge 2). Adhering to the principle of chaining [39], ChainOperation condenses multi-layered nested operations into a single ultimate operation and eliminates the necessity of using intermediate variables, thereby handling the issue of callback hell. The lazy evaluation strategy of ChainOperation decouples the dependency between communication information and the algorithm itself, thereby reducing unnecessary coupling. Developers can design algorithms with ChainOperation either directly used or passed along to subsequent layers without concern for extra factors. From a debugging perspective, this approach enables developers to focus on the key parts, reducing unnecessary debugging work and avoiding being overwhelmed by the details of data handling.
simulation function. Subsequently, the new network simulation feature will be triggered and executed as expected. Of course, as stated in Section IV, one coroutine can be downgraded to several callbacks. VI. B UILDING N ETWORK F UNCTIONS WITH C O DES To demonstrate the effectiveness of CoDES, we utilize it to deploy three typical and widely used network functions at different network layers that are highly challenging for callbackbased implementation with complex state management in NS3 [6, 18]: MPI, HPCC, and RIP, referred to as CoDES-based MPI, CoDES-based HPCC, and CoDES-based RIP. Across all three cases, the primary challenge arises from managing long-lived and evolving protocol states in the presence of asynchronous events. MPI demands precise inter-process synchronization and coordinated data distribution, where blocking and non-blocking communications must strictly respect execution dependencies. HPCC requires finegrained queue state management at both switches and endpoints, with queue states being frequently updated by packet arrivals and timer-driven events, such as PFC pause and recovery. RIP, in turn, maintains routing tables that continuously evolve through periodic updates and triggered messages from neighboring routers, necessitating careful coordination of concurrent timers and message receptions. CoDES addresses these challenges by enabling protocol logic to be expressed using coroutines. Interdependent operations in MPI, timed queue recovery in HPCC, or periodic and triggered updates in RIP can be naturally expressed using co await, which implicitly manages suspension and resumption. Shared protocol states can be accessed directly within a coroutine without being explicitly propagated through callback chains, thereby mitigating the risk of state inconsistency.
C. CoDES Compatibility Existing network simulation environments are based on callbacks. With minor adaptations, CoDES can reuse existing callback-driven functionality while providing coroutine-style benefits. Using the same approach, we integrate multiple network functions into the callback-based NS-3 simulator, as detailed in Section VI. Figure 8(a) illustrates the general scenario of upgrading callbacks to coroutines (this solution corresponds to challenge 3). Since any callback can be transformed into a coroutine body with only an exit point, as stated in Section IV, we wrap the original function and replace the callback with such a coroutine. When execution reaches the callback site, control yields to the caller, which then invokes the callback in a sequential workflow, without increasing nesting depth. Figure 8(b) depicts the general scenario of downgrading coroutines to callbacks (this solution corresponds to challenge 3). If we implement new network simulation features based on CoDES and need to connect with existing network simulation features implemented via callbacks, we don’t need to recursively modify all the interfaces. Instead, we can encapsulate the new network simulation feature within a callback form and put the callback into the interface of the old network
VII. E VALUATION In the evaluation, we focus on addressing two key questions: 1) How much workload can CoDES save in simulation w.r.t reducing lines of code (LOC) and simplifying code structures? 2) Can CoDES ensure a low overhead in accuracy loss, additional execution time, and extra runtime memory, while reducing the development workloads? This section compares
8
Frequency
CoDES-based implementations of MPI, HPCC, and RIP with their callback-based counterparts to answer these questions. A. System Setup We implement CoDES-based MPI, CoDES-based RIP, and CoDES-based HPCC using NS-3. The callback-based HPCC and RIP refer to the HPCC and RIP modules in NS-3, while the callback-based MPI corresponds to the MPI model in SST/Macro [23, 24], a widely used flow-level network simulator, as the callback-based MPI implementation is not available in NS-3. The server for running simulation is configured with an AMD EPYC 7R13 48-core processor and 251 GiB of memory. The benchmarks for MPI include various MPI functions and typical real-world traces from HPCG, LULESH, and HILO. The test suites for HPCC and RIP include the official NS-3 open-source codes for system test and unit test.
3
5 6 14 16 17 20 21 Nesting Depth in an MPI function
23
CoDES-based
Fig. 10: Comparison of nesting depth between callback-based and CoDES-based MPI. nesting depth in CoDES-based MPI is reduced by up to 82.6% compared to callback-based MPI. This reduction demonstrates that CoDES significantly simplifies code structure complexity, which eliminates the serious problems of callback hell and leads to reduced cognitive load during programming. C. Overhead To address the second question, we evaluate the simulationto-reality discrepancy using traces from 1000+ node Dragonfly clusters. We also compare the execution time and memory usage of CoDES versus callback-based implementations. 1) Accuracy Loss: As illustrated in Figure 11(a), the absolute simulation error, defined as the difference between the simulated and actual completion time of the application trace, remains within 0.05 seconds for both CoDES-based MPI and callback-based MPI. As depicted in Figure 11(b), both the value of the relative error of CoDES-based MPI and that of callback-based MPI exhibit high consistency. Due to page limit, we only show the simulation error of MPI in figures. For HPCC and RIP, the simulation error is also within reasonable range and basically less than that of MPI due to relatively simpler network operations and states. We also observe identical simulation time in both CoDES-based and callback-based bulk-send applications under parallel execution, demonstrating CoDES’s inherent support for parallel simulation. 2) Execution Time & Runtime Memory: For HPCC and RIP, the difference in simulation execution time between the callback-based and CoDES-based implementations is negligible, with a 0.35% variation in runtime and approximately 0.8% in memory usage. We do not compare the callback-based MPI with the CoDES-based MPI, as they are implemented on flowlevel [23] and packet-level simulators [6], respectively, and are therefore not directly comparable. In the parallel simulation demo, execution time differs by about 0.6%, and memory usage varies by roughly 2.6%.
To address the first question, we compare the LOC and nesting depth of callback-based versus CoDES implementations, demonstrating reduced development workload and cognitive complexity. 10000
Lines of Code
4
callback-based
B. Code Volume & Structure Complexity
8000 6000 4000 2000 0 MPI callback-based
10 8 6 4 2 0
HPCC RIP CoDES-based
Fig. 9: Comparison of LOC between callback-based and CoDES-based Implementations. 1) Lines of Code (LOC): As shown in Figure 9, the CoDES-based MPI reduces the LOC by 62.3% compared to the callback-based MPI. CoDES greatly simplifies the complex and extensive inter-process synchronization and data distribution in callback-based MPI. Similarly, the CoDESbased HPCC and RIP achieves 51.4% and 24.7% LOC reduction respectively. The CoDES-based RIP enables a smaller reduction compared to CoDES-based MPI and HPCC, because RIP involves relatively simpler network operations and states with only a limited number of asynchronous communication events, which limits CoDES to fully leverage its advantages in simplifying state management. As the complexity of states and operations increases, e.g., from RIP to HPCC and to MPI, the reduction in code volume achieved by CoDES becomes more significant. 2) Nesting Depth: Figure 10 compares the nesting depth within each of 20 kinds of commonly used MPI functions between CoDES-based MPI and callback-based MPI through a commercial code analysis tool [41]. It is observed that the nesting depth in CoDES-based MPI ranges from 3 to 6, whereas in callback-based MPI, it ranges from 14 to 23. The
VIII. R ELATED W ORK A. Asynchronous Programming The debate between continuation-passing style (CPS), e.g., callbacks [42], and async/await-style programming [28, 29, 43] (e.g., coroutines and fibers) has persisted for years. CPS is common in performance-critical domains such as lowlevel systems and network simulation, but it often leads to callback hell. Async/await improves structure and readability, yet it has long been limited by language support and runtime
9
Relative Error (%)
Absolute Error (s)
0.05 0.04 0.03 0.02 0.01 0 6
72 342 Number of Nodes
callback-based MPI
1 0.8 0.6 0.4 0.2 0 6
1056
72
342
1056
Number of Nodes
CoDES-based MPI
callback-based MPI
(a) Absolute Error
CoDES-based MPI
(b) Relative Error
Fig. 11: Absolute and relative simulation time error compared to the actual results in real-world network. overhead; it is now supported by mainstream languages such as JavaScript, Go, and C++20 [13, 27, 28, 29]. Prior work such as Protothreads [26] targets embedded systems and sacrifices local state for performance. Other work, including cooperative scheduling [11, 37] and quorum systems [20], relies on fibers and can incur substantial memory and context-switching overhead. Our work applies these principles to network simulation, balancing efficiency and usability.
[2] S. Bai, H. Zheng, C. Tian et al., “Unison: A parallelefficient and user-transparent network simulation kernel,” in Proceedings of the Nineteenth European Conference on Computer Systems, 2024, pp. 115–131. [3] K. Gao, L. Chen, D. Li et al., “Dons: Fast and affordable discrete event network simulation with automatic parallelization,” in Proceedings of the ACM SIGCOMM 2023 Conference, 2023, pp. 167–181. [4] A. Varga, “Discrete event simulation system,” in Proc. of the European Simulation Multiconference (ESM’2001), vol. 17, 2001. [5] G. S. Fishman, Discrete-event simulation: modeling, programming, and analysis. Springer, 2001, vol. 537. [6] T. R. Henderson, M. Lacage, G. F. Riley et al., “Network simulations with the ns-3 simulator,” SIGCOMM demonstration, vol. 14, no. 14, p. 527, 2008. [7] X. Chang, “Network simulations with opnet,” in Proceedings of the 31st conference on Winter simulation: Simulation—a bridge to the future-Volume 1, 1999, pp. 307–314. [8] A. Varga, “Omnet++,” in Modeling and tools for network simulation. Springer, 2010, pp. 35–59. [9] L. Bajaj, M. Takai, R. Ahuja et al., “Glomosim: A scalable network simulation environment,” UCLA computer science department technical report, vol. 990027, no. 1999, p. 213, 1999. [10] R. C. Martin, “The open-closed principle,” More C++ gems, vol. 19, no. 96, p. 9, 1996. [11] A. Adya, J. Howell, M. Theimer et al., “Cooperative task management without manual stack management,” in 2002 USENIX Annual Technical Conference (USENIX ATC 02), 2002. [12] J. Edwards, “Coherent reaction,” Tech. Rep., 2009. [13] A. Fertig, Programming with C++ 20: Concepts, Coroutines, Ranges, and more. Fertig Publications, 2021. [14] D. W. Walker and J. J. Dongarra, “Mpi: a standard message passing interface,” Supercomputer, vol. 12, pp. 56–68, 1996. [15] Y. Li, R. Miao, H. H. Liu et al., “Hpcc: High precision congestion control,” in Proceedings of the ACM special interest group on data communication, 2019, pp. 44–58. [16] R. Stark, “Secularization, rip,” Sociology of religion, vol. 60, no. 3, pp. 249–273, 1999. [17] A. Musa and I. Awan, “Functional and performance analysis of discrete event network simulation tools,” Simulation Modelling Practice and Theory, vol. 116, p.
B. Network Simulation & Network Emulation Network simulation broadly falls into time-slot-driven [24, 44] and event-driven [6, 7, 8, 9] categories. Event-driven, or DES simulators, favored for their scalability and efficiency over time-slot-driven ones, are all based on callback. Unlike existing parallel simulation frameworks [2, 3] that prioritize execution speed, CoDES focuses on improving DES development efficiency. These two objectives are theoretically orthogonal. Since CoDES dispatches coroutines via an underlying event loop, it is compatible with parallel simulators provided they enforce valid execution ordering—a fundamental requirement for correctness. This compatibility is experimentally verified in Section VII-C1. Compared with network simulators, network emulators [45, 46, 47] offer a high-fidelity simulation environment where users can flexibly combine virtual machines. Due to resource limitations, large-scale network function evaluations mainly rely on simulators. Existing commonly used emulators are also developed with callback functions. We envision that CoDES can be applied to all simulation scenarios affected by callback hell, enhancing development efficiency. IX. C ONCLUSION It is observed that the heavy development workloads involved are often overlooked. We point out that a key cause is the inability of callbacks to naturally simulate network events. We propose CoDES, a coroutine-based paradigm that enables complete network-operation simulation with sequential workflows and simpler code, improving productivity. Results show CoDES reduces code size and structural complexity by up to 62.3% and 82.6%, respectively. R EFERENCES [1] C. Li, A. Nasr-Esfahany, K. Zhao et al., “m3: Accurate flow-level performance estimation using machine learning,” in Proceedings of the ACM SIGCOMM 2024 Conference, 2024, pp. 813–827.
10
102470, 2022. [18] G. F. Riley and T. R. Henderson, The ns-3 network simulator. Springer, 2010, pp. 15–34. [19] S. Das, Z. Xiang, and L. Ren, “Asynchronous data dissemination and its applications,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021, pp. 2705–2721. [20] X. Luo, W. Shen, S. Mu, and T. Xu, “Depfast: Orchestrating code of quorum systems,” in 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022, pp. 557– 574. [21] E. Zamora-Gómez, P. Garcı́a-López, and R. Mondéjar, “Continuation complexity: A callback hell for distributed systems,” in Euro-Par 2015: Parallel Processing Workshops, S. Hunold, A. Costan, D. Giménez et al., Eds. Cham: Springer International Publishing, 2015, pp. 286– 298. [22] J. J. Dongarra, S. W. Otto, M. Snir et al., “An introduction to the mpi standard,” Communications of the ACM, vol. 18, p. 11, 1995. [23] J. J. Wilke and J. P. Kenny, “Using discrete event simulation for programming model exploration at extremescale: Macroscale components for the structural simulation toolkit (sst).” Sandia National Lab.(SNL-CA), Livermore, CA (United States), Tech. Rep., 2015. [24] G. Kim, C. Kim, J. Jeong et al., “Contention-based congestion management in large-scale networks,” in 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2016, pp. 1–13. [25] B. Belson, J. Holdsworth, W. Xiang, and B. Philippa, “A survey of asynchronous programming using coroutines in the internet of things and embedded systems,” ACM Transactions on Embedded Computing Systems (TECS), vol. 18, no. 3, pp. 1–21, 2019. [26] A. Dunkels, O. Schmidt, T. Voigt, and M. Ali, “Protothreads: Simplifying event-driven programming of memory-constrained embedded systems,” in Proceedings of the 4th international conference on Embedded networked sensor systems, 2006, pp. 29–42. [27] B. Belson, W. Xiang, J. Holdsworth, and B. Philippa, “C++ 20 coroutines on microcontrollers—what we learned,” IEEE Embedded Systems Letters, vol. 13, no. 1, pp. 9–12, 2020. [28] G. Nishanov, “Fibers under the magnifying glass,” ISO C++ Committee, Tech. Rep. P1364R0, Nov. 2018. [29] J. Schuchart, C. Niethammer, and J. Gracia, “Fibers are not (p) threads: The case for loose coupling of asynchronous programming models and mpi through continuations,” in Proceedings of the 27th European MPI Users’ Group Meeting, 2020, pp. 39–50. [30] I. Zhang, A. Raybuck, P. Patel et al., “The demikernel datapath os architecture for microsecond-scale datacenter systems,” in Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, 2021, pp. 195–211. [31] M. Welsh, D. Culler, and E. Brewer, “Seda: An archi-
tecture for well-conditioned, scalable internet services,” vol. 35, no. 5. ACM New York, NY, USA, 2001, pp. 230–243. [32] A. L. D. Moura and R. Ierusalimschy, “Revisiting coroutines,” ACM Transactions on Programming Languages and Systems (TOPLAS), vol. 31, no. 2, pp. 1–31, 2009. [33] M. Felleisen, “On the expressive power of programming languages,” Science of computer programming, vol. 17, no. 1-3, pp. 35–75, 1991. [34] S. Klabnik and C. Nichols, The Rust programming language. No Starch Press, 2023. [35] C. Lou, C. Chen, P. Huang et al., “Resin: a holistic service for dealing with memory leaks in production cloud infrastructure,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 109–125. [36] N. M. Josuttis, “The c++ standard library: a tutorial and reference,” 2012. [37] R. Von Behren, J. Condit, F. Zhou et al., “Capriccio: Scalable threads for internet services,” ACM SIGOPS Operating Systems Review, vol. 37, no. 5, pp. 268–281, 2003. [38] J. Launchbury, “A natural semantics for lazy evaluation,” in Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, 1993, pp. 144–154. [39] R. C. Martin, Clean code: a handbook of agile software craftsmanship. Pearson Education, 2009. [40] J. Hughes, “Why functional programming matters,” The computer journal, vol. 32, no. 2, pp. 98–107, 1989. [41] T. Mens, S. Demeyer, M. D’Ambros et al., “Analysing software repositories to understand software evolution,” Software evolution, pp. 37–67, 2008. [42] M. Madsen, O. Lhoták, and F. Tip, “A model for reasoning about javascript promises,” Proceedings of the ACM on Programming Languages, vol. 1, no. OOPSLA, pp. 1–24, 2017. [43] G. Bierman, C. Russo, G. Mainland et al., “Pause’n’play: Formalizing asynchronous c,” in European Conference on Object-Oriented Programming. Springer, 2012, pp. 233–257. [44] N. Jiang, D. U. Becker, G. Michelogiannakis et al., “A detailed and flexible cycle-accurate network-on-chip simulator,” in 2013 IEEE international symposium on performance analysis of systems and software (ISPASS). IEEE, 2013, pp. 86–96. [45] B. Pfaff, J. Pettit, T. Koponen et al., “The design and implementation of open vswitch,” in 12th USENIX symposium on networked systems design and implementation (NSDI 15), 2015, pp. 117–130. [46] S. Hemminger et al., “Network emulation with netem,” in Linux conf au, vol. 5, 2005, p. 2005. [47] O. Sefraoui, M. Aissaoui, M. Eleuldj et al., “Openstack: toward an open-source solution for cloud computing,” International Journal of Computer Applications, vol. 55, no. 3, pp. 38–42, 2012.
11