Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin Weqaar Janjua1,2,*
Eoin O’Connell
Mihai Penica
1 University of Limerick, Ireland
University of Limerick, Ireland
University of Limerick, Ireland
2 Packet Five Networks Ltd., Ireland
[email protected] [email protected]
arXiv:2609.15889v1 [cs.DC] 14 Sep 2026
Abstract
than continuously researcher-controlled machines (Duplyakin et al., 2019; Keahey et al., 2020). Cnuas is intended to broaden access to rack-scale AI/HPC systems for academic and industrial research and development. Within this context, it supports experimentation, learning and software development. Its goal is to expose and exercise device interfaces, protocols and interactions across the software stack, rather than reproduce the throughput or latency of physical hardware. This paper evaluates the implemented software functionality. Cnuas complements physical testbeds and hardware-assisted virtualization, rather than serving as a performance-equivalent replacement. This distinction includes single-root I/O virtualization (SR-IOV): supported NICs and GPUs expose PCI Express (PCIe) virtual functions (VFs), which can be assigned to virtual machines or made available as VF-backed devices to container workloads through host drivers. These paths retain a dependency on physical devices; Cnuas instead provides modifiable software models of the implemented device interfaces and protocols. A further motivation for the Open Rack v3 (ORV3) architecture (Open Compute Project Foundation, 2022) is to provide developers and engineers with a concrete mental model of the infrastructure underlying their workloads. Software interfaces often conceal the arrangement of compute blades, accelerators, network adapters, switches and shared power equipment. The Cnuas web interface places equipment and connections in a visual rack context; Figure 2 shows the underlying reference deployment. The intention is to help users relate application execution to device placement, data movement, communication paths and shared infrastructure. This is particularly relevant to distributed AI and HPC workloads, whose execution can span multiple accelerators and servers. The rack therefore serves both as an architectural basis for emulation and as a reference for understanding the system being programmed. Existing simulators primarily support performance and architectural analysis. Cycle-level GPU simulators produce timing estimates for a modeled architecture (Bakhoda et al., 2009; Khairy et al., 2020; Sun et al., 2019). Network simulators reproduce protocol behavior and queueing in a discrete-event framework (Riley and Henderson, 2010; Varga and Hornig, 2008); some also connect real software stacks to simulated networks (ns-3 Project, n.d.). Cnuas instead focuses on a guest-visible device boundary spanning the application, provider library, kernel driver and emulated adapter. For example, ibv_rc_pingpong runs through libibverbs and a kernel driver bound to an enumerated PCI device. This preserves the software-layer boundaries used with physical adapters.
Modern AI and HPC systems integrate accelerators, high-speed networks, and management controllers at rack scale. Developing software for this infrastructure typically requires access to scarce, costly hardware, while software abstractions can obscure how workloads depend on resources across servers and accelerators. This paper presents Cnuas, an open-source, experimental rack-scale emulation platform whose baseline architecture follows the Open Compute Project (OCP) Open Rack v3 specifications. Through functional emulation, it supports experimentation, learning and software development within academic and industrial research and development, rather than matching the throughput or latency of physical hardware. Its web-based user interface visualizes racks, devices and their interconnections to help developers build a system-level mental model of the infrastructure supporting their workloads. At its core, CnuasNIC and CnuasSwitch implement a guestvisible remote direct memory access (RDMA) adapter and a hostresident hybrid software switch supporting both RoCEv2 and native InfiniBand. The platform also provides a dedicated AI/ML accelerator (GPU) peer fabric and OpenBMC-based rack management with executable power supply and battery backup firmware over RS-485. These components support the study of device, driver, and firmware interfaces on commodity hosts. The accelerator software stack remains an early research prototype, and facility modeling with OpenUSD is an exploratory extension. The paper presents the architecture, interfaces, and bounded prototype results as a basis for community collaboration across the core platform and its extensions.
Keywords AI/HPC infrastructure emulation, Open Rack v3, OpenBMC, RDMA, accelerator emulation, exploratory facility modeling, virtual time
1
Introduction
AI rack software depends on hardware-specific interfaces. A device driver binds to a PCI function; a verbs provider uses its queue and doorbell layout. A subnet manager configures the fabric; rack firmware communicates with the power shelf over a sideband bus. These paths are normally exercised together after the hardware is installed. This delays rack-scale software integration and limits its availability for continuous integration. Procurement cost and lead time further restrict access to switches, adapters and accelerators. Shared facilities provide scheduled allocations rather ∗ Corresponding author: Weqaar Janjua, [email protected].
1
Janjua et al.
Cnuas1 uses functional device emulation for this purpose. Each guest-facing adapter or accelerator is a model presented over an emulated PCI bus to a guest running a stock kernel with a Cnuas driver module and otherwise standard user space. The software under study therefore crosses the same application, driver and device boundaries as it would on a physical system. Emulation reproduces the programming interface and protocol behavior, but it does not by itself reproduce physical timing; Section 6 describes the separate analytic timing model and its limitations. Cnuas can demonstrate what a queue pair is, how a subnet manager brings a fabric up, how a collective is decomposed, and how a kernel reaches an accelerator. Implemented capabilities and outstanding work are summarized in Table 1. Cnuas is being developed as an open-source software product and an extensible research framework. Its scope exceeds the capabilities currently implemented and evaluated. Existing components include research prototypes evaluated through focused functional tests; substantial parts of the wider architecture remain under development or are planned. The accelerator software stack is at an early experimental stage. Public release is intended to support collaborative development of the core platform and its extensions. Rack-level development also depends on management, power and cooling. An early facility-modeling prototype explores connecting emulated OpenBMC power-shelf telemetry to OpenUSD scenes (Alliance for OpenUSD, 2025) and simplified load calculations. Section 4 describes this exploratory extension. The contribution of this paper is threefold. (1) A paired RDMA device and fabric implementation: CnuasNIC presents a guest PCI adapter with Linux drivers and a standard verbs provider, while CnuasSwitch implements a host-resident hybrid soft RoCE and soft InfiniBand switch. The pair exposes both protocol paths within the emulated system, making device, driver and switch behavior available for joint development rather than proxying RDMA to a physical host adapter. (2) A vertically integrated rack architecture based on Open Rack v3 specifications that presents guest-visible RDMA and accelerator devices, separate network and accelerator fabrics, real OpenBMC images, executable PSU and BBU firmware on an RS-485 multi-drop bus. The architecture also provides an integration point for exploratory facility modeling. (3) A reproducible platform-validation workflow combining component tests and firmware demonstrations, with unmet environment requirements reported as explicit gates rather than successful results. The evaluation combines named component tests and recorded firmware demonstrations. These establish the behavior exercised; broader integration and application suitability require separate evaluation. Outstanding tests and unimplemented capabilities are identified throughout the paper.
2 Related Systems and Scope 2.1 Accelerator simulation GPGPU-Sim established detailed simulation of GPU workloads and remains the reference point for microarchitectural studies (Bakhoda et al., 2009). Accel-Sim extended the approach with a validated framework and broader front-end support (Khairy et al., 2020). MGPUSim addressed multi-accelerator systems and the interconnect between them (Sun et al., 2019), and Multi2Sim modeled combined CPU and GPU execution (Ubal et al., 2012). These tools answer questions about performance and architecture using traces or emulated instruction streams. Cnuas prioritizes guest device and driver interfaces rather than cycle-accurate microarchitectural timing. MIAOW is the closest open-RTL precedent at the accelerator block level. It implements an AMD Southern Islands GPGPU in RTL for low-level architecture experiments and OpenCL compatibility (Balasubramanian et al., 2015). MIAOW and Cnuas expose different experimental surfaces: open GPU RTL in the former and an integrated software-emulated rack platform in the latter.
2.2
Full-system simulation and emulation
gem5 provides full-system simulation including devices and operating system boot (Binkert et al., 2011), and FireSim accelerates cycle-exact scale-out simulation using FPGAs in the cloud (Karandikar et al., 2018). Both deliver fidelity that Cnuas does not attempt, at a cost in wall-clock time and, for FireSim, in infrastructure. QEMU supplies the machine emulation that Cnuas builds on (Bellard, 2005), and Renode provides the embedded simulation used for the management-plane peripherals (Antmicro, n.d.). Cnuas contributes device models rather than a new emulator. The design targets interactive execution of the integrated rack software stack on a workstation; the current evaluation is described in Section 7.
2.3
Shared testbeds
Emulab pioneered configurable network experimentation (White et al., 2002), and CloudLab and Chameleon provide bare-metal, reconfigurable resources to the research community (Duplyakin et al., 2019; Keahey et al., 2020). These support physical measurements but provide allocated rather than continuously researcher-controlled access. Cnuas supports locally controlled interface and integration work; physical testbeds remain necessary for hardware measurements. There is a second distinction beyond availability. Access to a physical AI or HPC testbed ordinarily permits experiments above the interfaces exposed by its installed accelerators; it does not give a user a modifiable implementation of proprietary accelerator RTL, internal firmware, math blocks, or fabric logic. Cnuas trades physical fidelity for a vertically open research surface in which a change can cross the device model, instruction set, compiler, kernel driver, runtime, libraries, fabrics and rack management, with facility modeling as an exploratory extension. This provides a basis for cross-layer co-design questions that a fixed cluster cannot, while the physical testbed remains necessary for timing, power, manufacturability, and compatibility measurements.
1 Pronounced “Knoo-us”; the name comes from the Irish Gaeilge word for “cluster”.
2
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
Table 1: Implementation status and evaluation scope of the experimental platform. Component
Existing implementation
Evaluation status
Planned or not established
RDMA and switch
Guest PCI adapter, kernel drivers, verbs provider, RoCEv2 and InfiniBand data paths, top-of-rack switch and integrated subnet manager
Research implementation; component checks do not establish complete guest integration, which requires a separate guest-based evaluation
CnuasGPU
Software-emulated accelerator with PCI, driver and runtime interfaces; standalone host modes
Early research component; GPU-specific evaluation is outside this paper’s scope
CnuasLink
Independent peer-fabric frames, discovery, forwarding and collective payload transport
Research implementation with host fabric and client tests
Rack management
Real OpenBMC images, QEMU/Renode controllers, six PSU and six BBU firmware instances, RS-485 Modbus RTU and Redfish telemetry OpenUSD scene generation, simplified load arithmetic and telemetry bridge
Firmware and management-interface tests within emulation
ETS bandwidth scheduling and further subnet/performance management planned; SA PathRecord, PMA/perfquery, external OpenSM interoperability and physical throughput not established Experimental FPGA design in progress; RTL implementation planned. CXL planned before UALink; detailed GPU evaluation reserved for a separate paper Further physical-link attachments planned; vendor-fabric compatibility and performance equivalence not established Additional devices planned; electrical protection and physical power-hardware equivalence not established
Deterministic event model, profiles, fitting and error reporting
Experimental analytic model; no physical calibration dataset
Facility modeling
Virtual time
2.4
Early exploratory prototype with software consistency checks
Software RDMA and fabric emulation
Scenario validation and reference comparisons required; finished twin, physical accuracy and operational suitability not established Target-specific calibration planned; prediction accuracy and live cross-emulator clock synchronization not established
device, which could itself be SoftRoCE. The device model did not implement a fabric or switch; InfiniBand management traffic needed an external multiplexer, and the device was deprecated in QEMU 8.2 and removed in 9.1. Cnuas terminates the transport inside the emulated devices and carries it over an emulated fabric, so the protocol behavior under study belongs to the model rather than to a host stack beneath it.
The Linux SoftRoCE driver implements the RoCE transport in software over an ordinary Ethernet interface (Linux RDMA Project, 2016), and SoftiWARP does the same for iWARP (Metzler et al., 2010). Both make verbs programming possible without an RDMA adapter. They terminate the transport in the host kernel rather than in an emulated PCI adapter and do not themselves model a switch or fabric. Host transport mechanisms, including TCP congestion control for SoftiWARP, are distinct from switch-level forwarding and fabric management. CnuasNIC is an emulated adapter behind a PCI function with its own kernel drivers and verbs provider. Its connection to CnuasSwitch exposes forwarding tables, subnet management and management datagrams for modification and study. QEMU’s pvrdma is a close precedent for presenting RDMA behind a PCI function. It implemented VMware’s paravirtual RDMA interface, registering base address registers and MSI-X vectors and binding the in-tree vmw_pvrdma driver in the guest (QEMU Project, 2024). It proxied the transport to the host: QEMU’s documentation describes it as acting “like a proxy between the Guest Driver and the host ibdevice interface”, mapping each guest queue pair and completion queue onto a resource requested from a host RDMA
2.5
Single-machine emulation of a cluster
Phantora studies distributed training on a single machine by intercepting accelerator and collective-library calls (Qin et al., 2026). It replaces behavior below those calls with a model for machinelearning system performance estimation. Cnuas instead retains the driver, provider, runtime and application above its emulated device boundary. Its current timing model produces analytic estimates, not validated predictions of physical-system performance. Phantora therefore addresses training-time estimation, while Cnuas emphasizes device-interface and cross-layer integration behavior.
3
Janjua et al.
2.6
Collective communication and programming models
backends. These requirements apply to the KVM-accelerated x86-64 rack deployment; other host architectures have not been validated.
The accelerator software stack follows the shape established by CUDA (Nickolls et al., 2008), by collective libraries such as NCCL (NVIDIA, n.d.-a), and by partitioned global address space messaging as standardized in OpenSHMEM (Chapman et al., 2010). ASTRA-sim models distributed training on simulated platforms and is the closest work in intent at the collective layer (Rashidi et al., 2020). Cnuas additionally provides a callable collective library whose payloads traverse its peer fabric.
2.7
3.2
3.2.1 CnuasNIC. CnuasNIC is a functional, software-emulated RDMA network adapter that speaks both RoCEv2 and native InfiniBand to the CnuasSwitch fabric. The cnuas-vnic QEMU PCIe device exposes memory-mapped control registers and interrupts to two guest Linux modules: cnuas_net supplies the network-device interface and cnuas_ib registers the RDMA device. A provider implements the rdma-core verbs application binary interface (ABI), allowing applications to use standard libibverbs calls through the Linux ib_uverbs interface. Implemented operations include send and receive, RDMA read and write, and atomic compare-and-swap and fetch-and-add. The adapter supports reliable connected (RC) and unreliable datagram (UD) queue pairs, shared receive queues and InfiniBand multicast. Multi-packet transfers honor the negotiated path maximum transmission unit. The invariant cyclic redundancy check (ICRC) is computed on transmission and validated on reception. Management queue pairs QP0 and QP1 support subnet management and general services, including management-datagram processing and the subnet management agent. Memory registration covers host memory and an experimental DMA-BUF import interface for peer-memory integration. RoCEv2 carries RDMA packets over UDP/IPv4 and Ethernet, using UDP destination port 4791. The native InfiniBand path instead carries a local routing header (LRH), base transport header (BTH) and InfiniBand opcodes, preserving its own addressing and management behavior without Ethernet/IP encapsulation. These are two protocol paths in the adapter, not alternative labels for the same Ethernet transport. The current CnuasNIC endpoint requires the emulated PCI function in a guest. Building its provider outside the rdma-core source tree does not remove the kernel-driver dependency. Its contribution is the modifiable device, driver and provider path connected to the emulated switch; the recorded integration coverage and remaining guest-environment gates are distinguished in Section 7.
Facility modeling
Warehouse-scale computing established the discipline of reasoning about a data center as one machine, including its power and cooling (Barroso et al., 2018), and power usage effectiveness is the standard efficiency metric (ISO/IEC, 2026). Facility digital twins are established practice in industry and in research, and OpenDT is a recent example that combines live telemetry with simulation to study data center performance and sustainability (Nicolae et al., 2026). Such twins are normally driven by traces or by telemetry from a physical site. Cnuas explores the use of emulated rack-firmware telemetry as an input to a campus scene. This early prototype is not evaluated as a substitute for a validated facility twin such as those studied in this literature.
2.8
Platform component families
The first four families implement the compute and communication data planes described below. CnuasBMC addresses rack management (Section 5); the early facility extension is discussed separately in Section 4.
The gap
Prior work provides mature or specialized artifacts at individual layers, and the layers are usually studied apart. The literature reviewed for this paper shows that the systems differ principally in where they place their modeling boundary and which research questions they address, as summarized in Table 2. No prior system identified in this review integrates guest-visible accelerator and RDMA devices, independent network and accelerator fabrics, executable Open Rack v3 management and power-shelf firmware, together with an extensible path toward facility modeling. Cnuas investigates this combination with an emulation boundary below ordinary guest software and above the modeled devices. Its contribution is the experimental rack-scale product, its integration interfaces and bounded prototype results.
3 System Architecture 3.1 Deployment model
3.2.2 CnuasSwitch. CnuasSwitch is a hybrid soft RoCE and soft InfiniBand RDMA switch. Its cnuas-vswitchd daemon runs directly on the host and presents ten logical ports: eight fabric ports, an Ethernet uplink and a console/observability port. One daemon serves both link layers, with Ethernet-only, InfiniBand-only or automatic classification selectable per fabric port. The Ethernet path models MAC-frame forwarding in the IEEE 802.3 Ethernet family (IEEE 802.3 Working Group, n.d.), including VLAN-aware source-address learning and destination lookup in a forwarding database (FDB). It carries ordinary Ethernet traffic as well as RoCEv2 packets. The native InfiniBand pipeline forwards LRH/BTH frames using destination local identifiers and a linear
Figure 1 shows the architecture and integration interfaces, not uniform maturity across its components. The reference deployment, shown in Figure 2, follows the Open Compute Project Open Rack v3 layout (Open Compute Project Foundation, 2022). Equipment height is expressed in Open Rack units (OU). Each rack holds two 1OU top-of-rack switches, a management host running the daemons, and eight 2-OU blades acting as compute nodes. Each blade is a guest virtual machine with an emulated RDMA adapter and one or more emulated accelerators. Two racks are modeled in the present tree, connected through an inter-rack fabric link. The host requirement is an x86-64 machine with AVX2, 16 GB of memory as a minimum with 64 GB recommended, and KVM enabled. AVX-512 is used when present for the accelerator compute 4
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
Table 2: Scope comparison with representative systems. “Physical” means the result comes from allocated hardware rather than a timing model. System
Modeling boundary
Guest device and driver
Fabric, rack and facility scope
Research focus
GPGPU-Sim / Accel-Sim (Bakhoda et al., 2009; Khairy et al., 2020) gem5 (Binkert et al., 2011)
GPU instruction and microarchitecture model
No guest PCI driver path
GPU and modeled interconnect, not rack management or facility
GPU performance and architecture estimates
Full-system machine model FPGA-hosted RTL model
Machine and modeled interconnect; no Open Rack management or facility twin Cycle-exact scale-out system; no facility twin
Architecture and full-system simulation
FireSim (Karandikar et al., 2018) SoftRoCE (Linux RDMA Project, 2016) pvrdma (QEMU Project, 2024) Phantora (Qin et al., 2026)
Model-dependent guest OS and devices Workload runs on modeled hardware
Host-kernel RDMA transport
No emulated PCI device
Functional verbs and transport on the host
Guest PCI proxy to host RDMA
In-tree guest driver
Accelerator and collective library interception No model at the device boundary
No driver or device path
Host network transport; no modeled switch, rack management or facility Host-backed RDMA proxy; no implemented switch, rack or facility Modeled accelerator cluster above the library boundary
Physical devices and drivers
Allocated physical testbed; no user-modifiable facility twin
Physical compatibility and performance
Guest device, fabric and rack-management models; exploratory facility extension
Guest PCI devices, Cnuas drivers and standard user space
Two data fabrics, Open Rack v3 management and early facility prototype
Bounded functional results and analytic virtual time; not product readiness or validated physical performance
CloudLab / Chameleon (Duplyakin et al., 2019; Keahey et al., 2020) Cnuas
Cycle-exact scale-out simulation
Guest access to host RDMA
Machine-learning system performance estimation
Table 3: Platform component families and roles. See Table 1 for implementation and maturity. Component
Function
CnuasNIC
Guest PCI RDMA adapter supporting RoCEv2 and native InfiniBand, with Linux network/RDMA drivers and a standard verbs provider Host-resident, ten-port hybrid soft RoCE and soft InfiniBand switch; Ethernet forwarding, selected DCB mechanisms and integrated subnet management Experimental software-emulated accelerator with device memory, guest PCI attachment, and driver and runtime interfaces Eight-port accelerator-to-accelerator fabric, modeling a peer interconnect distinct from the network fabric OpenBMC-based sled, switch and rack management, connected to emulated PSU and BBU firmware over an RS-485 Open Rack v3 power shelf Exploratory OpenUSD scene generation, simplified load calculations and connections to emulated CnuasBMC telemetry
CnuasSwitch CnuasGPU CnuasLink CnuasBMC Facility prototype
5
Janjua et al.
Experimental Cnuas architecture and device-boundary emulation seam Workloads and control Exploratory OpenUSD view
Verbs applications
AI applications
Cnuas CLI / REST Redfish
Guest and host software Verbs provider
CnuasGPU software stack Early experimental implementation
OpenBMC firmware
Prototype telemetry bridge
Cnuas emulation seam: real software above; modeled platform below Alternative boundaries: library calls | host kernel | PCI proxy | whole machine
Emulated devices and firmware CnuasNIC PCI device
CnuasGPU Emulated accelerator
BMC processors QEMU / Renode
6 PSU + 6 BBU firmware requests / telemetry
Independent fabrics and sideband bus CnuasSwitch RoCEv2 / InfiniBand
CnuasLink peer fabric
ORv3 RS-485 Modbus RTU
Early facility prototype Scene, simplified loads and emulated telemetry
Figure 1: The experimental Cnuas architecture and its emulation seam. Ordinary software above the seam exercises the modeled platform below it. The accelerator stack is early-stage and the facility extension exploratory. Representative alternative boundaries correspond to the systems compared in Table 2. forwarding table (LFT), retaining a separate addressing and management path. Selected data center bridging (DCB) mechanisms are modeled for the Ethernet path: priority-based flow-control pause handling (PFC, IEEE 802.1Qbb) (IEEE 802.1 Working Group, n.d.-a) and configuration of enhanced transmission selection (ETS, IEEE 802.1Qaz) (IEEE 802.1 Working Group, n.d.-b). ETS traffic-class mappings and bandwidth settings are exposed through management; bandwidthenforcing scheduling remains development work. IPv4 explicit congestion notification (ECN) marking follows the field semantics of IETF RFC 3168 (Ramakrishnan et al., 2001); ECN is distinct from the IEEE DCB amendments. The models cover these selected protocol mechanisms, not complete IEEE conformance or physical-link behavior. The integrated subnet manager and subnet management agent support subnet sweeps, directed-route management packets, localidentifier assignment and route-table programming. This brings up the modeled InfiniBand fabric without a separate OpenSM process. A JSON management interface exposes port, forwarding, flowcontrol and subnet-manager state, with telemetry for observability. Fabric ports use UNIX-domain SOCK_SEQPACKET sockets. QEMU devices and host processes implementing the port framing can
therefore attach to the same switch. Host-only clients support protocol and forwarding experiments without virtual machines; they do not by themselves expose a Linux verbs device. Together with CnuasNIC, this separates host-resident fabric development from guest device/driver integration while preserving both RoCEv2 and native InfiniBand paths. 3.2.3 CnuasGPU. CnuasGPU supplies the experimental accelerator component of the rack. It presents a software-emulated PCI device with device memory. In the guest, the Linux driver exposes a character device, /dev/cnuasgpuN, where 𝑁 is the device index. CnuasDev provides the device-access interface, and CnuasRT supplies the application runtime above it. Its accelerator-peer connection uses CnuasLink, separately from the CnuasNIC network path. Standalone Soft-GPU modes retain these application interfaces on the host CPU without QEMU or a guest. The cnuasgpu_host kernel module exposes /dev/cnuasgpu_hostN, backed by kernel-owned host memory, for applications needing a Linux device node. The in-process mode instead uses an arena in the calling process and requires neither a kernel module nor a device node. Default discovery prefers the guest PCI device, then the host character device, and finally the in-process mode.
6
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
Cnuas, Two-Rack TOR Aggregation OCP ORv3 inter-TOR fabric link RoCE/IB vm-a* ⇄ vm-b* cross-rack traffic
Cnuas-R1 · ORv3 · 24 OU
Cnuas-R2 · ORv3 · 24 OU
TOR · 10 ports · DCB · FDB-learning
TOR · 10 ports · DCB · FDB-learning
CnuasSwitch A · cnuas-vswitchd
CnuasSwitch B · cnuas-vswitchd
CnuasLink A · cnuasgpu-link-switchd
CnuasLink B · cnuasgpu-link-switchd
rack-local GPU peer fabric
rack-local (no inter-rack GPU cable)
Mgmt host · cnuas1
Mgmt host · cnuas2
runs CnuasSwitch A CnuasLink A daemons
runs CnuasSwitch B CnuasLink B daemons
vm-a0
10.42.0.1
vm-b0
10.42.1.1
vm-a1
10.42.0.2
vm-b1
10.42.1.2
vm-a2
10.42.0.3
vm-b2
10.42.1.3
vm-a3
10.42.0.4
vm-b3
10.42.1.4
vm-a4
10.42.0.5 · idle
vm-b4
10.42.1.5 · idle
vm-a5
10.42.0.6 · idle
vm-b5
10.42.1.6 · idle
vm-a6
10.42.0.7 · idle
vm-b6
10.42.1.7 · idle
vm-a7
10.42.0.8 · idle
vm-b7
10.42.1.8 · idle
cable management
cable management
Power · 2 1600W 11 PSU 1
Power · 2 1600W 11
PSU 2
CnuasSwitch RoCE/IB port
PSU 1
CnuasLink GPU peer) port
PSU 2
Inter-TOR fabric port
active
idle
Figure 2: The Cnuas reference deployment, two racks modeled on the Open Compute Project Open Rack v3. Each rack carries two 1-OU top-of-rack switches, one for the RoCE and InfiniBand fabric and one for the accelerator fabric, a management host, and eight 2-OU blades. Every blade is a guest with an emulated RDMA adapter and one or more emulated accelerators; the scope of the current evaluation is described in Section 7. Experimental FPGA design work is in progress; RTL implementation is planned. A separate CnuasGPU paper is planned to describe its architecture, software stack, numerical validation and performance evaluation. Compute Express Link (CXL) integration is planned ahead of UALink. CXL specifies cache-coherent connectivity among processors, memory and accelerators (Compute Express Link Consortium, n.d.). The planned Cnuas work addresses host/device memoryinterconnect experiments, beginning with selection of a specification revision and device profile, assessment of QEMU and Linux support, and a reproducible validation path. UALink remains a later accelerator-to-accelerator attachment. This ordering is a development priority; the two interconnects serve different roles. Both remain planned extensions. PCIe provides the existing guest device attachment, while CnuasLink carries the current modeled accelerator-peer traffic.
3.2.4 CnuasLink. Accelerator-to-accelerator traffic uses a separate fabric with its own switch daemon, frame format, forwarding database, and discovery protocol, modeling the peer interconnect of a multi-accelerator node rather than reusing the network fabric. Keeping the two fabrics apart reflects how such systems are actually built and allows their control and data paths to be changed independently.
3.3
Control plane
One command surface drives the switch fabric, the accelerator fabric, the accelerators, the adapters, and the lifecycle of the guests, and the same service layer is exposed as a REST interface for automation. Component daemons are reached over JSON on UNIX sockets. The uniformity avoids a collection of unrelated component tools and gives automation one control surface. 7
Janjua et al.
Host and guest data paths QEMU guest A
Application
Provider + runtime
NIC + GPU drivers
CnuasNIC + GPU NIC socket
CnuasSwitch RoCEv2 / InfiniBand
GPU socket
CnuasLink accelerator peer frames
NIC socket
GPU socket
QEMU guest B
Application
Provider + runtime
NIC + GPU drivers
CnuasNIC + GPU
Figure 3: Host and guest data paths. Applications cross ordinary userspace, kernel-driver and emulated-device boundaries. RoCEv2 and InfiniBand traffic uses CnuasSwitch, while accelerator peer traffic uses the independent CnuasLink fabric. A web-based frontend complements these programmatic interfaces with rack and switch-chassis views. It displays equipment placement, inter-rack links and switch-port status, giving users a visual context for inspecting the modeled system.
4
check specified arithmetic, configuration handling, scene structure and telemetry processing. Engineering or operational use requires physical and scenario validation beyond the present work. The early prototype will be made available for research and community development, with further development and validation continuing after release.
Exploratory Facility Modeling of the Hyperscale Data Center
5
Facility modeling with OpenUSD (Alliance for OpenUSD, 2025) and NVIDIA Isaac Sim (NVIDIA, n.d.-b) is an early-stage exploratory direction. The initial Cnuas Facility Twin prototype investigates scene generation, simplified load calculations and connections to emulated rack telemetry. The prototype describes a campus declaratively, from buildings and data halls to rack rows, support rooms, plant and landscaped areas. It generates OpenUSD geometry for exploration in Isaac Sim. Figure 4 illustrates this scene-generation work. The load arithmetic sums configured rack ratings and applies an assumed power usage effectiveness (PUE) multiplier. PUE is formally an energy ratio over a common measurement period (ISO/IEC, 2026); its use here as a power multiplier is a simplifying scenario assumption. Heat-load and plant-count calculations likewise use simplified assumptions. An experimental bridge reads the emulated Open Rack v3 shelf over Modbus RTU and writes PSU measurements and BBU state into OpenUSD attributes and time samples. This explores a connection between firmware and a scene. Substantial scenario-specific validation remains necessary. This includes checking geometry and equipment assumptions, comparing electrical and thermal behavior with appropriate reference data, exercising telemetry timing and fault conditions, and evaluating calibrated models on held-out scenarios. Existing software tests
CnuasBMC Rack and Platform Management
Rack management connects the compute nodes to the power architecture and sideband buses of Open Rack v3 (Open Compute Project Foundation, 2022). CnuasBMC reproduces these interfaces so that power-control and telemetry workflows can be exercised with the rest of the emulated rack.
5.1
The OpenBMC port
CnuasBMC is the Cnuas OpenBMC port (OpenBMC Project, n.d.). Its meta-cnuas Yocto layer defines three machines: sled, top-ofrack switch and rack management controller. Each has a Cnuas device tree and branded firmware image. The images boot a real Linux kernel and bootloader on an emulated ASPEED AST2600. The userspace is OpenBMC rather than a replacement: sensors are published on D-Bus and served by bmcweb over Redfish, inventory comes from Entity Manager, host power passes through the normal power-control service, the managed console uses Serial over LAN, and IPMI answers over the network. Redfish and IPMI power requests drive GPIO transitions that control a separate QEMU blade and receive power-good and POST state in return.
8
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
Cnuas Ridge AI Campus
1,920 racks in 16 data halls across 3 buildings, 1000 by 800 metre site
DC3, E-shaped building DH5
DH6
DH3
DH4
Utility substation, 230 kV to 34.5 kV
DC3 heat rejection yard, 29 units
Cooling water storage
DC3 generator yard, 38 at 3 MW
DH1
DH2
DC2, H-shaped building DH4
DH5
DC2 heat rejection yard, 29 units
DH6
DC2 generator yard, 38 at 3 MW
DH1
DH2
DH3
Administration and operations building
DC1 heat rejection yard, 20 units
DC1, linear building DH2
DH3
DH4
DC1 generator yard, 26 at 3 MW
Data hall
Support room
Electrical plant
Staff parking
Mechanical plant
Rack
Rack emulated by Cnuas
Retention pond
Planting
Figure 4: Illustrative campus layout from the exploratory facility prototype. Equipment placement and plant representations are provisional.
5.2
RS-485 multi-drop power-shelf bus
The cnuas-rackmond master sweeps the twelve nodes, converts the registers to engineering units and publishes 56 D-Bus sensor objects for power, voltage, current, temperature, fan speed and battery state of charge. bmcweb serves those objects through the unmodified Redfish chassis sensor collection. The exploratory bridge in Section 4 can consume measurements originating in firmware responding to a Modbus request. This provenance distinguishes the data from display-only values. Physical accuracy and facilityscenario validation remain separate research requirements.
The emulated RS-485 segment preserves a shared-bus topology: one master addresses multiple peripheral nodes, every node receives each request, and only the addressed node replies. A Cnuas Renode plugin models that shared wire and exposes one TCP master endpoint to the AST2600 UART. The power peripherals run as Cortex-M machines in Renode (Antmicro, n.d.), while QEMU runs OpenBMC. This split preserves the multi-drop topology that a QEMU serial character device alone cannot express. The emulated shelf contains six PSU firmware instances at addresses 0xC0 through 0xC5 and six BBU firmware instances at 0x40 through 0x45. One portable firmware image uses straps to select role, slot and address for each of the twelve Cortex-M instances. They speak Modbus RTU at 19,200 baud using register maps from Meta rackmon (Meta Platforms, 2026). Framing, CRC-16, address filtering, holding-register reads, protected writes, exception responses, PSU load behavior, mains loss, battery discharge and fault state all execute in firmware on the addressed node.
6
Virtual Time and Calibration Model
Timing analysis is a supplementary research capability, separate from Cnuas’s primary role as a functional platform for academic and industrial research and development. Functional emulation reproduces interface and protocol behavior. Physical performance models instead estimate timing. Host wall-clock time does not provide a reliable mapping between them: dynamic translation, host scheduling, acceleration mode and workload on the development 9
Janjua et al.
Rack management and exploratory facility telemetry Control plane
management
bmcweb Redfish / IPMI
OpenBMC D-Bus 56 sensors live sensors
cnuas-rackmond Modbus master
Experimental telemetry bridge
requests / replies
6 PSU firmware
RS-485 multi-drop 19,200 baud
OpenUSD time samples
6 BBU firmware
Early facility prototype Scene and simplified load arithmetic
Figure 5: Rack-management interfaces and the exploratory facility-telemetry path. Emulated PSU and BBU readings pass through normal OpenBMC sensor services; the scene connection is an early prototype. machine all affect it. QEMU instruction counting supplies a deterministic guest clock (Bellard, 2005; QEMU Project, n.d.), but an instruction count does not state how long PCIe, DMA, a device queue, accelerator memory, a fabric hop, an interrupt or contention would take. Cnuas includes the Cnuas Calibrated Virtual-time and Performance Model. A versioned profile gives each component a fixed cost and rates for the resources it consumes: instructions, cycles, bytes, packets and operations. Resources may overlap, in which case the bottleneck term sets the dynamic service time, or execute serially. An integer-picosecond discrete-event scheduler assigns work to stable component lanes and records submission, queue, start and completion times. Repeated inputs produce an identical event trace. The initial profile covers the guest CPU, PCIe, CnuasNIC, CnuasSwitch, CnuasGPU compute and memory, CnuasLink, the CnuasBMC RS-485 segment and the exploratory facility-update cadence. For overlapping resources the uncalibrated service time follows a resource-bottleneck, or roofline-style, model (Williams et al., 2009): 𝐼 𝐶 8𝐵 𝑃 𝑂 , , , , . (1) 𝑡𝑎 = 𝑡 0 + max 𝑅𝐼 𝑓 𝑅𝐵 𝑅𝑃 𝑅𝑂 Here 𝑡 0 is fixed latency; 𝐼 , 𝐶, 𝐵, 𝑃 and 𝑂 are respectively instruction, cycle, byte, packet and operation counts. Rates 𝑅𝐼 , 𝑓 , 𝑅𝐵 , 𝑅𝑃 and 𝑅𝑂 are measured in instructions/s, cycles/s, bits/s, packets/s and operations/s, respectively; the factor eight converts bytes to bits. The maximum is used because overlapping resources complete when their slowest required resource completes. For resources
configured as serial, the scheduler adds their service times instead. A target-specific calibration then uses ordinary least squares to fit the affine correction 𝑡𝑝 = max(0, 𝛼𝑡𝑎 + 𝛽)
(2)
to training measurements, where 𝛼 is a scale factor, 𝛽 is a constant offset and the outer maximum prevents a negative predicted duration. A separate validation operation reports mean absolute error, root mean squared error, mean absolute percentage error and maximum absolute error on held-out observations. The target name and fitted coefficients become part of the component profile. This design follows the general requirement that a performance model be validated rather than assumed (Karandikar et al., 2018; Khairy et al., 2020), while extending one event model across the network, accelerator, management and facility domains. The implementation has 46 automated cases covering unit conversion, serial and overlapping resources, queue contention, parallel lanes, deterministic replay, strict profile loading, QEMU icount mapping, fitting and held-out error. These cases cover the listed model behaviors, not prediction accuracy for a physical target. The shipped values that are not fixed by protocol are labeled modeling assumptions. No physical calibration dataset is supplied, so no measured prediction error is reported. The current implementation evaluates explicit pipelines offline, emits a deterministic QEMU TCG and icount configuration, and maps supplied instruction counts. It does not synchronize the live clocks of QEMU, Renode and the host switch daemons. A fixed conversion such as ten virtual packets per
10
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
7.3
second to one physical packet per second is therefore not portable or supported by the present results.
7 Evaluation 7.1 Methodology The evaluation uses a validation matrix to associate each component with its test coverage and recorded results. A passing test establishes only the behavior exercised under its recorded configuration. Tests requiring unavailable devices or environments are reported as skipped, with the missing requirements recorded. They are not counted as passes.
7.2
Results
Table 4 summarizes the platform-related suites selected from a recorded run on a single workstation. The automated test runner records host and tool versions, exact commands, exit codes, durations, raw logs, JUnit records and unavailable environment requirements. It generates CSV, Markdown and LATEX summaries from these records and includes SHA-256 checksums for file-integrity verification. The Python suites cover the control plane, tooling, facility prototype, calibrated virtual-time model, switch data plane, and end-toend RDMA paths. CnuasBMC adds native host checks for its Open Rack v3, rack-controller, and front-panel firmware.
8
Cases
Passed
Skipped
70
45
25
52 123
52 123
0 0
66 46 18
66 46 18
0 0 0
Total Python
375
350
25
CnuasBMC Open Rack v3 firmware CnuasBMC rack controller CnuasBMC front-panel firmware
177 45 98
177 45 98
0 0 0
Superproject, switch data plane and end-to-end RDMA Control plane Build, image and guest lifecycle tooling Facility prototype software checks Calibrated virtual-time model Accelerator fabric switch and client
Limitations
Table 1 summarizes the component-level limitations. The following limitations govern interpretation of the results. Research maturity. Cnuas is an experimental software product. The accelerator stack is early-stage; substantial development across the wider platform remains in progress or planned. General reliability, usability and production readiness require broader evaluation. Physical-performance prediction. Section 6 demonstrates deterministic virtual time, queueing semantics, versioned profiles, calibration fitting, and held-out error calculation. The supplied profile has not been fitted to measurements from a named physical target. Its durations are therefore analytic estimates. Physical-target prediction accuracy remains to be evaluated. Executable scale. The repository’s reference topology defines two racks with eight blade slots each. The current recorded host measurement boots one blade, and the host test run does not include full two-node RDMA execution. Concurrent execution of all sixteen blades remains to be demonstrated. Facility modeling. The facility extension is an exploratory prototype. Scenario-specific reference comparisons, calibration and fault studies remain necessary to assess physical accuracy and engineering suitability. Advanced InfiniBand management. The integrated subnet manager assigns local identifiers and programs the implemented fabric routes. Subnet Administration PathRecord responses, Performance Management Agent support for perfquery, and interoperability with an external OpenSM instance are outside the present validated surface.
Table 4: Recorded platform test results. Cases requiring unavailable devices, guests or lab setups are reported as skipped. Suite
Lab Testbed Specifications
The lab testbed comprises two Lenovo ThinkStation P520 workstations. Each workstation has an Intel Xeon W-2133 CPU at 3.60 GHz with six physical cores and twelve hardware threads, 128 GB of DDR4 memory and a 1 TB SSD. Each CPU core has 32 KiB of L1 data cache, 32 KiB of L1 instruction cache and 1 MiB of L2 cache, giving aggregate L1 data, L1 instruction and L2 capacities of 192 KiB, 192 KiB and 6 MiB per workstation. The shared L3 cache is reported as 8.3 MiB. The processor supports AVX-512F, AVX-512DQ, AVX-512CD, AVX-512BW and AVX-512VL. Each workstation is equipped with a Mellanox MCX354A-FCCT ConnectX-3 Pro adapter on a PCIe x8 interface. The adapter provides two QSFP ports supporting FDR InfiniBand and 40-Gigabit Ethernet.
9 The facility cases check software behavior within the prototype; they are not physical or scenario validation. Test counts come from JUnit testcase records or totals emitted by native test programs. Build-only checks are reported through their commands, exit status and logs, without testcase counts. Full two-node RDMA testing requires a guest environment and was not executed in this host run. The run manifest records its prerequisites and reproduction commands.
Conclusion
This paper presents Cnuas as an experimental contribution to rackscale AI/HPC emulation: an open-source software product under development, with an extensible architecture, explicit interfaces and bounded prototype results. Existing device, fabric and rackmanagement implementations provide a basis for further work. The primary objective is accessible functional experimentation with the AI/HPC software stack for academic and industrial research and development, rather than parity with hardware execution speed. The accelerator software stack remains early-stage, and facility modeling is an exploratory extension. 11
Janjua et al.
CnuasNIC and CnuasSwitch are central to this contribution: they place the dual RoCEv2/native-InfiniBand adapter and hybrid software switch within one modifiable rack environment, while retaining the Linux device, driver and verbs interfaces used by applications. The intended development model is collaborative, allowing community contributions to the Cnuas core, accelerator stack and extensions after public release. The early facility prototype will be made available in that spirit. Further work includes full-rack functional integration, characterization of host resource use and broader reliability testing. Component-specific follow-on papers are planned to describe four parts of this architecture in greater depth: CnuasSwitch and its RoCEv2 and native InfiniBand data paths; CnuasGPU and its accelerator architecture and software stack, including planned CXL and later UALink integration; CnuasNIC and its RDMA, InfiniBand, and RoCEv2 device, driver, and provider implementations; and CnuasLink as the separate accelerator peer fabric. Planned communication-software work includes assessing and integrating Berkeley’s UCCL (UC Berkeley Sky Computing Lab, n.d.) over Cnuas’s emulated RDMA interfaces, starting with transport compatibility and bounded transfer and collective-correctness experiments. UCCL support remains a future integration target, distinct from the existing CnuasCCL library.
maintainer trees, QEMU’s qemu-devel process, and OpenBMC Gerrit, respectively. The versioned artifact carries the evaluated component patches, so their public availability does not depend on upstream acceptance. Product and company names used in this paper are the property of their respective owners and are used for identification only. Mention of a product or specification does not imply affiliation, endorsement, certification or conformance.
References Alliance for OpenUSD (2025) OpenUSD Core Specification, Version 1.0. Available at: https://aousd.org/usd-core-specification/ (accessed 12 September 2026). Antmicro (n.d.) Renode, open source simulation framework for complex embedded systems. Available at: https://renode.io/ and https: //github.com/renode/renode (accessed 12 September 2026). Bakhoda, A., Yuan, G.L., Fung, W.W.L., Wong, H. and Aamodt, T.M. (2009) ‘Analyzing CUDA workloads using a detailed GPU simulator’, IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 163–174. https://doi.org/10. 1109/ISPASS.2009.4919648. Balasubramanian, R., Gangadhar, V., Guo, Z., et al. (2015) ‘Enabling GPGPU low-level hardware explorations with MIAOW: an opensource RTL implementation of a GPGPU’, ACM Transactions on Architecture and Code Optimization, 12(2), Article 21, 25 pages. https://doi.org/10.1145/2764908. Barroso, L.A., Hölzle, U. and Ranganathan, P. (2018) The Datacenter as a Computer: Designing Warehouse-Scale Machines. 3rd edn. Morgan and Claypool. Bellard, F. (2005) ‘QEMU, a fast and portable dynamic translator’, USENIX Annual Technical Conference, FREENIX Track, pp. 41–46. Binkert, N., Beckmann, B., Black, G., et al. (2011) ‘The gem5 simulator’, ACM SIGARCH Computer Architecture News, 39(2), pp. 1–7. https://doi.org/10.1145/2024716.2024718. Chapman, B., Curtis, T., Pophale, S., et al. (2010) ‘Introducing OpenSHMEM: SHMEM for the PGAS community’, Fourth Conference on Partitioned Global Address Space Programming Model (PGAS), pp. 1–3. https://doi.org/10.1145/2020373.2020375. Compute Express Link Consortium (n.d.) About CXL. Available at: https://computeexpresslink.org/about-cxl/ (accessed 13 September 2026). Duplyakin, D., Ricci, R., Maricq, A., et al. (2019) ‘The design and operation of CloudLab’, USENIX Annual Technical Conference, pp. 1–14. https://www.usenix.org/conference/atc19/presentation/ duplyakin. IEEE 802.1 Working Group (n.d.-a) 802.1Qbb—Priority-based Flow Control. Available at: https://1.ieee802.org/dcb/802-1qbb/ (accessed 14 September 2026). IEEE 802.1 Working Group (n.d.-b) 802.1Qaz—Enhanced Transmission Selection. Available at: https://1.ieee802.org/dcb/802-1qaz/ (accessed 14 September 2026). IEEE 802.3 Working Group (n.d.) IEEE 802.3 Ethernet Working Group. Available at: https://www.ieee802.org/3/ (accessed 14 September 2026).
Acknowledgements AI tools assisted in the development and refinement of this work under the Linux Kernel AI Coding Assistants Policy (Linux Kernel Documentation, 2026). The lead author independently verified all outputs and assumes full responsibility for the final design, analysis, and written content.
Software and Data Availability Project site, documentation, component datasheets and validation matrix: https://cnuas.io The project website also provides an illustrative walkthrough of the early facility prototype. The public source-repository location will be announced there after release review. The early-stage facility prototype will be made available for experimental use and community development. The versioned upstream contribution artifact is archived on Zenodo at https://doi.org/10.5281/zenodo.22650437. It contains the evaluated component patches and associated provenance, licensing, citation, and integrity metadata, but not the complete Cnuas source repository or evaluation dataset. Upstream submissions remain pending. They comprise the CnuasNIC Linux netdev and RDMA drivers and UAPI identifier described in Section 3.2.1; QEMU models for CnuasNIC and CnuasGPU, together with the ASPEED GPIO and CMIS optical-module support used by CnuasBMC, described in Sections 3.2.1, 3.2.3, and 5; and the CnuasBMC OpenBMC machine layer, rack-management service, and Entity Manager configuration described in Section 5. Their intended review paths are the Linux networking and RDMA
12
Cnuas: A Software-Defined AI/HPC Rack-scale Emulation Platform and Hyperscale Data Center Facility Twin
ISO/IEC (2026) ISO/IEC 30134-2:2026, Information technology — Data centres key performance indicators — Part 2: Power usage effectiveness (PUE). 2nd edn. https://webstore.iec.ch/en/publication/ 111538. Karandikar, S., Mao, H., Kim, D., et al. (2018) ‘FireSim: FPGAaccelerated cycle-exact scale-out system simulation in the public cloud’, ACM/IEEE International Symposium on Computer Architecture (ISCA), pp. 29–42. https://doi.org/10.1109/ISCA.2018.00014. Keahey, K., Anderson, J., Zhen, Z., et al. (2020) ‘Lessons learned from the Chameleon testbed’, USENIX Annual Technical Conference, pp. 219–233. https://www.usenix.org/conference/atc20/ presentation/keahey. Khairy, M., Shen, Z., Aamodt, T.M. and Rogers, T.G. (2020) ‘AccelSim: an extensible simulation framework for validated GPU modeling’, ACM/IEEE International Symposium on Computer Architecture (ISCA), pp. 473–486. https://doi.org/10.1109/ISCA45697.2020. 00047. Linux Kernel Documentation (2026) AI Coding Assistants. Available at: https://docs.kernel.org/process/coding-assistants.html (accessed 12 September 2026). Linux RDMA Project (2016) RXE, software RDMA over Ethernet (SoftRoCE). Mainlined in Linux 4.8. Documentation available at: https://github.com/linux-rdma/rdma-core/blob/master/ Documentation/rxe.md (accessed 12 September 2026). Meta Platforms (2026) OpenBMC rackmon: ORV3 PSU and BBU register-map configurations and serial Modbus implementation. Matching upstream snapshot, commit 37f00cca103f5f6305e2859d2459d5ee96a95e82, 8 July. Available at: https://github.com/facebook/openbmc/tree/ 37f00cca103f5f6305e2859d2459d5ee96a95e82/common/recipescore/rackmon2/rackmon. Metzler, B., Frey, P. and Trivedi, A. (2010) ‘SoftiWARP, project update, a software iWARP driver for OpenFabrics’, OpenFabrics Alliance Workshop, Sonoma, CA. Presentation available at: https://www.openfabrics.org/downloads/Media/Sonoma2010/ Sonoma_2010_Wednesday_softiwarp_2010.pdf. Nickolls, J., Buck, I., Garland, M. and Skadron, K. (2008) ‘Scalable parallel programming with CUDA’, ACM Queue, 6(2), pp. 40–53. https://doi.org/10.1145/1365490.1365500. Nicolae, R., van der Toorn, J., Kraniti, S., Liu, H. and Iosup, A. (2026) ‘OpenDT: exploring datacenter performance and sustainability with a self-calibrating digital twin’, Companion of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE Companion), pp. 142–147. https://doi.org/10.1145/3777911. 3800634. ns-3 Project (n.d.) Tap Bridge. ns-3 Model Library. Available at: https://www.nsnam.org/docs/models/html/tap.html (accessed 12 September 2026). NVIDIA (n.d.-a) NVIDIA Collective Communication Library (NCCL) user guide. Online documentation. Available at: https://docs. nvidia.com/deeplearning/nccl/user-guide/docs/ (accessed 12 September 2026). NVIDIA (n.d.-b) NVIDIA Isaac Sim Documentation. Available at: https://docs.isaacsim.omniverse.nvidia.com/latest/ (accessed 12 September 2026). Open Compute Project Foundation (2022) Open Rack Base Specification Version 3, Revision 1.0. Available at: https://www.
opencompute.org/wiki/Open_Rack/SpecsAndDesigns. OpenBMC Project (n.d.) OpenBMC documentation. The Linux Foundation. Available at: https://github.com/openbmc/docs (accessed 12 September 2026). QEMU Project (2024) Paravirtualized RDMA device (pvrdma). Present in QEMU from 2.12, deprecated in 8.2 and removed in 9.1. Archived documentation from the QEMU 9.0 release series available at: https://github.com/qemu/qemu/blob/v9.0.0/docs/pvrdma. txt. QEMU Project (n.d.) TCG instruction counting. QEMU system emulator documentation. Available at: https://www.qemu.org/docs/ master/devel/tcg-icount.html (accessed 12 September 2026). Qin, J., Chen, J., Kong, X., et al. (2026) ‘Phantora: maximizing code reuse in simulation-based machine learning system performance estimation’, 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 1809–1825. https: //www.usenix.org/conference/nsdi26/presentation/qin. Ramakrishnan, K., Floyd, S. and Black, D. (2001) The Addition of Explicit Congestion Notification (ECN) to IP. RFC 3168, September. https://doi.org/10.17487/RFC3168. Rashidi, S., Sridharan, S., Srinivasan, S. and Krishna, T. (2020) ‘ASTRA-SIM: enabling SW/HW co-design exploration for distributed DL training platforms’, IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 81–92. https://doi.org/10.1109/ISPASS48437.2020.00018. Riley, G.F. and Henderson, T.R. (2010) ‘The ns-3 network simulator’, in Modeling and Tools for Network Simulation. Springer, pp. 15–34. https://doi.org/10.1007/978-3-642-12331-3_2. Sun, Y., Baruah, T., Mojumder, S.A., et al. (2019) ‘MGPUSim: enabling multi-GPU performance modeling and optimization’, ACM/IEEE International Symposium on Computer Architecture (ISCA), pp. 197–209. https://doi.org/10.1145/3307650.3322230. Ubal, R., Jang, B., Mistry, P., Schaa, D. and Kaeli, D. (2012) ‘Multi2Sim: a simulation framework for CPU-GPU computing’, International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 335–344. https://doi.org/10.1145/2370816.2370865. UC Berkeley Sky Computing Lab (n.d.) UCCL: An Efficient Collective Communication Library for GPUs. Available at: https://sky.cs. berkeley.edu/project/uccl/ (accessed 13 September 2026). Varga, A. and Hornig, R. (2008) ‘An overview of the OMNeT++ simulation environment’, First International Conference on Simulation Tools and Techniques (SIMUTools). https://doi.org/10.4108/ICST. SIMUTOOLS2008.3027. White, B., Lepreau, J., Stoller, L., et al. (2002) ‘An integrated experimental environment for distributed systems and networks’, USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 255–270. https://www.usenix.org/conference/osdi-02/integratedexperimental-environment-distributed-systems-andnetworks. Williams, S., Waterman, A. and Patterson, D. (2009) ‘Roofline: an insightful visual performance model for multicore architectures’, Communications of the ACM, 52(4), pp. 65–76. https://doi.org/10. 1145/1498765.1498785.
13