HyNoC: A Hybrid Circuit-Switch/Wormhole Network-on-Chip for Distributed VLIW Computing on FPGA Christophe Clienti
arXiv:2607.02729v1 [cs.AR] 2 Jul 2026
Independent Researcher [email protected] https://github.com/cclienti/verilog-ip/tree/master/hw/network/hynoc HyNoC takes a different approach: it combines the pathdedication property of circuit switching—through a sourcerouted request/grant handshake at each hop—with the flit-level, FIFO-backed flow control of wormhole switching. This hybrid model is particularly well suited to FPGA targets, where block RAMs used as FIFOs are scarce resources, and to distributed VLIW computing workloads, where communication patterns are largely known at compile time and can therefore be routed statically to avoid congestion. Originally inspired by the Hermes NoC [4], HyNoC departs significantly from its ancestor by introducing distributed arbitration, relative source routing with multicast support, perport asynchronous clock domains, and a parallel round-robin arbiter that grants requests in a one or two cycles. The contributions of this paper are: • A precise characterization of HyNoC’s hybrid switching model and its implications for latency and area; • A discussion of static route management as an alternative to virtual channels, supported by a combinatorial analysis of shortest-path counts in mesh and higher-dimensional topologies; • An analysis of the scope and limitations of the no-VC design choice; • An open-source RTL implementation targeting both simulation and FPGA synthesis, complemented by the Veriparse toolkit [5] for source-to-source elaboration of advanced RTL constructs and faster simulation; I. I NTRODUCTION • Verilator co-simulation results measuring the deterministic, As the number of processing elements integrated on a single linear per-hop latency and demonstrating link-disjoint chip increases, scalable on-chip communication becomes a firstquadrant isolation under a LLaMA 3 8B-scale distributed class design concern. Network-on-Chip (NoC) architectures GeMV workload. have emerged as the standard answer to this challenge, replacing The remainder of the paper is organized as follows. Section II shared buses and point-to-point links with a structured, packet- reviews related work. Section III describes the HyNoC archiswitched interconnect fabric [1]. tecture. Section IV details the hybrid switching model and the Existing NoC designs span a broad design space. Wormhole rationale for avoiding virtual channels. Section V discusses the switching minimizes buffering by forwarding flits as soon as target application domain. Section VI presents implementation the header is decoded, but exposes the network to head-of-line details. Section VII reports simulation results, and Section VIII blocking. Virtual channels (VCs) were introduced to mitigate concludes. this blocking [2], at the cost of duplicated buffers that can II. R ELATED W ORK represent the dominant area contribution of a router [3]. Circuit switching provides fully dedicated, contention-free paths but NoC architectures span a broad design space, comprehenrequires path reservation before any data can flow. sively covered in [6]; this section reviews the dimensions most
Abstract—Network-on-Chip (NoC) architectures have become the standard interconnect fabric for many-core systems, yet most proposals face a fundamental trade-off between latency, area, and congestion management. This paper presents HyNoC (Hybrid Network-on-Chip), an open-source NoC architecture that combines circuit-switch path establishment with wormhole data transfer, targeting distributed computing systems built around VLIW processor cores on FPGA. HyNoC employs source routing, where the complete path through the network is encoded in the packet header by the sender or statically at compile time, enabling both deterministic low-latency transfers and software-level hotspot avoidance without the area overhead of virtual channels. The router features a parallel round-robin arbiter (PRRA) with fixed grant latency, per-port independent clock domains, and support for both unicast and multicast routing. We discuss the design rationale, the hybrid switching model, and positioning relative to prior NoC art, and argue that a richer topology combined with compiler-assisted static routing is a competitive alternative to virtual channels for FPGA-based distributed VLIW systems. Verilator co-simulation measures the deterministic per-hop latency and benchmarks the design at LLaMA 3 8B FFN up-projection scale: a four-master quadrant configuration partitions the mesh into traffic-isolated quadrants with zero crossquadrant link traffic, yielding a 5× throughput improvement over a single master. The complete RTL implementation is available as open-source hardware under the CERN-OHL-P v2 license. Index Terms—Network-on-Chip, FPGA, source routing, wormhole switching, circuit switching, VLIW, parallel round-robin arbiter, open-source hardware.
directly relevant to HyNoC’s design choices. A. Switching Strategies
entered, a router with N ports offers only N −1 egress choices, encoded in ⌈log2 (N −1)⌉ bits per hop. This reduces both header overhead and crossbar complexity simultaneously.
Three main switching strategies are found in the NoC C. Virtual Channels literature. Store-and-forward switching requires a complete Virtual channels (VCs), introduced by Dally [2], multiplex packet to be buffered at each router before forwarding, several logical channels over a single physical link, providing providing clean error isolation but imposing large buffer additional routing paths and breaking deadlock cycles. Their requirements and high latency [1]. Wormhole switching [7] main drawback is area: each VC requires a dedicated buffer, pipelines flits across routers, dramatically reducing buffer and [3] shows that router area scales nearly linearly with VC depth requirements; latency depends on path length and count. On FPGAs, this overhead is particularly significant contention. Circuit switching [8] reserves a dedicated path because both the application logic and the communication before transmission, eliminating mid-network contention at the fabric draw from a shared pool of Block RAM (BRAM) and cost of a setup overhead proportional to route length before distributed LUT RAM resources: adding VCs directly reduces the first data flit is delivered. the memory budget available to the application. Hybrid approaches have been explored. Æthereal [9] comSeveral works have explored VC reduction or elimination. bines guaranteed-throughput circuit-switched slots with bestTurn-model routing [14] achieves deadlock freedom without effort wormhole traffic on the same physical links using timeVCs by restricting the set of turns allowed in the routing graph. division multiplexing. HyNoC takes a different hybrid point: it HyNoC’s approach—providing path diversity through richer establishes a dedicated path through a hop-by-hop request/grant topologies and compile-time route selection—aligns with this handshake (the circuit phase), then streams payload flits in trend, and is analyzed quantitatively in Section IV. wormhole fashion along that reserved path (the data phase). Unlike Æthereal, HyNoC does not multiplex traffic in time: D. NoC Arbitration once a path is open, it is held exclusively for the duration of Arbiter design is a central concern in input-queued routers, the packet, and no other packet can pre-empt it on the same where multiple ingress ports compete for the same egress [6]. egress port. The setup latency — one request/grant round per Matrix arbiters evaluate all (ingress, egress) request pairs in hop — is paid once per packet and is bounded by the route parallel and can grant in a single cycle regardless of port length, which is acceptable for the long-burst transfers typical count, but require O(N 2 ) state and combinational logic, making of VLIW inter-core communication. them area-expensive for large routers. Separable allocators decompose the two-dimensional matching problem into two B. Routing Algorithms independent one-dimensional arbitration steps (one per ingress, XY deterministic routing [1] is simple and deadlock-free on one per egress), reducing area at the cost of potentially requiring 2D meshes but provides only a single path between any pair multiple rounds to reach a stable matching [6]. Sequential of nodes. Adaptive routing [10] improves load balancing but round-robin FSMs scan requests cyclically, introducing up to requires more complex router logic and typically needs virtual N − 1 cycles of latency before granting a pending request. channels to avoid deadlock. HyNoC implements a parallel round-robin arbiter (PRRA): Source routing encodes the complete route in the packet a set of N − 1 combinatorial LUT-based state functions that header at the sender. This approach, studied for NoCs in [11] evaluate all pending requests simultaneously and jump directly and [12], shifts routing complexity from the router to the to the winning state, regardless of how many ports are idle. The sending node or to an offline compilation step. The router itself PRRA matches the single-cycle grant latency of a matrix arbiter becomes simpler: it only needs to decode the next hop from for the common case, while its critical path does not grow the current flit, forward it, and update the index. Source routing with N since each LUT is evaluated independently. Fairness naturally supports any topology without per-router routing table is maintained through a rotating priority identical to classical storage, and allows the sender to choose among multiple paths. round-robin. This compile-time route assignment property is particularly attractive for VLIW many-core systems, where communication E. FPGA-Targeted NoCs patterns are statically known and can be scheduled to avoid FPGA implementations impose constraints not present in contention (see Section V). ASIC designs: Block RAMs (BRAMs) are a finite, shared Source routing has also been adopted in production VLIW resource, so buffer-heavy NoC designs directly reduce the many-core NoCs: the Kalray MPPA2-256 uses a wormhole memory budget available to the application; clock domain NoC with feed-forward source routing, where the complete crossing between the NoC fabric and the attached processing route is encoded in the header flit and each router forwards elements must be handled explicitly; and routing resources based solely on a local offset extraction [13]. This production constrain achievable clock frequencies [15]. deployment validates the compile-time route assignment model CONNECT [15] provides a parameterized, synthesizable that HyNoC targets. NoC generator for FPGAs supporting configurable topologies, HyNoC adopts source routing with a compact relative hop VC counts, and buffer depths, demonstrating that generalencoding: since a packet cannot exit from the same port it purpose FPGA NoCs are feasible. Its VC-inclusive design
Wormhole TDM + Wormhole Wormhole Deflection Circuit+Wormhole
XY (dest.) Table Configurable XY (dest.) Source (rel.)
No Yes Yes No No
ASIC/FPGA ASIC FPGA FPGA FPGA
Ingress egress
P egress 3 Ingress
N od e P egress 3 Ingress P 1
N od e
Ingress egress
P 1
Ingress P egress 0 Ingress egress
Ingress P egress 0
P 4
N od e
Ingress egress
N od e
Router
P 4
L ss I re s s eg re g In 2
Hermes [4] Æthereal [9] CONNECT [15] HopliteRT [17] HyNoC
Router
P
Target
L ss I re s s eg re g In 2
VCs
P egress 3 Ingress
P
Routing
Ingress P egress 0
Ingress egress
Switching
Ingress egress
Design
P 4
P 1
L ss I re s s eg re g In 2
TABLE I P OSITIONING OF REPRESENTATIVE N O C S
P 1
P
Table I summarizes how HyNoC relates to the principal design axes discussed above.
P egress 3 Ingress
L ss I re s s eg re g In 2
G. Positioning
Ingress P egress 0
P
Distributed VLIW many-core systems—exemplified by the Kalray MPPA [20]—replicate simple, compiler-scheduled cores [21] connected through a dedicated NoC. Later generations [13] adopt source routing to preserve deterministic latency guarantees across the fabric, the same design point HyNoC occupies. Because the communication graph of such systems is known at compile time, the bounded circuit-establishment latency of HyNoC’s hybrid model (one PRRA grant per hop) feeds directly into the compiler’s timing model. We develop this target application in detail in Section V.
Router
P 4
Ingress egress
F. VLIW Many-Core Systems
Router
Ingress egress
trades BRAM for throughput robustness, which is appropriate when traffic patterns are unpredictable at design time. Hoplite [16] introduced deflection routing on FPGA overlays, achieving very small area by eliminating input buffers entirely and routing contending flits onto an alternative output rather than buffering them. HopliteRT [17] extended Hoplite with a real-time traffic class, but deflection fundamentally introduces non-deterministic, potentially unbounded delivery latency for best-effort traffic — a property incompatible with the timing constraints of VLIW inter-core communication. Xilinx’s Versal ACAP [18] embeds a hardened, fully programmable NoC in silicon alongside the FPGA fabric, demonstrating that on-chip network infrastructure has become a first-class concern even in commercial FPGA products. Its hardened implementation achieves high bandwidth with low latency, but is vendor-specific and not accessible as a soft IP. Hermes [4], the direct ancestor of HyNoC, demonstrated lowarea wormhole switching on FPGA with a 2D mesh topology and XY routing. Its asynchronous variant Hermes-A [19] explored per-port independent clock domains through asynchronous handshaking. HyNoC addresses the same requirement using synchronous dual-clock FIFOs, which bridge arbitrary clock domain boundaries at each ingress port without requiring asynchronous logic. HyNoC targets a distinct point in this space: topologyagnostic source routing with bounded latency, no virtual channels, and LUT-RAM-based FIFOs that avoid BRAM consumption, suited to predictable VLIW workloads.
Fig. 1. Router interconnections overview.
III. H Y N O C A RCHITECTURE A. Overview A HyNoC network is built by assembling routers with a configurable number of ports N (3 to 9 in the current implementation), where each flit is K + 1 bits wide (K = PAYLOAD_WIDTH, the extra bit being the stop bit). Each router port is full-duplex, composed of an ingress interface and an egress interface. Two neighboring routers are connected by cross-linking their egress and ingress interfaces. Processing nodes are attached through a local interface, which adds an extra output FIFO to decouple the node’s clock domain from the router. The router implements a full crossbar: any ingress port can simultaneously communicate with any egress port, subject to arbitration. Multiple paths can be open concurrently inside the same router, allowing concurrent transfers between different ingress-egress pairs. Figure 1 shows the internal organization of 5-port routers and their interconnections. B. Ingress Port The ingress port receives flits from the egress of a neighboring router. It buffers incoming flits in a dual-clock FIFO (2D entries of K + 1 bits), which simultaneously handles clock domain crossing between the upstream router’s clock domain and the local router domain. Once flits are buffered, the FSM inspects the two most significant bits of the first flit to identify the flit type (routing or payload). It then extracts the next-hop egress port identifier, asserts the matching request bit among the N − 1 available egress ports, and waits for the corresponding grant. Upon grant, the ingress forwards the routing flit updated to the next hop
Router fifo_wlevel fifo_wen fifo_wdata
Ingress 0
Egress 0
fifo_wlevel fifo_wen fifo_wdata
fifo_wlevel fifo_wen fifo_wdata
Ingress 1
Egress 1
fifo_wlevel fifo_wen fifo_wdata
fifo_wlevel fifo_wen fifo_wdata
Ingress 2
Egress 2
fifo_wlevel fifo_wen fifo_wdata
Ingress 0
fifo_wlevel fifo_wen fifo_wdata
Ingress 1
fifo_wlevel fifo_wen fifo_wdata
Ingress 2
Fig. 2. Internal data path of a 3-port router. D 1
fifo_wen fifo_wdata
fifo_rlevel D 1 K +2
Request Decoder
fifo_rempty fifo_ren fifo_rdata K +2
egress_afull
request_set
N−1
to_egress_write from_egress_grant
to_ingress_afull to_ingress_grant
Almost Full Detection
grants requests
Egress 0
fifo_wlevel fifo_wen fifo_wdata
Egress 1
fifo_wlevel fifo_wen fifo_wdata
Egress 2
fifo_wlevel fifo_wen fifo_wdata
grants requests
grants requests
Fig. 4. Internal control path of a 3-port router. 1
Controller write_new_header
fifo_wlevel
1
request_reset
Clock domain of the upstream router
Router
fifo_wlevel fifo_wen fifo_wdata
N−1
N−1
The comparator output is expended to N-bit
Level ≤ 5
D
fifo_wlevel
1
N−1 N−1
N−1
N−1
Index Countdown K +2
from_egress_afull
from_ingress_req
to_egress_request
from_ingress_write to_egress_data
Fig. 3. Ingress port internal architecture: dual-clock FIFO and routing FSM.
(index decremented); if the index has already reached zero, the routing flit is consumed locally and not forwarded. Subsequent payload flits are streamed directly to the granted egress until the stop bit of the last flit is detected, at which point the channel is released. Flow control is managed by monitoring both the downstream router’s ingress FIFO level and the local ingress FIFO level. The ingress stalls when either FIFO approaches full, preventing flit loss. The internal data path of a 3-port router is illustrated in Figure 2. C. Egress Port The egress port is connected to N − 1 ingress ports via a (K + 1)-bit wide data path. It arbitrates among concurrent write requests using the PRRA, which responds in one cycle when the port is idle, and grants one ingress at a time without starvation. Once a grant is issued, the data and write-enable signals from the winning ingress are routed directly to the downstream router’s ingress FIFO. The downstream FIFO level is forwarded back to the granted ingress for flow control. The control path is shown in Figure 4: contrary to the data path, control signals are point-to-point between each ingress and each egress, avoiding any broadcast of control information. D. Parallel Round-Robin Arbiter (PRRA) The standard sequential round-robin FSM scans requests one by one in a fixed order, introducing up to N − 1 cycles of latency before granting a pending request (Figure 6). HyNoC replaces this with a parallel round-robin arbiter (PRRA): a set of N − 1 combinatorial LUT-based state functions that evaluate all pending requests simultaneously and jump directly to the winning state, regardless of how many ports are idle.
from_ingress_data
N−1
PRRA
grant state
⌈ log2 N⌉
N−1
(N−1 )×(K+ 1 )
1
1
K+1 K +1
fifo_wen fifo_wdata
Fig. 5. Egress port internal architecture: PRRA arbiter and data multiplexer.
The architecture instantiates N − 1 prra_lut modules in parallel (Figure 7), each pre-computing the next granted port for a different starting priority offset. The LUT corresponding to the current state is selected by a multiplexer; its output feeds the state register. The state transitions only when the currently granted request is de-asserted (end of packet) or when no request is active, locking the grant for the entire packet transfer and preventing spurious re-arbitration. The arbiter responds in one or two clock cycles depending on the PIPELINE parameter: without pipelining the LUTs are purely combinatorial and the grant is issued in one cycle; with pipelining an extra register stage on the LUT outputs reduces the critical path at the cost of one additional cycle. The critical path does not grow with N because each LUT is independent and selected, not chained. Starvation is prevented by the rotating priority: after serving port k, priority rotates to port k + 1 mod N . E. Clock Domain Architecture Each router port operates in its own clock domain. The dual-clock FIFO at each ingress interface handles the crossing between the upstream port clock and the local router clock. For timing-critical designs, a single-clock mode (SINGLE_CLOCK_ROUTER=1) is available, reducing crossing latency. F. Local Interface Any router port can be connected to a processing node instead of a neighboring router. The hynoc_local_interface module handles this attachment by instantiating an additional FIFO at the egress output. This FIFO serves two purposes: it fully decouples the node’s clock domain from the router clock domain, and it smooths traffic bursts, preventing the
32 31
P3
0 P0
P1
0
0
P2
0
P2
0
P1
P0
P3
1
1 P0
2
1
P3
2
P2
3
3
(*) Chip Enable when request[state] negedge or when request is zero (one cycle delay)
==0
Decoder
Requests
LUT 0
2
0
CE*
1 0
Grants 4
0
CE*
State
1 2
LUT 1
2
3
2 Requests
LUT 2
2
LUT 3
2
Optional Registers
···
1
Last payload flit
3 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
2 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1
1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1
0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
LUT id 0 0 0 1 1 2 2 1 1 3 3 1 1 2 2 1 1
1 1 0 1 0 2 2 2 2 3 3 3 3 2 2 2 2
2 2 0 1 0 2 0 1 0 3 3 3 3 3 3 3 3
32 31
28 Proto
0
Routing flit (hops + index) Last payload flit
1
Fig. 6. Parallel Round-Robin FSM: direct state transitions avoid scanning idle ports.
4
Payload flit
. . .
0
P3
3
0
Fig. 8. Packet structure: routing flit(s) followed by payload flits. The MSB is the stop bit (0=intermediate, 1=last flit).
2
P0
3
0
Routing flit (hops + index)
P1
P1
2
1
P2
28 Proto
3 3 0 1 0 2 0 1 0 3 0 1 0 2 0 1 0
Fig. 7. PRRA architecture: N − 1 parallel LUTs, one per priority offset.
node from becoming a bottleneck when it cannot consume data fast enough. From the router’s perspective, the local interface is indistinguishable from a downstream router’s ingress port, preserving the uniformity of the flow control mechanism across all port types. AXI-Stream compatibility. The port-level handshake is deliberately modeled after the AXI-Stream protocol [22]: the write signal corresponds to TVALID, the full signal to the negation of TREADY, the data bus to TDATA, and the stop bit to TLAST. This mapping makes the interface immediately familiar to hardware designers working in the FPGA ecosystem. The
Fig. 9. Minimal HyNoC packet: one routing flit (stop bit = 0) followed by one payload flit (stop bit = 1). The routing flit stop bit must be zero since the ingress FSM rejects routing flits with stop bit set.
only extension beyond standard AXI-Stream is the fifo_level sideband, which exposes the downstream buffer occupancy to allow the sender to anticipate back-pressure before the full condition is reached. G. First Layer Protocol 1) Packet and Flit Structure: The network operates on flits of K + 1 bits (K = PAYLOAD_WIDTH). The most significant bit is a stop bit: when set, it marks the last flit of a packet. A packet is built with at least one Network Hops flit (routing flit) followed by at least one payload flit, as shown in Figure 8. If multiple routing flits are present, intermediate routers treat all but the first as payload; each router consumes one routing flit from the front of the chain. The MSB of a routing flit is always zero, reserving space for future sub-type extensions. Figure 9 shows the smallest valid packet: one routing flit (stop bit must be zero) followed by one payload flit. The routing flit carries a 4-bit Proto field (Table II): TABLE II S UPPORTED ROUTING PROTOCOLS Proto
Routing method
4’b0000 4’b0001 4’b1000 4’b1111
Unicast circuit switch Multicast circuit switch XY routing (slot reserved, not yet implemented) Forbidden
The proto value 4’b1000 is reserved for future XY routing support. XY routing would allow destination-addressed packets using (x,y) coordinates rather than explicit hop lists, enabling simpler packet construction at the cost of restricting the topology to 2D meshes. The ingress FSM currently flushes any packet carrying this proto value. 2) Unicast Source Routing: The Network Hops flit structure is shown in Figure 10. It carries H hop fields of ⌈log2 (N − 1)⌉ bits each, encoding the relative egress port ID at the next router, together with an index field initialized to H − 1. The index
32 31 0
28 27 26
5
Hops 10 – 1
Proto H11
4
3
0
H0 Index
Fig. 10. Network Hops flit structure for a 5-port, 32-bit router (unicast, 12 hops of 2 bits, 4-bit index). The layout from MSB to LSB is: stop bit, Proto, Gap (if any), Hop H − 1 . . . Hop 0, Index. The Gap absorbs unused bits between the Proto field and the first hop, and is zero in this particular configuration. The Index field (initialized to H − 1) selects the active hop. Router 1 2
3 0
3 2
3
0 (0,0)
2
1
0
1
(0,1)
TABLE III M AXIMUM HOP COUNT PER ROUTING MODE (K = 32, INDEX_WIDTH=4)
0
2
(1,1)
(1,2)
0
3 1
(2,1)
Router
Unicast bits/hop Hmax
Multicast bits/hop Hmax
3-port 5-port 9-port
1 2 3
2 4 8
3 1
(2,0)
Table III gives the resulting values for the three valid router sizes in the default configuration (K = 32, INDEX_WIDTH = 4), for both unicast and multicast routing (which uses N −1 bits per hop instead of ⌈log2 (N − 1)⌉). The wider multicast hop field halves the maximum hop count relative to unicast, reflecting the inherent trade-off between fanout expressiveness and path length.
(0,2)
Node
(1,0)
is K − 3 − INDEX_WIDTH, and the maximum hop count is: K − 3 − INDEX_WIDTH Hmax = (1) ⌈log2 (N − 1)⌉
2 (2,2)
Fig. 11. Relative hop encoding for a 3 × 3 mesh network built with 5-port routers. The hop list [0 → 2 → 0 → 1 → 2] establishes a path from node (0, 0) to node (2, 2). Each hop ID is relative to the ingress port: hop 0 denotes the next counterclockwise egress, excluding the ingress port itself.
24 12 8
12 6 3
3) Multicast Source Routing: In multicast mode, each hop field is a bitmask of width N − 1, where each bit selects one egress port. The packet is duplicated and forwarded to all asserted egress ports simultaneously. Multiple routing flits with different multicast masks can be chained to address complex multicast trees. This is particularly useful for broadcast operations in VLIW many-core workloads. IV. H YBRID S WITCHING M ODEL AND D ESIGN R ATIONALE
A. The Hybrid Switching Model directly feeds the multiplexer selecting the active hop in the HyNoC’s switching model is best described as circuitingress port. It is decremented after each hop is consumed; if the index is already zero before path opening, the routing flit switched wormhole: it combines a circuit-establishment phase is not forwarded downstream. The relative hop encoding on a with a wormhole data phase. In the circuit-establishment phase, the ingress port asserts a 3 × 3 mesh of 5-port routers is illustrated in Figure 11. request to a specific egress port (identified from the routing flit) Since a packet cannot exit through the same port it entered, and waits for a grant. This request/grant handshake propagates each ingress port has only N − 1 reachable egress ports. along the route: once the first router grants the request, the Hop fields therefore require only ⌈log2 (N − 1)⌉ bits, which ingress begins forwarding flits, which in turn trigger the same yields particularly compact encodings for the most common router sizes: a 3-port router encodes hops on 1 bit, a 5-port handshake at the next router. A dedicated forwarding path router on 2 bits, and a 9-port router on 3 bits. This reduction through the network is thus progressively opened, hop by hop. In the wormhole data phase, payload flits are streamed simultaneously shrinks the routing flit header overhead and the internal crossbar size, since the crossbar itself does not need flit-by-flit along the established path, regulated by FIFO-level to include the loopback path from ingress to its paired egress. flow control. The path remains dedicated to this packet until the More precisely, the encoding constrains N to satisfy stop bit is observed, ensuring in-order, contention-free delivery ⌈log2 (N −1)⌉ + 1 = N , meaning only N ∈ {3, 5, 9} are valid once the channel is open. 2 port counts. This constraint is enforced at elaboration time in This combination provides the low buffering advantage of the RTL. The relative hop ID is computed counterclockwise wormhole switching (only the ingress FIFO is needed, no full from the ingress port: hop 0 always refers to the next packet storage) while offering the path dedication advantage of counterclockwise egress, and the ingress FSM resolves the circuit switching (no mid-network arbitration contention during relative ID into a one-hot physical egress request. payload transfer). The trade-off is a path-establishment latency Given a flit of K + 1 bits, a 4-bit Proto field, and an proportional to the route length before the first payload flit is INDEX_WIDTH-bit index field, the number of available hop bits delivered at the destination.
B. End-to-End Latency (R − 1) + (C − 1) [(R − 1) + (C − 1)]! Let H denote the number of hops from source to destination (3) p= = (R − 1)! (C − 1)! R−1 (including the final local hop), and P the number of payload flits in the packet. End-to-end latency decomposes into two This generalizes to a K-dimensional network n1 × n2 × additive terms. · · · × nK as: Setup phase. The routing flit propagates hop by hop. At each hP i K hop it traverses a fixed pipeline of register stages—the ingress i=1 (ni − 1) ! FIFO read, the routing-FSM request register, the PRRA grant, (4) p = QK and the egress data register—and, in dual-clock mode, the i=1 (ni − 1)! gray-code synchroniser of the ingress FIFO [23]. This pipeline Note that (3) and (4) give the number of shortest paths is identical for every hop and contains no contention-dependent between the most distant node pair (corner-to-corner). For an path once a request is granted, so the per-hop cost is a fixed arbitrary pair separated by (d , d ) hops the count is dx +dy , x y dx constant α. which is lower. Table IV reports corner-to-corner values for Transfer phase. Once the path is established, the P payload representative 2D mesh topologies. flits are pipelined at one flit per hop: the first flit reaches the destination after H cycles and the last after H + P − 1 cycles. TABLE IV End-to-end latency is therefore strictly linear and determinS HORTEST PATH COUNTS IN 2D MESH N O C S ( CORNER - TO - CORNER ) istic: Topology Routers Shortest paths Path length L = αH +P −1 (2) with no congestion-dependent term once the path is open. The per-hop constant α depends only on the FIFO clocking mode— it is larger in dual-clock mode by the depth of the gray-code synchroniser—and is measured directly by co-simulation in Section VII. C. Static Route Management as an Alternative to Virtual Channels Virtual channels were introduced primarily to: (1) provide additional logical paths to avoid deadlock; and (2) reduce headof-line blocking under congestion. HyNoC addresses both goals differently. Deadlock avoidance. Deadlock in wormhole networks arises from cyclic dependencies in the channel dependency graph (CDG) [7]: a set of flits each waiting for a channel held by the next forms an unbreakable cycle. With source routing, the complete path is fixed at injection time by the sender. Deadlock freedom therefore reduces to a property of the route set: if every injected route follows a routing function whose CDG is acyclic — dimension-order routing being the canonical example — no cycle can form, regardless of traffic load. This analysis is performed once, statically, by the compiler or network configurator; the router itself carries no deadlockprevention logic. The guarantee is strictly stronger than VCbased approaches, where deadlock freedom depends on runtime routing decisions and requires the router to maintain channel state across multiple virtual queues. Congestion avoidance is addressed at two levels. First, the compiler or runtime system assigns routes across multiple available shortest paths, distributing traffic away from known hotspots. The number of shortest paths between two nodes in an R × C mesh is a classical combinatorial result [24]: each path consists of a fixed sequence of moves along each dimension, taken in any order, and the count is given by the multinomial coefficient:
2×2 4×4 8×8 16 × 16
4 16 64 256
2 20 3 432 155 117 520
2 6 14 30
Second, for applications with more dynamic traffic, the network topology can be enriched with additional relay routers (without local interfaces), increasing path diversity proportionally to the added area. D. Area Comparison with Virtual Channels Following [3], router area scales linearly with the number of VCs (each VC requires an additional input buffer per port). An 8 × 8 mesh with 4 VCs consumes the equivalent area of a 16 × 16 mesh without VCs, yet the latter offers over 45,000× more shortest paths between corner nodes (Table IV: 155,117,520 vs. 3,432). On FPGA, where block RAMs are scarce, minimizing buffer count is a primary concern. The VC area model is itself optimistic in two respects: (1) the crossbar logic and VC scheduler also grow with VC count; (2) VC path diversity is an upper bound, since in practice the VC is assigned at the source and remains fixed for the entire path, providing less diversity than a true additional physical dimension. Conversely, the physical router area model does not capture link wiring cost, which may be relevant in ASIC contexts. E. Head-of-Line Blocking and Mitigations Once a channel is granted, it remains dedicated until the stop bit of the last flit is received. A long packet therefore delays other packets competing for the same egress port. Two complementary mitigations are available in HyNoC: 1) Network enrichment: adding more routers distributes traffic across more egress ports, reducing the probability that multiple flows compete for the same port. The cost is increased end-to-end latency due to additional hops—this is the primary performance trade-off, not throughput.
2) Multiple local interfaces: a node can expose several local interfaces connected to different router ports, providing as many independent logical channels as needed. This replicates the path diversity benefit of VCs without duplicating internal router buffers. It is the recommended approach when a node requires simultaneous, nonblocking communication with multiple peers. F. Scope and Limitations The topology-based approach is most effective when traffic patterns are known at compile time and routes can be assigned statically. When traffic is entirely unpredictable at design time and topology enrichment is not feasible, virtual channels remain the more appropriate solution. HyNoC targets systems— distributed VLIW processors on FPGA—where compile-time route assignment is natural and FPGA resource budgets make buffer minimization a primary concern. V. TARGET A PPLICATION : D ISTRIBUTED VLIW C OMPUTING A. VLIW Cores and Compile-Time Scheduling VLIW processors expose multiple functional units to the compiler, which schedules operations explicitly in long instruction words [21]. This design philosophy shifts control complexity from hardware to software, resulting in simple, area-efficient cores well suited for replication in many-core arrays. Signal processing, scientific computing, and neural network inference kernels are representative workloads. In a distributed VLIW many-core system, each core executes a portion of a data-flow graph, passing intermediate results to neighboring cores through the NoC. The communication graph is typically known at compile time, enabling the compiler to: 1) Assign each communication a source route through the NoC; 2) Schedule packet injections to avoid simultaneous contention on the same egress port; 3) Guarantee bounded communication latency for critical paths. This compile-time visibility makes HyNoC’s hybrid model particularly effective: the circuit-establishment latency is paid once per packet and is deterministic (bounded by route length and PRRA grant latency), and the wormhole payload transfer proceeds without further arbitration overhead.
in the testbench environment, verifying the correctness of the NoC under various load conditions. C. Multicast for Broadcast Operations VLIW many-core applications frequently require broadcast operations: distributing a coefficient matrix, a lookup table, or a control token to multiple cores simultaneously. HyNoC’s multicast routing protocol supports this natively: a single packet with a multicast routing flit is replicated at each router toward all targeted egress ports, without requiring the sender to issue multiple unicast packets. This reduces network load and sender overhead for one-to-many communication patterns common in data-parallel VLIW workloads. VI. I MPLEMENTATION A. Repository Organization HyNoC is part of a larger open-source IP library hosted at https://github.com/cclienti/verilog-ip. The repository is organized into two complementary layers under the hw/ directory: • hw/lib/: a collection of reusable, standalone Verilog IP cores covering arithmetic, memories, FIFOs, and arbiters. Each module ships with its own parameterized RTL and self-checking testbench. • hw/network/hynoc/: the HyNoC subsystem, which builds on top of hw/lib/ and is described in detail in this paper. HyNoC directly instantiates three library modules: dclkfifolut (dual-clock FIFO mapped to distributed LUT RAM, used in hynoc_ingress and hynoc_local_interface), sclkfifolut (single-clock variant, used in single-clock mode), and prra together with its prra_lut sub-module (the parallel round-robin arbiter, used in hynoc_egress). These components are independently verified and reusable outside HyNoC. Relying on shared library primitives rather than vendor-specific FIFO or arbitration IP is what makes the design portable across FPGA families. B. RTL Organization
The HyNoC RTL is written in Verilog and organized as follows: • hynoc_ingress: ingress port (dual-clock FIFO via dclkfifolut/sclkfifolut, routing FSM, unicast and multicast protocol decoder); • hynoc_egress: egress port (data multiplexer, PRRA via B. Integration with the Local Interface prra/prra_lut); • hynoc_router_base: parameterized router base (crossbar Each VLIW core connects to the NoC through a local wiring, port instantiation); interface, which presents a simple FIFO-based write/read API. • hynoc_router_3p / hynoc_router_5p: 3-port and 5-port The core writes routing flits followed by payload flits into the router instances with self-checking testbenches; local interface’s ingress FIFO; the local interface manages all • hynoc_local_interface: local interface for node attachhandshaking with the router. On the receive side, incoming ment (output FIFO via dclkfifolut/sclkfifolut); flits are deposited in the egress FIFO and read by the core • hynoc_stream_reader / hynoc_stream_writer: simulaat its own pace. The dual-FIFO structure fully decouples the tion traffic generators used in testbenches to inject and core’s clock domain from the router domain. check random packet streams. The hynoc_stream_writer and hynoc_stream_reader modules are simulation utilities that generate and check random The design is parameterized by: payload width packet streams, respectively. They serve as traffic generators (PAYLOAD_WIDTH), FIFO depth (LOG2_FIFO_DEPTH: 2
to 64 entries), number of ports (NB_PORTS: 3 to 9), index width (INDEX_WIDTH), optional multicast routing (ENABLE_MCAST_ROUTING), and single-clock mode (SINGLE_CLOCK_ROUTER).
The design has been validated on both Intel (Altera) and AMD (Xilinx) FPGA families; reference board support is included for the DE0-Nano (Cyclone IV) and the ZedBoard (Zynq-7000). Both dclkfifolut (dual-clock) and sclkfifolut (single-clock) map exclusively to distributed LUT RAM, avoiding dependency on vendor-specific FIFO C. Simulation and Verification IP and ensuring portability across FPGA families. Each module is delivered with a self-checking testbench. The AXI-Stream-compatible port interface (see Section III) The 3-port router testbench exercises both unicast and multicast simplifies integration with standard FPGA IP cores. On Zynq scenarios over a 2-router, 4-node topology. The 5-port router devices for instance, the HyNoC local interface connects testbenches cover five distinct traffic scenarios, including directly to the Processing System AXI-Stream DMA engine simultaneous multi-path transfers that verify the PRRA’s with only a thin adapter to bridge the fifo_level sideband. starvation-free behavior. This reduces integration effort and allows reuse of existing Simulation is supported under Icarus Verilog and ModelSim. AXI-Stream verification infrastructure. Verilator is integrated in the build infrastructure both for linting (–lint-only) and for C++ co-simulation: a multi-instance E. RTL Elaboration with Veriparse Verilator model is used in Section VII to characterize end-toend latency and link utilization on a 4 × 4 mesh. Waveform HyNoC makes use of advanced Verilog constructs — in inspection is automated using Wavedisp [25], a Python module particular, initial blocks containing for loops and integer that generates simulator-specific signal configuration scripts arithmetic — to pre-compute the PRRA lookup tables at from a single source description, targeting GTKWave, Model- elaboration time. While this is perfectly legal Verilog, some Sim, and Riviera-PRO. FPGA toolchains have limited support for such constructs in synthesizable RTL. Veriparse [5] is a companion source-to-source transformation D. FPGA Target toolkit that addresses this issue. Its veriflat tool performs a seTable V reports post-place-and-route results on two Xilinx quence of elaboration-time passes on the full module hierarchy: 7-series devices: a Kintex-7 (xc7k160t-fbg484, speed grade −2) parameter and localparam inlining substitutes all parameter targeting 370 MHz and a Zynq-7020 (xc7z020clg484, speed values at instantiation sites; constant folding and expression grade −1) targeting 200 MHz. Both runs use 32-bit payload evaluation reduce symbolic expressions to integer literals; loop and 32-entry single-clock FIFOs (sclkfifolut). WNS (Worst unrolling expands for, while, and repeat constructs; generateNegative Slack) is the tightest setup slack across all registerblock elaboration resolves generate/endgenerate sections; to-register timing paths; a positive value means the target is variable folding evaluates initial blocks whose control flow met, and fmax = 1 / (Tclk − WNS). is entirely determined by inlined constants; and dead-code elimination prunes branches that can never be taken after TABLE V constant propagation. The module flattener then inlines the full P&R RESULTS ON TWO X ILINX 7- SERIES DEVICES (32- BIT, 32- ENTRY instantiation hierarchy into a single, parameter-free module. The sclkfifolut; 24 LUT-RAM/ PORT FOR INGRESS FIFO S ). WNS IN NS ; fMAX IN MH Z . resulting RTL contains only plain assignments, registers, and combinational logic, with no remaining parametric expressions Kintex-7 −2 Zynq-7020 −1 — portable across any standard Verilog toolchain. (370 MHz) (200 MHz) Figure 12 illustrates parameter inlining and constant folding Router LUT LUT FFs WNS fmax WNS fmax on the module boundary: the parameterized hynoc_router_5p (log.) (RAM) (ns) (MHz) (ns) (MHz) declaration, whose port widths are symbolic expressions 3-port 253 72 327 +0.079 381 +0.034 201 of PAYLOAD_WIDTH and LOG2_FIFO_DEPTH, becomes a flat 5-port 596 120 590 +0.127 389 +0.101 204 declaration with all widths resolved to integer literals. The 57-port 1259 168 889 +0.070 380 +0.032 201 9-port 1806 216 1215 +0.003 371 +0.034 201 port, 32-bit configuration yields [32:0] data buses and [5:0] FIFO level signals in the output. Figure 13 shows a complementary transformation class LUT-RAM usage is exactly 24 per port in every configuration, reflecting the fixed-depth FIFO at each ingress; no block RAM supported by veriflat: variable folding combined with loop is consumed. LUT counts are device-independent to within unrolling. The initial block of prra_lut — which computes 1% (synthesis tool variation). Logic LUTs grow superlinearly the round-robin priority table through nested for loops and with port count, driven by the PRRA whose table size scales modular arithmetic over inlined constants — is fully evaluated as 2N −1 per egress port. FFs grow nearly linearly (109 to at elaboration time; the resulting initial block contains only 135 per port), consistent with the per-port register cost of the explicit constant assignments. ingress FSM and FIFO control. All four port counts meet their The flattened output also benefits simulation: a single-module respective frequency targets on both devices, with the 5-port design with no hierarchy reduces simulator overhead and configuration consistently achieving the best timing margin. can significantly speed up large testbench runs. Veriparse
Before (parameterized): module hynoc_router_5p #(parameter PAYLOAD_WIDTH = 32, parameter LOG2_FIFO_DEPTH = 5, ...) (... input wire [PAYLOAD_WIDTH:0] port0_ingress_data, output wire [LOG2_FIFO_DEPTH:0] port0_ingress_fifo_level, ...);
After veriflat (parameters inlined): module hynoc_router_5p (... input wire [32:0] port0_ingress_data, output wire [5:0] port0_ingress_fifo_level, ...); Fig. 12. veriflat resolves parameterized port widths into constants.
Before (algorithmic initial block): initial begin lut[0] = STATE_OFFSET[LOG2_WIDTH-1:0]; for (j=1; j<lut_length; j=j+1) begin lut_index = j[WIDTH-1:0]; value = -1; for (k=0; k<WIDTH; k=k+1) begin l = (k + STATE_OFFSET + 1) % WIDTH; if (lut_index[l] == 1'b1) if (value == -1) value = l; end lut[j] = value[LOG2_WIDTH-1:0]; end end
After veriflat (pre-computed, STATE_OFFSET=0): initial begin lut[0]=2'd0; lut[1]=2'd0; lut[2]=2'd1; lut[3]=2'd1; lut[4]=2'd2; lut[5]=2'd2; lut[6]=2'd1; lut[7]=2'd1; lut[8]=2'd3; lut[9]=2'd3; lut[10]=2'd1; lut[11]=2'd1; lut[12]=2'd2; lut[13]=2'd2; lut[14]=2'd1; lut[15]=2'd1; end Fig. 13. veriflat evaluates the prra_lut initial block: nested loops and modular arithmetic are replaced by 16 explicit constant assignments (4-port arbiter, STATE_OFFSET=0).
0,0 M
0,1
0,2
0,3
1,0
1,1
1,2
1,3
2,0
2,1
2,2
2,3
3,0
3,1
3,2
3,3
Fig. 14. 4 × 4 mesh topology. M = master at (0, 0); remaining nodes are workers at (r, c). Bidirectional links are full-duplex (independent ingress and egress). The gray arrow illustrates the XY route from the master to worker (2, 3): three East hops followed by two South hops, then a final local hop (H = 6).
1) Latency characterization. A lightweight GeMV (General Matrix-Vector multiplication, y = Ax) with a 16 × 4 matrix produces clean per-hop measurements that establish the deterministic per-hop latency discussed in Section IV. 2) LLaMA 3 8B FFN benchmark (single master). A full-scale GeMV at LLaMA 3 8B FFN up-projection dimensions (14336×4096) with Q8_0 weights and BF16 activations demonstrates the network under realistic AI inference workload. 3) Four-master throughput scaling. The same LLaMA 3 8B-scale benchmark is replicated across four corner masters, each owning a 2 × 2 quadrant, to quantify the speedup from distributed injection. All three benchmarks share the same 16-router 4 × 4 mesh and cover the full communication cycle: data scatter from the master(s), computation at worker nodes, and result gather. A. Experimental Setup
is available at https://github.com/cclienti/veriparse under the LGPLv3 license. F. License HyNoC has been in development since 2013, as reflected in the copyright headers of every RTL source file. The source code was first published in 2019; the repository was formally placed under the CERN-OHL-P v2 license [26] in 2026. This permissive open hardware license allows use, modification, and redistribution with no copyleft obligation on the end product. The complete source, including RTL, testbenches, documentation, and build infrastructure, is available at: https: //github.com/cclienti/verilog-ip. VII. E VALUATION We present three Verilator [27]-based simulations on a 4 × 4 HyNoC mesh, each targeting a different evaluation goal:
Network Topology. The network consists of 16 hynoc_router_5p instances arranged as a 4×4 2D mesh, as shown in Figure 14. Routers are addressed by (r, c) with r ∈ {0, 1, 2, 3} (row, top-to-bottom) and c ∈ {0, 1, 2, 3} (column, left-to-right). Port convention per router: 0 = Local, 1 = East, 2 = South, 3 = West, 4 = North. Each router’s local port (port 0) attaches a processing node; neighboring routers are cross-linked on their directional ports (e.g., router (r, c) port 1 connects to router (r, c+1) port 3). The master node resides at (0, 0). It sequentially distributes data to all 15 worker nodes and collects their results. B. Experiment 1: Latency Characterization 1) Computation and Communication Model: The benchmark computes y = Ax with A ∈ R16×4 and x = [1, 2, 3, 4]T . Worker (r, c) is responsible for one scalar inner product: y4r+c = ⟨A4r+c , x⟩.
Forward packet (master to worker (r, c)): one routing flit, one tag flit, four flits for the matrix row A4r+c , and four flits for x, totaling ten flits. After the routing flit is consumed by the network, the worker receives Pfwd = 9 payload flits. Return packet (worker to master): one routing flit, one tag flit, and one result flit. The master receives Pret = 2 payload flits. Workers compute the inner product instantaneously, so the total simulation time is dominated by communication latency and the sequential injection schedule at the master. Route construction. Hop lists are built following dimensionorder (XY) order: East/West moves bring the packet to the target column, North/South moves bring it to the target row, and a final local hop delivers it. This eliminates cyclic channel dependencies and guarantees deadlock freedom [7]. All packets use standard unicast source routing (proto 4’b0000); the dedicated XY proto field (4’b1000) reserved in Table II is not exercised in these experiments. A route from (r0 , c0 ) to (r1 , c1 ) requires H = |c1 − c0 | + |r1 − r0 | + 1
(5)
hops, where the +1 accounts for the final local hop at the destination. Throughout this section H always counts all hops including that local hop. For the master at (0, 0), this simplifies to H = r + c + 1. Routes are encoded as source-routing flits using the relative hop encoding described in Section III. Simulation Infrastructure. The Verilator C++ model instantiates 16 Vhynoc_router_5p objects sharing one VerilatedContext. Inter-router links are modeled as one-cycle latches: egress signals are captured after each rising clock edge and applied to the neighbor’s ingress before the next rising edge. This one-cycle propagation delay is physically accurate for a registered synchronous interconnect. Each router drives a single clock (router_clk and all portX_ingress_clk toggled together). In dual-clock mode (SINGLE_CLOCK_ROUTER=0), the ingress dual-clock FIFOs internally use gray-code CDC synchronization even when source and destination clocks are the same signal; in single-clock mode (SINGLE_CLOCK_ROUTER=1), the gray-code pipeline is bypassed. 2) Latency Results: Table VI reports the return-path latency for each hop count H in both configurations. Return-path measurements are free of source-queuing effects (each worker sends exactly one result as soon as computation completes), yielding clean linear fits. The measured return-path latencies satisfy exact linear fits: Ldual = 12H + P − 1
(6)
Lsingle = 9H + P − 1
(7)
matching the linear, deterministic form of (2) with a fixed perhop cost of α = 12 cycles in dual-clock mode and α = 9 in single-clock mode, and no load-dependent term. This per-hop cost is the pipeline traversed at each router—ingress FIFO read, routing-FSM request register, PRRA grant, and egress data register—together with the one-cycle registered inter-router link
TABLE VI M EASURED RETURN - PATH LATENCY (P = 2, CYCLES ) VS . HOP COUNT. D UAL = DUAL - CLOCK FIFO (SINGLE_CLOCK_ROUTER=0); S INGLE = SINGLE - CLOCK FIFO (SINGLE_CLOCK_ROUTER=1); ∆ = D UAL−S INGLE . H
Dual
Single
∆
2 3 4 5 6 7
25 37 49 61 73 85
19 28 37 46 55 64
6 9 12 15 18 21
of the co-simulation harness. The 3-cycle-per-hop difference between the two modes (α = 12 versus 9) is exactly the depth of the dual-clock FIFO’s gray-code synchroniser [23], which the single-clock FIFO does not instantiate. This difference is constant across all hop counts, so the latency stays perfectly linear and predictable in both modes. The total simulation time is 391 cycles in dual-clock mode and 349 cycles in single-clock mode (−10.7%). All 16 computed results are correct in both runs. 3) Link Utilization: Table VII summarizes link utilization over the 391-cycle dual-clock run. Traffic is inherently asymmetric: the master sends all forward packets via its East port first, concentrating load on the East-bound links of row 0. TABLE VII L INK UTILIZATION (%) BY DIRECTED LINK ( DUAL - CLOCK , 391 CYCLES ). E ACH ROW NAMES ONE ROUTER - TO - ROUTER LINK . L INKS WITH <0.5% UTILIZATION OMITTED ; OVERALL AVERAGE : 3.3%. Direction
Link
→East, row 0
(0, 0) → (0, 1) (0, 1) → (0, 2) (0, 2) → (0, 3)
Util (%) 30.7 20.5 10.2
(0, 0) → (1, 0) →South, col 0 (1, 0) → (2, 0) (2, 0) → (3, 0)
7.7 5.1 2.6
→North, col 0
(1, 0) → (0, 0)
9.2
→West, row 0
(0, 1) → (0, 0) (0, 2) → (0, 1) (0, 3) → (0, 2)
2.3 1.5 0.8
East-bound utilization halves at each column of row 0, consistent with XY routing: each router delivers one packet to its local worker and forwards the rest eastward. The overall average of 3.3% reflects the sequential master bottleneck: with a single sender issuing one packet at a time, most links are idle. C. Experiment 2: LLaMA 3 8B FFN Benchmark 1) Single-Master Configuration: To assess HyNoC under a realistic AI inference workload we scale the benchmark to the LLaMA 3 8B [28] FFN up-projection layer. In transformer models, the Feed-Forward Network (FFN) consists of two linear projections with a nonlinearity in between; the up-projection is the first of these, expanding the representation from the model dimension Din = 4096 to the intermediate dimension
Dout = 14336. The benchmark computes y = W x with W ∈ TABLE VIII L INK UTILIZATION (%) BY DIRECTED LINK , LL A MA 3 8B SINGLE - MASTER R14336×4096 and x ∈ R4096 . RUN (43 102 246 CYCLES ). A LL FOUR S OUTH COLUMNS ARE SYMMETRIC . Weights are stored in the Q8_0 quantization format used by R ETURN - PATH LINKS (<0.1%) OMITTED ; OVERALL AVERAGE : 6.7%. llama.cpp [29]: each weight is an 8-bit signed integer, and one FP16 (16-bit floating-point) scale factor is shared per block of Direction Link Util (%) 32 weights to recover approximate FP32 values at decode time. (0, 0) → (0, 1) 79.9 Mapping to flits: the FP16 scale occupies the lower 16 bits of →East, row 0 (0, 1) → (0, 2) 53.2 (0, 2) → (0, 3) 26.6 one 32-bit flit (upper 16 bits zero), and the 32 int8 weights (0, k) → (1, k), k = 0–3 20.0 are packed four per flit, requiring 8 flits — hence 9 flits per →South, all cols (1, k) → (2, k), k = 0–3 13.3 Q8_0 block. Activations use the BF16 format (Brain Float 16: (2, k) → (3, k), k = 0–3 6.7 1 sign bit, 8 exponent bits, 7 mantissa bits), packed two values per 32-bit flit. Results are accumulated and returned in FP32 (IEEE 754 single precision). workers; South per column: 3, 2, 1 workers), with no hot-spots Forward packet (master to worker, for one output row): beyond the first East segment. Nonetheless, the bottleneck one routing flit, one tag flit, 128 Q8_0 blocks × (1 scale remains the single injection port, motivating the multi-master flit + 8 weight flits), and 2048 BF16 activation flits, totaling extension. Pfwd = 3201 payload flits. 2) Four-Master Quadrant Configuration: The single master Return packet: one routing flit, one tag flit, and one FP32 at (0, 0) creates an injection bottleneck: all 15 forward packets result flit — Pret = 2 payload flits, identical to the latency per round enter through one port, and all result packets return experiment. to that same port. We now distribute injection across the four Computation model. Each of the 15 worker nodes is corner nodes by assigning each a 2 × 2 quadrant. assigned 14336/16 = 896 output rows; the master at (0, 0) also Table IX lists the assignment. Master m handles Q = computes 896 rows locally. Workers decode Q8_0 weights to D /4 = 3584 output rows. With NW = 3 workers per out FP32 and accumulate the dot product in FP32. Local computaquadrant and one row computed locally, each master executes tion is modeled as instantaneous: injecting one forward packet ROUNDS = Q/(NW + 1) = 896 rounds—where one round takes 3202 clock cycles (one per flit), which already exceeds the consists of sending one forward packet to each of the NW dot-product budget of a real VLIW unit over 4096 elements, so workers and computing one row locally, then collecting NW arithmetic is not the bottleneck. All 14 336 results are verified result packets—the same count as the single-master case, but for exact FP32 equality against a software reference. all four masters proceed simultaneously. Throughput. The simulation completes in 43 102 246 cycles (dual-clock FIFO, SINGLE_CLOCK_ROUTER=0) with the master’s TABLE IX single egress port sustaining near-100% injection throughout. Q UADRANT ASSIGNMENT: MASTERS , WORKERS , AND ROW RANGES . Link utilization (Table VIII) shows the consequence of this sustained injection: link (0, 0) → (0, 1) reaches 79.9%, carrying Master Position Workers Rows packets bound for all 12 workers that need at least one East M0 (0, 0) (0, 1), (1, 0), (1, 1) 0–3583 hop. Utilization falls in proportion to the number of remaining M1 (0, 3) (0, 2), (1, 3), (1, 2) 3584–7167 M2 (3, 0) (3, 1), (2, 0), (2, 1) 7168–10751 destinations on each successive link. South-bound links absorb 1 M3 (3, 3) (3, 2), (2, 3), (2, 2) 10752–14335 of the total traffic each. Return-path links (West and North) 4 are negligible at <0.1%. The overall average of 6.7% reflects The structural property exploited here is link disjointness: the single-port injection bottleneck: 47 of 48 directed links are underutilised while one port saturates. Repeating the every XY route from a corner master to its three workers stays run in single-clock mode (SINGLE_CLOCK_ROUTER=1) yields within the 2 × 2 quadrant. No packet crosses the horizontal 43 102 204 cycles, only 42 cycles below the dual-clock result. midline (between rows 1 and 2) or the vertical midline (between This confirms that the workload is injection-bound rather columns 1 and 2), so the four quadrants are traffic-isolated by than latency-bound: runtime is dominated by the one-flit-per- construction, without any runtime coordination. Figure 15 illustrates the topology and the three routes from cycle injection of the 3201-flit forward packets, so the perhop latency saved by single-clock operation—the 9H versus M0 : one East hop to (0, 1), one South hop to (1, 0), and 12H slope of (7) and (6)—is negligible against the tens of East+South to (1, 1). millions of injection cycles. The four-master run is likewise Computation model. Each master computes one row locally unchanged (8 620 466 cycles single-clock versus 8 620 484 dual- per round (row mQ+t(NW +1) in round t, for t = 0, . . . , 895), clock), so the 5× speedup is mode-independent. Single-clock for a total of 896 local rows per master and 3584 across all four. operation benefits only latency-dominated traffic, such as the Local dot products are modeled as instantaneous: a VLIW unit Experiment 1 latency test, where it reduced the total from 391 over Din = 4096 elements completes well within the 3 × 3202 to 349 cycles. clock-cycle injection window per round, so arithmetic is not The load on each segment is strictly proportional to the on the critical path. All 14 336 results are verified for exact number of destinations reachable through it (East: 12, 8, 4 FP32 equality against a software reference.
TABLE XI R EPRESENTATIVE LINK UTILIZATION (%), FOUR - MASTER RUN (8 620 484 CYCLES ). B Y SYMMETRY ALL FOUR QUADRANTS ARE IDENTICAL ; CROSS - BOUNDARY LINKS ARE EXACTLY 0%.
0,0 M0
0,1
0,2
0,3 M1
1,0
1,1
1,2
1,3
Direction
Link
→East, row 0 →West, row 0 →East, row 3 →West, row 3 →South, col 0 →South, col 1 →North, col 0 →North, col 1 Cross-boundary
(0, 0) → (0, 1) (0, 3) → (0, 2) (3, 0) → (3, 1) (3, 3) → (3, 2) (0, 0) → (1, 0) (0, 1) → (1, 1) (3, 0) → (2, 0) (3, 1) → (2, 1) (any)
2,0
2,1
2,2
2,3
3,0 M2
3,1
3,2
3,3 M3
Overall average
Fig. 15. Four-master 4 × 4 mesh. Corner nodes M0 –M3 (shaded) each coordinate a 2 × 2 quadrant; dashed lines mark the mesh midpoints. Arrows show M0 ’s three routes to workers (0, 1), (1, 0), and (1, 1). By symmetry, routes from all four masters remain within their respective quadrant.
Util (%) 66.6 66.6 66.6 66.6 33.3 33.3 33.3 33.3 0.0 11.1
VIII. C ONCLUSION
HyNoC is a hybrid circuit-switch/wormhole Network-onChip designed for distributed VLIW computing on FPGA. By combining source routing—which establishes a dedicated path 3) Throughput Scaling Results: The four-master simulation through a distributed request/grant handshake—with wormhole completes in 8 620 484 cycles, a 5.0× speedup over the flit streaming, HyNoC achieves bounded, deterministic transfer 43 102 246 cycles of the single-master run (Table X). latency without the area overhead of virtual channels. Congestion is managed statically by the compiler or system TABLE X software, which selects routes across the rich set of shortest S INGLE - VS . FOUR - MASTER THROUGHPUT (DOUT = 14336, PFWD = 3201 paths available in mesh and higher-dimensional topologies. PAYLOAD FLITS ). The combinatorial analysis presented in Section IV shows that enriching the physical topology is more area-efficient Configuration Masters Cycles Speedup than adding virtual channels for the targeted workload class, Single master 1 43 102 246 1× provided that a compile-time scheduler distributes traffic across Four masters 4 8 620 484 5× available paths. The parallel round-robin arbiter provides starvation-free, The 5× gain exceeds the 4× expected from simple injection parallelism. The additional margin comes from backpressure fixed-latency arbitration at each egress port, and per-port relief: each corner master serves only three workers, whereas independent clock domains allow the NoC to interface cleanly the single master drives fifteen. With fewer concurrent result with heterogeneous processing elements running at different packets converging on one egress port, injection stalls are frequencies. The multiple local interface mechanism offers a practical alternative to virtual channels when a node requires shorter. Table XI reports the utilization pattern. The quadrant isola- simultaneous non-blocking communication with multiple peers. Verilator co-simulation on a 4 × 4 mesh measures a tion property is confirmed quantitatively: every link crossing either mesh midpoint registers exactly 0%. This zero-overhead deterministic per-hop latency: return-path latency satisfies isolation is a direct consequence of source routing: routes are L = 12H + P − 1 (dual-clock) and L = 9H + P − 1 (singlecomputed statically and encoded entirely in the packet header, clock), the 3-cycle-per-hop difference being exactly the depth so no router-internal mechanism is needed to enforce quadrant of the dual-clock FIFO’s gray-code synchroniser, which singleboundaries at runtime. Within each quadrant, the first East or clock operation omits. A LLaMA 3 8B FFN up-projection West hop from the corner master reaches 66.6% because two benchmark (Q8_0 weights, BF16 activations, 14336 × 4096) of the three forward routes share it (the direct worker and the with a single master completes in 43 M cycles; a four-master diagonal worker); each South or North second-hop segment quadrant configuration, where each corner master owns a 2 × 2 carries one route at 33.3%. By symmetry all four quadrants traffic-isolated quadrant, reduces this to 8.6 M cycles—a 5× show identical figures. The overall average of 11.1% contrasts speedup with zero cross-quadrant link traffic, confirming the with the 3.3% of the single-master run, reflecting simultaneous deterministic throughput scaling predicted by the link-disjoint injection from four ports and correspondingly more uniform topology design. HyNoC is released as open-source hardware under the CERNmesh loading. All results in this section are obtained from Verilator OHL-P v2 license, providing a complete RTL implementation co-simulation; on-device measurement on physical FPGA with simulation testbenches and FPGA board support. hardware remains open and is identified as future work in Future work includes: formal deadlock-freedom analysis Section VIII. for arbitrary source route sets; synthesis results and latency
characterization on representative FPGA devices (Section VII provides Verilator-based characterization; on-device measurement remains open); integration with a VLIW compiler backend for automatic route assignment and injection scheduling; and extension of the XY routing protocol placeholder to a full hardware implementation. ACKNOWLEDGMENT The author used Claude Sonnet 4.6 (Anthropic) as an AI writing assistant during the preparation of this manuscript, including for drafting and revising text, literature analysis, and LATEX editing. All technical content, claims, and results have been verified by the author. R EFERENCES [1] W. J. Dally and B. Towles, “Route packets, not wires: On-chip interconnection networks,” in Proc. Design Automation Conference (DAC), 2001, pp. 684–689. [2] W. J. Dally, “Virtual-channel flow control,” IEEE Transactions on Parallel and Distributed Systems, vol. 3, no. 2, pp. 194–205, 1992. [3] A. Mello, L. Tedesco, N. Calazans, and F. Moraes, “Virtual channels in networks on chip: Implementation and evaluation on Hermes NoC,” in Proc. 18th Annual Symposium on Integrated Circuits and System Design (SBCCI), 2005, pp. 178–183. [4] F. Moraes, N. Calazans, A. Mello, L. Möller, and L. Ost, “HERMES: an infrastructure for low area overhead packet-switching networks on chip,” Integration, the VLSI Journal, vol. 38, no. 1, pp. 69–93, 2004. [5] C. Clienti, “Veriparse: A source-to-source transformation toolkit for verilog/systemverilog,” https://github.com/cclienti/veriparse, 2024, lGPLv3 license. [6] W. J. Dally and B. Towles, Principles and Practices of Interconnection Networks. Morgan Kaufmann, 2004. [7] W. J. Dally and C. L. Seitz, “Deadlock-free message routing in multiprocessor interconnection networks,” IEEE Transactions on Computers, vol. C-36, no. 5, pp. 547–553, 1987. [8] S. Kumar, A. Jantsch, J.-P. Soininen, M. Forsell, M. Millberg, J. Öberg, K. Tiensyrjä, and A. Hemani, “A network on chip architecture and design methodology,” in Proc. IEEE Computer Society Annual Symposium on VLSI (ISVLSI). IEEE, 2002, pp. 117–124. [9] K. Goossens, J. Dielissen, and A. Radulescu, “Æthereal network on chip: Concepts, architectures, and implementations,” IEEE Design & Test of Computers, vol. 22, no. 5, pp. 414–421, 2005. [10] J. Duato, “A new theory of deadlock-free adaptive routing in wormhole networks,” IEEE Transactions on Parallel and Distributed Systems, vol. 4, no. 12, pp. 1320–1331, 1993. [11] M. Liwei and S. Yihe, “On-chip network design automation with source routing switches,” Tsinghua Science and Technology, vol. 12, no. 1, pp. 77–85, 2007. [12] S. Mubeen and S. Kumar, “Designing efficient source routing for mesh topology network on chip platforms,” in Proc. 13th Euromicro Conf. on Digital System Design (DSD), 2010, pp. 181–188. [13] B. D. de Dinechin and A. Graillat, “Feed-forward routing for the wormhole switching Network-on-Chip of the Kalray MPPA2 processor,” in Proc. 10th International Workshop on Network on Chip Architectures (NoCArc), 2017. [14] C. J. Glass and L. M. Ni, “The turn model for adaptive routing,” in Proc. International Symposium on Computer Architecture (ISCA), 1992, pp. 278–287. [15] M. K. Papamichael and J. C. Hoe, “CONNECT: Re-examining conventional wisdom for designing NoCs in the context of FPGAs,” in Proc. ACM/SIGDA International Symposium on FPGAs (FPGA), 2012, pp. 37–46. [16] N. Kapre and J. Gray, “Hoplite: Building austere overlay NoCs for FPGAs,” in Proc. 25th International Conference on Field Programmable Logic and Applications (FPL), 2015, pp. 1–8. [17] S. Wasly, R. Pellizzoni, and N. Kapre, “HopliteRT: An efficient FPGA NoC for real-time applications,” in Proc. International Conference on Field-Programmable Technology (ICFPT), 2017, pp. 64–71.
[18] I. Swarbrick, D. Gaitonde, S. Ahmad, B. Gaide, and Y. Arbel, “Networkon-chip programmable platform in Versal ACAP architecture,” in Proc. ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA), 2019, pp. 212–221. [19] J. Pontes, M. Moreira, F. Moraes, and N. Calazans, “Hermes-A: An asynchronous NoC router with distributed routing,” in Proc. PATMOS, 2011, pp. 150–159. [20] B. D. de Dinechin, R. Ayrignac, P.-E. Beaucamps, P. Couvert, B. Ganne, P. G. de Massas, F. Jacquet, S. Jones, N. M. Chaisemartin, F. Riss, and T. Strudel, “A clustered manycore processor architecture for embedded and accelerated applications,” in Proc. IEEE High Performance Extreme Computing (HPEC), 2013, pp. 1–6. [21] J. R. Ellis, “Bulldog: A compiler for VLIW architectures,” Ph.D. dissertation, Yale University, 1986. [22] ARM Limited, AMBA AXI and ACE Protocol Specification, ARM Limited, 2011, iHI0022D, version D. [23] C. E. Cummings, “Simulation and synthesis techniques for asynchronous FIFO design,” in Synopsys Users Group Conference (SNUG), 2002. [24] D. E. Knuth, The Art of Computer Programming, Volume 1: Fundamental Algorithms, 3rd ed. Addison-Wesley, 1997, section 1.2.5: Permutations and Factorials. [25] C. Clienti, “Wavedisp: A python-based waveform display configuration tool,” https://github.com/cclienti/wavedisp, 2019, gPLv3 license. [26] “CERN open hardware licence version 2 – permissive,” https://ohwr.org/ cern_ohl_p_v2.pdf, 2020. [27] W. Snyder et al., “Verilator: The fastest Verilog/SystemVerilog simulator,” https://www.veripool.org/verilator/, 2024, open-source under LGPLv3 or Artistic-2.0. [28] A. Grattafiori et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407. 21783 [29] G. Gerganov et al., “llama.cpp: LLM inference in plain C/C++,” https: //github.com/ggml-org/llama.cpp, 2023, open-source under MIT license.