NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems

NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems Conor James Green1,2 , William Won1 , Tuan Ta1 , and Bradford M. Beckmann1 1 AMD Research and Advanced Development

2 Purdue University

arXiv:2608.00867v1 [cs.DC] 1 Aug 2026

Abstract Graphics processing unit (GPU) architectures are growing in size to meet the increasing compute and memory requirements. As GPU sizes increase, intra-socket wire transfer delay increases significantly. While previous research has optimized for compute and memory locality within a socket, the spatial impact on inter-GPU communication has not been well-studied. We introduce the term non-uniform network access (NUNA) to describe this emerging optimization dimension in multi-GPU systems. We specifically focus on latency-sensitive collective communication, common in machine learning inference. First, we highlight the need for NUNAaware routing (NAR), choosing optimized, spatially-aware interGPU paths in large scale-up network topologies. Second, we introduce NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic. We demonstrate that the NAP optimizations alone offer up to 1.5× collective speedups over a locality-unaware baseline. Combining NAP with NAR yields up to 1.8× faster collectives over the locality-unaware baseline. This leads to 7% mean (28% max) time per output token speedup in machine learning inference.

1

Figure 1: Possible high and low latency inter-GPU communication paths. A threadblock (blue) is copying from a local source (orange) to remote destination (blue) buffers. Scenario a represents physically far accesses between CU, memory, and I/O ports, whereas b shows low-latency accesses. To further satisfy the massive AI compute and memory demands, hardware vendors have recently introduced novel scale-up networks where multiple GPUs can communicate across a single shared address space using load and store instructions [34]. This enables AI models to be distributed across multiple GPUs through parallelization techniques [17, 62, 72, 103]. However, the parallelization requires collective communications among GPUs to communicate activations and gradients, which can consume more than 40% of total execution time [41, 64, 80]. Hardware vendors continue to improve direct GPU-to-GPU interconnects, such as AMD Infinity Fabric™ [83] or NVIDIA NVLink/NVSwitch technologies [53], to address this scale-up network bottleneck. Specifically, scale-up network bandwidth has increased across GPU generations by increasing the number of I/O ports per socket and data rates per port. Network traffic is hashed across the socket’s I/O ports for load balancing [6, 7, 63, 66–68, 86]. Overall, while the proliferation of large multi-die GPU sockets connected together using massive scale-up networks is an unprecedented, exciting time in computer architecture, their non-uniform communication latencies must be addressed to reach the full capability of these impressive systems. In the research community, NUMA and non-uniform cache access (NUCA) effects within a single socket have been analyzed for decades [16, 21, 38, 48, 57] and have recently been extended to distributed systems [13, 32, 53, 83]. These prior NUMA and NUCA solutions have mainly focused on reducing the initial and subsequent data access penalties within a socket or a small handful of sockets. NUMA or NUCA solutions have not looked at collective communication and the multiple communication paths provided by large scale-up networks. Specifcially,

Introduction

Artificial intelligence (AI) models have been growing in size and require significantly more compute and memory than what a single graphics processing unit (GPU) can provide. Prominent large language models (LLMs) such as GPT-4 [72] and Llama-4 [62] have billions to trillions of parameters, requiring significant on-chip memory capacity and compute capability. To meet this thrust for AI compute and memory throughput, designers have recently adopted aggressive packaging technology where individual GPUs are composed of multi-chip modules (MCMs) and 3D-integrated chiplet architectures [11, 76]. With the AI chip market expected to expand to hundreds of billions of annual sales by the 2030s [24, 60], hardware vendors can affordably leverage more extravagant power delivery and cooling solutions, integrating even more reticle-limit compute and memory dies into a single package [77, 89, 90]. The resulting multi-die GPUs provide significant improvements by doing more work per socket. However, the longer physical distances between compute units (CUs), memory, and input/output (I/O) ports significantly impact intra-socket latency [44, 57]. While recent work has highlighted optimizations for intra-socket non-uniform memory access (NUMA) locality [73, 92], the impact of intra-socket latency on inter-GPU communication has not been well-studied. Correspondence to: Conor Green <Conor.Green@amd.com, green456@purdue.edu>, William Won <William.Won@amd.com>. 1

Prefill (64 GPUs) 100% 75% 50% 25% 0% 128 KiB

512 KiB

2 MiB

8 MiB

32 MiB

Cumulative Proportion

Cumulative Proportion

to the best of our knowledge, prior NUMA works that focused on GPUs [13, 18, 19, 32, 47, 49, 63, 104, 108] have not considered the physical location of the I/O ports. This paper is the first to address the non-uniform network access (NUNA) latencies arising in large multi-die GPUs and their impact on scale-up communication. In the same way that NUMA optimizations spurred decades of research in high-performance computer architectures, we believe NUNA will be a critical element for optimizing future high performance multi-GPU systems. Figure 1 illustrates the impact spatial distances have on interGPU communication. For a threadblock to put data off-chip, the far paths (highlighted in red) traverse significantly more on-chip interconnect than the near paths (highlighted in green). Imagine that collectives are naïvely implemented without considering NUNA effects. The collective begins with ( 1a ) a GPU threadblock a reading a value from the local high bandwidth memory (HBM) stack located physically far away. Once the value is read, the value is then written remotely by ( 2a ) first routing it to a far I/O port, statically chosen based on address, and then ( 3a ) traversing the scale-up network and storing the value in the remote GPU’s HBM stack located far from the I/O port it was received on. In contrast, the NUNA-aware collective begins with ( 1b ) a threadblock b reading the local value from the HBM stack closest to it. Then, ( 2b ) the threadblock writes the data remotely by communicating across the I/O port closest to it and ( 3b ) storing the data to the HBM stack closest to the receiving I/O port. Section 3.2 evaluates NUNA’s impact on current generation GPUs and identifies the best and worst case communication latencies vary by almost 2×. Quantifying NUNA’s impact for next-generation GPUs shows that the worstand average-case network-on-chip (NoC) latencies can differ by 3× and 2×, respectively, compared to the best case. This results in uncontended GPU-to-GPU transfer slowdowns of 1.8× and 1.45× for the worst and average cases, respectively. To systematically exploit these non-uniform intra-socket latencies for inter-GPU communications, we introduce two policies that ensure collectives leverage the best possible communication paths. First, we introduce NUNA-aware routing (NAR) to prioritize latencysensitive traffic, utilizing physically closer I/O ports. We find that applying NAR alone can improve collective performance by up to 35%. Second, we introduce NUNA-aware placement (NAP) of threadblocks and memory pages. NAP complements NAR by mapping threadblocks and data to CUs and HBM stacks physically closer to the I/O ports. NAP and NAR applied together can reduce collective execution time by up to 80%. To summarize, we make the following contributions:

Decode (64 GPUs) 100% 75% 50% 25% 0% 2 KiB

Cumulative Proportion

Output Buffer Size

8 KiB

32 KiB

128 KiB

Output Buffer Size

DeepSeek V3.2

GPT−OSS−20B

Llama−70B

Mixtral−8x22B

DeepSeek V4 Flash

GPT−OSS−120B

Llama−405B

Qwen3−30B−A3B

DeepSeek V4 Pro

Llama−7B

Mixtral−8x7B

Qwen3−235B−A22B

Training 100% 75% 50% 25% 0% 1 KiB

4 KiB

16 KiB

64 KiB 256 KiB

1 MiB

4 MiB

16 MiB 64 MiB 256 MiB 1 GiB

4 GiB

16 GiB

Output Buffer Size ResNet50 (2 GPUs)

GPT−175B (64 GPUs)

Llama 7B (128 GPUs)

Mixtral 8x7B (64 GPUs)

DLRM (8 GPUs)

Llama 2 (8 GPUs)

Llama 70B (256 GPUs)

Mixtral 8x22B (8 GPUs)

GPT−5B (8 GPUs)

Llama 7B (16 GPUs)

Mixtral 8x7B (8 GPUs)

Mixtral 8x13B (128 GPUs)

Figure 2: Cumulative distribution of collective sizes (in output buffer size) for inference and training workloads.

2 Background 2.1 Collective Communications Collective communications have existed in high-performance computing (HPC) for decades [14, 61] and have recently been adopted by distributed AI training and inference. Both the AI model and the parallelization techniques used to distribute the model determine the required collectives. For example, mixture-of-experts (MoE) models use All-to-All to route tokens to experts and collect results [41]. Meanwhile, data and tensor parallelization require (i) All-Reduce for gradient aggregation and (ii) All-Gather to assemble split model weights [74]. Therefore, although there exist additional collective patterns, All-Gather, All-Reduce, and All-to-All are the most common patterns used by AI.

2.2

Latency-Sensitive Collectives

Analytical 𝛼–𝛽 modeling (using parameters in Section 3.1.2) shows that transmitting less than 100 MB of traffic over multi-terabyte scale-up links contributes only microseconds of serialization delay. This makes point-to-point latencies meaningfully impact the total communication time. Therefore, NUNA-aware collective optimizations show the most benefit on such latency-sensitive collectives, as they reduce the overall time for inter-GPU communication. Inference. Inference workloads account for the majority of the computational resources at data centers for companies such as Meta, Amazon, and Google [15, 81, 99]. Unlike training, inference workloads (i) often use small batches, and (ii) their decode operations operate token-by-token, resulting in small-size collectives. Furthermore, for quality of service guarantees, inference tasks need to optimize for latency such as time per output token (TPOT) [86]. Recent LLM-serving systems, vLLM [51] and SGLang [106], characterize decode as limited by per-iteration latency, rather than aggregate compute. These factors make collectives in inference more sensitive to network latency rather than network bandwidth. Consequently, optimizing latency-sensitive collectives is a critical performance challenge [9, 31, 88, 105]. For instance, Google highlighted network

• Define and analyze NUNA effects for scale-up systems of physically large GPUs. • Develop a NUNA-aware routing policy, NAR, that requires minimal hardware changes and significantly reduces intraGPU I/O port access latency for inter-GPU traffic optimization. • Propose a NUNA-aware placement policy, NAP, which places threadblocks and memory pages closer to the I/O ports to reduce inter-GPU communication latency. • Evaluate NAR and NAP to accelerate state-of-the-art collective communication. 2

latency optimization as one of the biggest hardware challenges that the AI industry faces for inference [59]. Figure 2 analyzes collective sizes in end-to-end prefill and decode from public models with different numbers of parameters and parallelization for 64 GPUs. The methodology is explained in Section 5.2 and the plots provide the cumulative distribution of output buffer sizes for each workload. Notably, the prefill phase introduces larger collectives, but their size is limited to tens of megabytes. The decode phase issues much smaller collectives, typically in the kilobytes range due to its token-by-token execution nature. NUNA significantly impacts these small, latency-bound collectives in LLM inference. Training. While inference is more latency-sensitive, training often includes small collectives as well. Prior works have found that communication dominates end-to-end LLM training time [30, 42, 79, 82]. While there exists a wide diversity of model and parallelism approaches, prior work consistently finds latency-sensitive collectives in training [30, 33, 42, 101]. To quantify the collective sizes used in training, we analyze public Chakra traces from MLCommons [39, 91] and the Scalable Parallel Computing Lab [3, 87]. These traces were collected from real systems spanning 2–256 GPUs and feature several different parallelization strategies. Figure 2 plots the collective size distribution. Across all traces, the majority of collectives are at or below 100 MB and several traces (e.g., ResNet, DLRM, and Mixtral variants) are mostly below 10 MB. In particular, the tensor-parallel traces (e.g., dense Llama- and GPT-class models) are dominated by larger (100 MB) Reduce-Scatter and All-Gather collectives, while the data-parallel traces (e.g., ResNet, DLRM) are dominated by smaller (10 MB) All-Reduce collectives.

Figure 3: Representative scale-up system evaluated in this work. Each GPU socket consists of four reticle-limit compute dies (yellow) with 16 HBM stacks (purple) along the perimeter. The inter-GPU network is a single-level Clos.

(a) AMD Instinct™ MI210

(b) AMD Instinct™ MI355X

(c) AMD Instinct™ MI300X

Figure 4: Real-system measured remote store latencies between two GPUs, across three clusters. The width indicates frequency and the dashed lines indicate quartiles.

3 Motivation 3.1 Scaling Trends 3.1.1 Large Multi-Die GPU Socket. To meet AI training and inference demands, vendors have increased compute and memory capacity per socket by adding higher throughput CUs and more HBM. Advanced packaging techniques enable integrating multiple dies per socket. [84, 89]. The physical size of a single GPU continues to increase, whether they scale with MCMs [13, 83], chiplet-based devices [89, 90], or wafer-scale systems [75]. Current devices such as NVIDIA Hopper/Blackwell [12, 21, 57, 93] and AMD Instinct™ MI300X/MI350X [10, 89, 90] GPUs are considerably larger and more power hungry than the prior generations. Recent public announcements from vendors show that the pace of GPU scaling is only expected to increase [11, 76]. Vendors first prioritize providing ample memory bandwidth. GPUs expand along the long dimension, to place compute dies adjacent to as many HBM stacks, resulting in an elongated cigarshaped socket with significant signaling beachfront. Consequently, inter-GPU I/O ports are located along the far edges. Figure 3 depicts a representative next-generation GPU design.

domain features relatively higher throughput and lower latency, compared to scale-out networks. Figure 3 illustrates a representative scale-up system (we omit the I/O ports to the CPU host or scale-out NICs for brevity) [2, 4, 11, 76]. A single-level Clos topology connects multiple sockets [22]. Each scale-up switch connects to the I/O ports with the same ordinal ID, creating an independent communication plane (i.e., rail-optimized). 3.1.3 Intra-Socket versus Scale-Up Latency. Intra-socket latencies are growing as the physical footprint scales. NoC latency significantly increases as on-chip wire pitches scale down [16]. However, scale-up link and switch latencies are expected to shrink (or stay constant) [45, 46, 58]. Low-attenuation inter-socket transmissions are less sensitive to physical distance [94]. Low-latency scale-up technologies, such as optical networks, further exemplify this. These trends made the intra-socket NoC and scale-up network latencies comparable. We analyzed the NoC and scale-up latencies for Figure 3, using recent commercial multi-die GPU [44, 56, 57] and scale-up network [45, 46, 58] values. For a remote write, the NoC transfer takes up to 0.9 𝜇s, while the scale-up latency is 1 𝜇s. Consequently, NoC transfer delay can greatly impact the total interGPU communication latency, when the communication is latencysensitive.

3.1.2 System Scaling Through Scale-Up Network. In addition to scaling a single GPU, modern AI systems leverage compute and memory across multiple devices. Specifically, scale-up networks create a shared memory with a single address space. The scale-up 3

3.2

Table 1: Summary of NUMA and NUNA techniques.

Non-Uniform Latency for Inter-GPU Communication

Mitigation Technique

The recent scaling trends emphasize the importance of understanding intra-socket latency for inter-GPU communications. However, there has been little prior evaluation on how the CU and memory location affects remote access latency. To demonstrate intra-socket latency effect on inter-GPU communication, we profiled real-system scale-up communication latencies. Specifically, we profiled three scale-up systems, each with different generations of GPUs: AMD Instinct™ MI210, MI355X, and MI300X. Each socket has one, two, and four interposer dies, respectively. We issued an atomic add operation from a CU to a remote GPU target address and measured the latency. To profile the spatial effect, we repeated this process over all CUs and target address range of a GPU, and measured the latency variations. The results are plotted in Figure 4. The profiling results demonstrate that the remote communication latency significantly varies by local compute and data locations. We observe multiple trends. First, the communication latency variation is proportional to the physical size of the socket (i.e., number of interposer dies). Compared to the single-die AMD Instinct™ MI210, the four-die Instinct™ MI300X has 3× and 5× higher variation amongst target addresses and CUs, respectively. As vendors physically scale next-generation GPUs, the non-uniformness becomes more prominent and should be addressed for latencysensitive communications. Second, some CUs have overall lower remote communication latency. Figure 4 shows that the latency gap between the fastest and slowest CUs, especially the tail latency, increases as the socket grows. Collectives are tail-latency-bound. Therefore, some CUs are better than others and should be prioritized for latency-bound collectives. Finally, the memory location also varies the remote access latency. A single CU still experiences significant latency variation dependent on the target address. Further reducing the tail latency requires considering physical data placement in HBMs. To summarize, the physical placement of compute and data significantly impacts the inter-GPU transfer latency. The spatial impact especially increases as the socket size scales. Therefore, we must mitigate this NUNA effect to optimize the latency-sensitive collective performance.

3.3

NUMA

NUNA

✔ ✘ ✘ ✘

✔ ✔ ✔ ✔

Local buffer locality to compute Fine-grain off-chip routing Remote buffer locality to I/O Threadblock scheduling for remote traffic

Fine-Grain Routing. Existing NUMA techniques ignore distinct I/O ports, combined with CU and memory locations, show nonuniform network latencies. Without considering such non-uniformity across I/O ports, locality optimizations cannot mitigate NUNA. For example, Khairy et al. address hierarchical applications through socket-level colocation [47], and Milic et al. discuss fine-grained I/O port assignment to dynamically adjust directional bandwidth. However, neither addresses non-uniform latency across I/O ports nor the routing scheme to mitigate such effect. Remote Buffer Locality. While previous NUMA works colocate compute to local buffers, none optimize remote data placement, especially relative to I/O [18, 47, 49, 50, 54, 108]. Previous techniques do not offer methods to resolve the conflict between increasing locality (conventionally achieved by placing data near accessing threadblocks) and keeping buffers remote (placed/pinned at their respective devices). Communication Threadblock Placement. Prior NUMA works often assume coarse-grained memory allocations to GPU sockets [49, 50] or only focus on intra-socket locality [19, 54] without distinguishing local and remote access patterns. They simply group threadblocks accessing the same data, and distribute them evenly across the entire GPU using simple round-robin scheduling [13, 47]. However, the communication threadblocks should be allocated to the CUs physically close to the I/O ports. This is especially true for latency-bound small collectives where each threadblock has very low resource usage but high latency sensitivity.

4

Optimizing for NUNA

Optimizing for NUNA requires multiple steps: • A compiler or profiling process decomposes the collective plan to identify each threadblock’s loads and stores. • NUNA-aware placement (NAP) groups threadblocks and the chunks they access into logical groups and round-robin assigns them to NUNA-aware routing (NAR) domains. • Using the logical assignments, NAP allocates threadblocks to CU(s) and memory chunks to HBM stacks. • The dispatcher and memory driver implement threadblock-toCU(s) and chunk-to-memory-stack mappings via CU masking and physical address placement, respectively. • The collective executes with NAP-optimized local and remote locality, utilizing NAR’s low-latency off-chip routing.

Limitations of NUMA Optimizations

While NUMA mitigations colocate compute with the memory it accesses, NUNA introduces a new dimension: I/O. Addressing NUNA requires colocating I/O in between compute and memory. Thus, we purposely introduced the term NUNA to highlight this parallel nature to NUMA. Concretely, prior NUMA techniques for distributed GPUs cannot adequately address NUNA effects. Table 1 compares them. NUMA techniques lack (i) hardware implementations to control off-chip routing, (ii) analysis of remote buffers, and (iii) algorithms to improve communication threadblock scheduling. These three deficiencies correspond to our three contributions: NAR, NAP memory placement, and NAP threadblock scheduling.

We first define and explain NAR as the foundation for NUNA optimization. Next, we describe the NAP algorithm, exploiting the express I/O port access capabilities from NAR. NAP analyzes a collective communication plan and decides (i) the best NAR granularity, (ii) threadblock placement, and (iii) chunk-to-page mappings. The algorithm decouples access patterns (logical allocation) from 4

(a)

(b)

Figure 5: Static partitioning some of the physical address space into segments of various NAR granularities, leaving the rest unmodified (baseline). Each NAR domain (color) hashes into a subset of I/O ports. physical placement to achieve high locality while reducing resource oversubscription. The complete NUNA-aware optimization scheme only require modest, backward-compatible changes across the stack. First, the NAP algorithm needs the backend communication library/compiler (e.g., MSCCL++ [85], NCCL [5], or RCCL [1]) to provide collective plans. We rely on existing collective plans and leave automatic collective detection via static code analysis [18, 19, 47, 49, 50, 54, 108] to future work. For placement, we utilize threadblock-to-CU affinity masks to prioritize a threadblock to be located over a specific CU. The driver controls memory page placement at HBM stack granularity. Both mechanisms build on prior work that coordinates fine-grained threadblock-to-CU and virtual-to-physical mappings [23, 47, 54, 96]. For the network, we propose modifying the address hashing logic to to limit the inter-GPU communication to a subset of I/O ports. Critically, all proposed modifications can run simultaneously with other workloads and be disabled for nonlatency-sensitive applications. Section 4.2.3 provides further implementation details.

4.1

(c) NUMA-aware placement with baseline routing

(d) NUNA-aware placement with NAR

Figure 6: Example of (a) baseline and (b) granularity-2 NAR address to I/O port mappings. (c) An example of baseline and NAR routing for local loads (solid arrows) and remote stores (dashed arrows) between threadblocks and data for NUMA-aware placement, and (d) NAP with NAR. route inter-GPU flits to a very specific subset of I/O ports physically close to it, reducing the latency. However, the number of I/O ports each CU can utilize is limited, hurting the network bandwidth for larger collectives. NAR leverages a statically partitioned physical address space, as illustrated in Figure 5. The physical address space is divided into distinct segments, each representing to a specific NAR granularity (e.g., granularity-2, granularity-4, etc.). Each segment utilizes all I/O ports, partitioned uniformly into subsets for each domain. To implement NAR, we propose modifying NoC flit generation logic. The upper address bits determine the segment. Each segment may utilize different NAR granularity or even baseline routing, providing routing flexibility. The number of I/O ports in each NAR domain is the total number of ports divided by granularity. For example, in Figure 5, the socket has four I/O ports; each NAR domain has two I/O ports when the granularity is two. Each NAR domain statically maps to a subset of ports, via offset. The yellow domain in Figure 5 has an offset of two to target I/O ports 2–3. Within a NAR domain, we load balance across I/O ports. This requires no changes to the NoC. Once the I/O port is chosen by NAR, the on-chip transfer continues as normal. Note this physical address partitioning scheme also facilitates page-to-HBM-stack placement implementation. The physical addresses for a NAR domain will not only be restricted to a subset of I/O ports, but also to a specific HBM stacks closer to the subset of I/O ports. Section 4.2 articulates the placement policy in detail. New extensions to backend toolchains will allow specifying the NAR policy and allocate memory to the statically partitioned address ranges. The information is then stored in the page table and

NUNA-Aware Routing

We propose NAR to directly address the unnecessarily long paths, and NoC congestion thereby, of inter-GPU flits between CUs and I/O ports. The baseline routing algorithm distributes off-chip traffic across all I/O ports for load balancing. Instead, NAR routes flits only to a subset of I/O ports physically closer to the CU to minimize the internal latency. 4.1.1 Baseline I/O Routing Policy. For pod-scale systems, a singlelevel Clos is emerging as the dominant inter-GPU topology since it achieves one-hop routing for peer-to-peer communication. Each I/O port per GPU attaches to a separate switch plane [4, 12]. The address to I/O ports mapping is typically determined by a boottime manager (e.g., NVIDIA Fabric Manager [65]). Commonly, it statically hashes lower-order physical address bits. This simple approach balances scale-up loads across all available ports, achieving improved bandwidth. Figure 5 illustrates an example of this baseline hashing scheme, where the white segment uniformly hashes physical addresses to all (four) possible I/O ports. 4.1.2 Improved I/O Routing Policy. NAR routes latency-sensitive traffic to only a subset of I/O ports physically closer to the CU, referred to as a NAR domain. To balance latency and bandwidth, we introduce NAR granularity, defined as the number of available NAR domains. Higher NAR granularity denotes fine-grained NAR group, each with smaller number of I/O ports. This scheme helps each CU 5

retrieved when the GPU accesses memory. Pages allocated to specified NAR granularities and domains simply need to have the correct virtual-to-physical translation, and hardware will handle off-chip routing accordingly. Within a NAR domain, remote traffic is still load-balanced amongst the subset of I/O ports. This segmentation of the address space will have a negligible impact on the overall GPU memory capacity because we target small collectives and only a few buffers; the size of segments is much smaller (e.g., 100s of MB) than the entire physical address space (e.g., 100s of GB). Furthermore, the size of the segments can be determined at boot to adjust to communication demand. To maximize benefit, NAR must avoid hotspotting on the I/O ports. Generally, for latency-bound collectives, load balancing to improve bandwidth is not a concern but for large collectives. In fact, as evaluated in Section 6, the speedups from NAR diminish for larger collectives. Thus, we rely on the compiler to select whether to use NAR for a collective or leverage the baseline routing scheme for better load balancing. It may exhaustively try all available NAR granularities (the algorithm runs quickly) or use heuristic thresholds1 .

4.2

and greedily places the threadblock(s) and chunk page(s) in the order physically closer to the I/O ports, keeping track of resource usage/contention. This process maintains compute-to-memory locality by assigning related threadblocks and buffers to the same logical group. For simplicity, we assume the communication schedule is explicitly provided via a collective plan (e.g., MSCCLang [25]), providing exact threadblock access patterns. Alternatively, if explicit plans are unavailable, these access patterns can be detected automatically from kernel code via static compiler analysis [18, 19, 47, 49, 50, 54, 108]. The plan statically defines chunks as fixed-size regions with a GPU buffer uniquely identified by the tuple (chunk_id, buffer_type, gpu_id). NAP colocates threadblocks that communicate with the same chunks via logical_allocation and places these threadblocks/chunks in appropriate NAR segments via physical_ placement. We use 4 kB pages, allowing NAP to control placement for even the smallest of latency-sensitive collectives. 4.2.1 Logical Allocation. logical_allocation assigns threadblocks and chunks into logical groups, resolving access dependencies before physical placement to prevent order-dependent fragmentation. Because hardware resources (𝑅𝑚𝑎𝑥 ) are limited within each NAR domain, performing physical_placement directly could split related groups across the die if they exceed resource budgets during sequential processing. By first partitioning the workload into logical groups, NAP can load-balance them across the GPU, ensuring related tasks remain colocated. Collective decomposition identifies access patterns. Next, logical_ allocation processes threadblocks sequentially to group them and their associated chunks into logical groups—threadblocks and chunks intended to be co-located on the same NAR domain and nearby CU/HBM. If a threadblock’s chunks are unassigned, it forms a new logical group; otherwise, the threadblock and chunks join the existing group with which it shares the most chunks to maximize overlap. This greedy policy prioritizes colocation and may produce groups of varying sizes depending on access patterns. logical_ allocation returns a list of logical groups, 𝐿, sorted in decreasing order of size, naturally load balancing subsequent physical_ placement. In practice, well-structured collective plans tend to minimize overlap. Many threadblocks access mutually exclusive chunk tuples (e.g., disjoint regions of output/input/scratch per GPU) through independent communication streams. Thus, logical allocation often produces domains that are perfectly or nearly disjoint in chunk access (e.g., Figure 7), and cross-domain accesses are rare. We handle conflicts deterministically in physical_placement.

NUNA-Aware Placement

At a high level, NAP not only places threadblocks and memory closer to themselves; we put them to the CUs and HBM stacks closer to the I/O ports (i.e., outwards to the socket) to minimize the inter-socket latency. In Figure 6, we illustrate our two proposed NUNA-aware optimizations, NAR and NAP. Figure 6c shows NUMA-optimized threadblock scheduling and memory placement utilizing the baseline off-chip routing (Figure 6a). In the baseline, addresses a–d are mapped one-to-one to any of the ports 1–4, achieving perfect load balancing across I/O ports. While an example NUMA-optimized collective assures near local loads for all chunks, it may route a flit to far I/O ports due to naïve hashing (e.g., d) or the placement of some buffers/chunks (e.g., c). These unnecessarily long paths (red dashed arrows) (i) increase individual latency of communication and (ii) increase congestion in the middle of the NoC. Figure 6b shows how NAR modifies the I/O port hashing scheme so that each illustrated NAR domain utilizes only a subset of ports closer to the CU. In Figure 6d, we combine NAR with NAP to schedule threadblocks and place memory to optimal CUs and HBMs, respectively. Addresses a and b are placed in NAR domain X and similarly for c and d in NAR domain Y. A NUNA-aware collective uses NAR for (nearest) I/O port and NAP for threadblocks and buffers to achieve near local and remote operations. The NAP placement algorithm is provided in Algorithm 1 and a visual example of the allocation algorithm for a two-GPU All-Gather is illustrated in Figure 7. The compiler/user specifies the collective communication schedule (𝑊 ) and NAR granularity (𝑛𝑎𝑟𝑚𝑎𝑥 ) as inputs. The nearest HBM stack for each CU (derived from 𝑡𝑜𝑝𝑜𝑙𝑜𝑔𝑦) and maximum resources available (𝑅𝑚𝑎𝑥 ) are assumed to be known for an architecture. The logical_allocation phase decomposes the target collective schedule and groups threadblocks and their associated chunks into logical groups. After the logical allocation, the physical_placement phase iterates over each logical domain

4.2.2 Physical Placement. Given the NAR domains, the physical placement process calculates the threadblock-to-CU (𝑀𝑐𝑜𝑚𝑝𝑢𝑡𝑒 ) and chunk-to-HBM (𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 ) mappings. Figure 7 shows an example of mapping the logical groups onto two NAR domains. CU and stack selection under resource constraints. In Algorithm 1, each logical group is assigned to a NAR domain, 𝑛𝑎𝑟 (line 10). Round-robin allocating logical groups to NAR domains (line 12) implicitly balances I/O port usage/load. The function physical_ placement returns the chosen CU and HBM stack, 𝑐𝑢 and 𝑠𝑡𝑎𝑐𝑘, for each threadblock and buffer in the logical group (line 16). For each threadblock, it selects the first CU whose incremental load

1 For individual GPU architectures, the optimal collective size threshold for using NAR

can be statically determined.

6

Figure 7: Visual example of logical allocation and physical placement steps for a two-GPU, two-channel All-Gather implemented as a local load, remote store using a single buffer. The threadblocks w-z in GPUs 0 and 1 are locally loading (solid arrow) and remotely storing (dashed arrow) from chunks a-d. In Logical Allocation, threadblocks and local/remote chunks are logically grouped (colored) and allocated to NAR domains 0-3 in round-robin fashion. In Physical Placement, the threadblocks are mapped to CUs and pages backing chunks are mapped to HBM stacks. For example, threadblock w on GPU 0 performs a near local load and a remote store using the shortest path(s) through NAR domain 0 Algorithm 1 NUNA-Aware Placement

allocation processing order. For sub-page chunks, we pack multiple chunks into the first-chosen page, consistent with a page-based virtual-to-physical translation.

1: Given: 𝑡𝑜𝑝𝑜𝑙𝑜𝑔𝑦 ⊲ CU, HBM, and I/O locations in topology 2: Given: 𝑅𝑚𝑎𝑥 ⊲ Resource constraints 3: Input: 𝑛𝑎𝑟𝑚𝑎𝑥 ⊲ # NAR domains 4: Input: 𝑊 = { (𝑔𝑝𝑢, 𝑡𝑏, 𝐶𝑡𝑏 ), . . .} ⊲ Workload description 5: Output: 𝑀𝑐𝑜𝑚𝑝𝑢𝑡𝑒 : 𝑡𝑏 ↦→ 𝑐𝑢 ⊲ Threadblock to CU assignment 6: Output: 𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 : 𝐶𝑡𝑏 ↦→ 𝑠𝑡𝑎𝑐𝑘 ⊲ Chunk to HBM assignment 7: ∀𝑐𝑢, 𝑅 [𝑐𝑢 ] ← 𝑅𝑚𝑎𝑥 8: ∀𝑠𝑡𝑎𝑐𝑘, 𝑅 [𝑠𝑡𝑎𝑐𝑘 ] ← 𝑅𝑚𝑎𝑥 9: 𝑛𝑎𝑟 ← 0 10: 𝐿 ← logical_allocation(𝑊 ) 11: for 𝐿𝑖 ∈ 𝐿 do 12: 𝑛𝑎𝑟 ← (𝑛𝑎𝑟 + 1) mod 𝑛𝑎𝑟𝑚𝑎𝑥 ⊲ Round-robin NAR domains 13: for (𝑡𝑏 𝑗 , 𝐶 𝑗 ) ∈ 𝐿𝑖 do 14: 𝑐𝑠 ← |𝐶 𝑗 | ⊲ Chunk size 15: ⊲ NUNA- and state-aware 16: 𝑐𝑢, 𝑠𝑡𝑎𝑐𝑘 ← physical_placement(𝑛𝑎𝑟 ,𝑐𝑠 ,𝑅 , 𝑡𝑜𝑝𝑜𝑙𝑜𝑔𝑦 ) 17: 𝑅 [𝑐𝑢 ] ← 𝑅 [𝑐𝑢 ] − 𝑐𝑠 ⊲ Update CU resource 18: 𝑅 [𝑠𝑡𝑎𝑐𝑘 ] ← 𝑅 [𝑠𝑡𝑎𝑐𝑘 ] − 𝑐𝑠 ⊲ Update memory resource 19: 𝑀𝑐𝑜𝑚𝑝𝑢𝑡𝑒 [𝑡𝑏 𝑗 ] ← 𝑐𝑢 20: 𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 [𝐶 𝑗 ] ← 𝑠𝑡𝑎𝑐𝑘 21: end for 22: end for

4.2.3 Implementing NAP. Algorithm 1 outputs two logical maps: threadblock-to-CUs (𝑀𝑐𝑜𝑚𝑝𝑢𝑡𝑒 ) and chunk-to-stack (𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 ). Realizing these allocations in hardware requires controlling two hardware mechanisms: the CU(s) that run a threadblock and the stack that holds a page. Existing hardware APIs can implement the threadblock allocation, with changes similar to those in existing academic and industry works allowing control of page-to-stack placement. Threadblock Dispatch. The dispatcher launches threadblocks to a subset of preferred CU(s) as specified through existing APIs. Existing hardware exposes scheduling at the CU granularity [8, 69]. Static control of threadblock placement at this granularity is also a standard assumption in prior GPU locality works [96, 108]. Assuming hardware support, the only requirement is passing GPUspecific topology and CU mappings (i.e., 𝑡𝑜𝑝𝑜𝑙𝑜𝑔𝑦 in Algorithm 1) to the compiler. Contention between existing threadblocks and NAP/collective threadblocks can be resolved by ignoring the CU affinity and defaulting to the baseline dispatch policy. Memory Placement. The driver’s memory manager realizes 𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 by mapping the virtual address(es) of a chunk’s page(s) onto the physical addresses backed by the desired stacks. We assume that NAR domains are statically partitioned into per-stack regions as illustrated in Figure 5. These partition boundaries can be configured at boot via firmware modifications to the data fabric routing and memory manager. The communication backend toolchain/compiler decides memory placement using the stackto-I/O spatial relationship (i.e., 𝑡𝑜𝑝𝑜𝑙𝑜𝑔𝑦 in Algorithm 1). Using that spatial mapping, the driver implements the desired placement policy without making any changes to address translation or the memory controller. Partitioning the physical address space into hardware-partitioned memory domains has existing implementations in GPUs (AMD NPS [73, 92] and NVIDIA MIG [70]) and CPUs (Linux OS memory drivers [52, 95]). Prior locality works control placement at the data structure [96], region [102], or page [40] granularity.

fits the CU resource budget and an HBM stack, preferring the CU’s nearest stack. The scheme can be extended to select a set of CUs if CU resource contention is expected to be high. In Algorithm 1, 𝑅𝑚𝑎𝑥 specifies the limiting resource(s) for CUs and memory. For simplicity of representation, we show the resource tracking variable, 𝑅, as keying off the resource type (CU or HBM stack IDs) and accepting chunk size (𝑐𝑠) in bytes inputs that are translated to the correct resource (outstanding requests or bandwidth). For CUs, 𝑅𝑚𝑎𝑥 represents the maximum outstanding request capacity (e.g., 700 requests per CU). For HBM stacks, 𝑅𝑚𝑎𝑥 is calculated based on the bandwidth-delay product, estimating the volume of data that can be in flight before saturation. After each decision, the resource state is updated (lines 17–18), and 𝑀𝑐𝑜𝑚𝑝𝑢𝑡𝑒 and 𝑀𝑚𝑒𝑚𝑜𝑟 𝑦 are recorded (lines 19–20). Page-level placement and conflicts. The first chunk that triggers the page placement fixes the page’s stack, and (potential) subsequent chunks in the same page may incur a non-ideal placement. If multiple chunks would prefer different stacks for the same page, we resolve the conflict by first-come, first-served in the logical_ 7

Table 2: Simulated system parameters. Config. Dies Figure 8: Evaluated on-chip network. 24×6 mesh connects the CUs in compute dies (yellow). Mesh routers in vertical edge connect to 12 I/O ports (green). Routers in the horizontal edges connect to HBM stacks (purple) with full connectivity to all 16 memory channels within each stack.

NoC Memory

5 Methodology 5.1 Simulation Infrastructure

Switch

We evaluate collective communication performance using the ASTRAsim simulator, which is real-system correlated and widely adopted [20, 37, 97]. Specifically, we use ASTRA-sim 3.0 [98] that models threadblocklevel operations at the CU-level. ASTRA-sim 3.0’s GPU model simulates NoC and scale-up network transactions at the 256 B cacheline-sized flits granularity. Each compute and reduction operation consumes CU resources (i.e., blocking other threadblocks) using a delay model based on hardware floating point operations per second (FLOPS) specifications. Also, we add a constant 120 ns latency for each memory access. This is because we evaluate collective communications without data reuse, resulting in cache misses. Finally, we evaluate latency-optimized collective algorithm (i.e., direct/allpairs) with the best-performing number of threadblocks, using threadblock-level MSCCLang collective representation [25].

5.2

End-to-End LLM Evaluation Methodology

For end-to-end evaluations, we target 12 dense and sparse (i.e., MoE) LLM architectures: Llama (7B/70B/405B) [36, 62], GPT-OSS (20B/120B) [71], Mixtral (8×7B/8×22B) [43], Qwen3 (30B-A3B/235B-A22B) [100], and DeepSeek (V3.2/V4-Flash/V4-Pro) [28, 29]. We used an in-house tool to generate forward pass inference trace (tensor shapes, kernel signatures, and communication events) for each input model, hardware, and parallelization configurations. Using system parameters from Table 2, the tool calculates compute time through a roofline model. Similarly, using NoC and scale-up network parameters, it estimates the collective communication time. Note that we model compute and communication serially since intra-batch overlap is ongoing research [35, 55, 78, 80]. We choose the batch size, sequence length, and parallelization strategy that minimizes time-to-first-token (TTFT) for prefill and time-per-output-token (TPOT) for decode.

5.3

Value 4× compute, 2× I/O per GPU 576 CUs, 700 outstanding requests per CU 120 ns (L1) latency from CU to NoC 100 PFLOPS fp4, 25 PFLOPS fp/bf16 24×6 mesh @ 1.2GHz 10 TB/s bisection bandwidth 10 cycle hop latency DOR (X-Y) routing 40 TB/s cumulative bandwidth, 150 ns access latency 16 HBM stacks Single level Clos 144 I/O ports (72 per die) 7.2 TB/s cumulative bandwidth 500 ns die-to-die latency

scaled to match measured values in real hardware [13, 27, 44, 57]. Note our profiling of current hardware shows fine-grained latency differences consistent with a multi-hop network. The scale-up network is a single-level Clos as described in Figure 3. The baseline system hashes and distributes inter-GPU traffic across all I/O ports for load balancing. NAR is implemented as described in Section 4.1, utilizing a subset of I/O ports based on the physical address (e.g., Figure 5). We model the threadblock scheduler, memory management unit, and virtual address space after modern shared memory GPU systems. The default threadblock allocation policy schedules threadblocks across dies in a roundrobin order, prioritizing free CUs for load balancing. The baseline round-robin allocates pages to HBM stacks. To model NUMA optimizations for local data access [47, 96], we assume baseline memory placement but run Algorithm 1 for threadblock scheduling near associated memory. We evaluate multiple configurations: • UA: NUNA-unaware baseline. Threadblocks and memory pages are mapped using round-robin placement and off-chip traffic hashes across all I/O ports. • UA+NAR: NAR is applied to round-robin threadblock/memory placement. • NUMA: Proxy to NUMA-aware optimizations with baseline memory allocation and nearby CU scheduling [47, 96]. • NUMA+NAR: NAR is applied to the NUMA threadblock and memory placement. • NAP: NUNA-aware threadblock and memory placement, but traffic is hashed across all I/O ports. • NAP+NAR: Fully NUNA-aware configuration, integrating both NAR and NAP techniques. • NAP*: NUNA-aware threadblock placement while employing the baseline round-robin memory placement policy.

Evaluated System and Configurations

We evaluate a scale-up pod of multi-die GPUs2 discussed in Section 3. Table 2 shows system parameters, based on previous work [13, 27, 44, 57] and industry trends [4, 11, 76]. Figure 8 depicts the target NoC, compatible with the different design philosophies and profiling of current architecture. We evaluate a global 24×6 mesh over four dies with X-then-Y routing [26]. The hop latencies are

6 Results 6.1 Microbenchmarks We measured the speedup of NUNA-aware techniques over the baseline (UA), to determine the utility of spatial optimization. We

2 In evaluation, “number of GPUs” refers to the number of sockets. Thus, the number

of individual dies evaluated is four times that value.

8

(a) UA

(a) All-Gather

(b) NAP+NAR

Figure 10: Latency distributions of remote stores in an eightGPU, 1 MB All-Gather for baseline (UA) and (NAP+NAR).

(b) All-Reduce

Figure 11: Isolation of techniques for small (100 kB) and medium (10 MB) All-Gather for eight GPUs.

The UA baseline exhibits higher average and maximum latencies compared to the NAP+NAR configuration. The UA baseline shows 920 ns and 1,320 ns for measured average and maximum latencies, respectively. NAP+NAR records 780 ns and 1,180 ns, respectively. The bell-shaped curve of the unaware baseline histogram in Figure 10a represents the Gaussian distribution of individual path latencies in a mesh, demonstrating symmetric non-uniformity. In contrast, the NUNA-aware configuration in Figure 10b shows a positively skewed normal curve by reducing both local and remote access latencies. The long tail indicates congestion at the bottlenecking I/O ports, but the maximum latency is still 10% lower than that of the unaware configuration.

(c) All-to-All

Figure 9: Execution speedups across pod and collective sizes for All-Gather, All-Reduce, and All-to-All.

evaluated All-Gather, All-Reduce, and All-to-All collectives ranging from 100 kB to 1 GB, across scale-up pods of size 2–64 GPUs. Figure 9 plots the relative speedups. For small collectives, the speedups of NUNA-aware techniques remain consistent across both collectives and pod size configurations. There is a consistent trend where NAR alone improves performance over the UA baseline. NUMA+NAR increases speedup by reducing local latencies to I/O ports. Finally, NAP+NAR is superior by reducing both local and remote latencies. As expected, the smaller, latency-bound collectives see the highest speedups. Notably, the lower speedup of NAP+NAR for 64-GPU, 10 MB collectives over the UA+NAR configuration is due to the oversubscription of each NAR domain. Threadblocks accessing the same chunk are collocated into a NAR domain so large clusters with many threadblocks oversubscribe the resources in each NAR domain.

6.2

6.3

Isolation of Techniques

To determine the individual impact of the NUNA-aware techniques, we evaluated multiple configurations where NAR and NAP were run in isolation. A small (100 kB) and a medium (10 MB) All-Gather were run for an eight-GPU configuration. Relative speedups over the UA baseline for each configuration are given in Figure 11. Notably, for all threadblock and memory placement schemes (UA, NUMA, NAP*, and NAP), NAR improves performance. Among the different placement policies, NAR has the greatest impact on NAP. For 10 MB collectives, placement policies (NUMA, NAP*, and NAP) alone without NAR introduce slowdowns due to NoC congestion caused by a large volume of hashed traffic without NAR. Furthermore, the isolation study highlights the importance of NAR in conjunction with NAP. The baseline placement evenly distributes threadblocks across a socket, causing traffic to experience average-case latencies. However, isolated NAP is not beneficial. Although threadblocks are placed outwards, as traffics are evenly hashed, many flits utilize I/O ports on the opposite edge and experience worst-case internal latency. This result demonstrates

Latency Distributions

To further understand how NAP+NAR mitigates intra-socket nonuniformity for inter-GPU communications, we measured the latency distributions of one-way, off-chip stores. Specifically, we evaluated an eight-GPU cluster with 1 MB All-Gather collectives for both UA and NAP+NAR configurations. The latency distributions are illustrated in Figure 10. 9

Figure 13: The speedup of NAR (UA+NAR) over the baseline (UA), over different NAR granularities, for an eight-GPU 1 MB All-Gather. The number of threadblocks was set to a high value to fill all CUs (many) or just a couple (few).

Figure 12: Speedup of NAP+NAR over the baseline (UA), and NAP+NAR scale-up network bandwidth utilization, for twoGPU All-Gather across distinct output buffer sizes. placement alone is not enough for optimizing inter-GPU communications; to mitigate the NUNA effect, both placement and routing should be jointly considered.

6.4

Sensitivity to Collective Size

Expanding on the claim that NUNA optimization is most effective for latency-bound collectives, we measured the effect of collective size on the NAP+NAR configuration. Figure 12 captures the result. NUNA-aware NAP+NAR speedup benefits are greater when the collectives are small, as the inter-GPU bandwidth is not saturated and transmission times are closer to link latencies, making them sensitive to intra-socket non-uniformity. However, as the collective size increases, the speedup diminishes. When the inter-GPU network becomes saturated, intra-socket non-uniformity is hidden by the dominant long queuing delays at the scale-up NICs and switches.

6.5

Figure 14: Impact of concurrent GEMM kernel on unaware baseline (UA) and NUNA-optimized configuration (NAP+NAR). Execution time relative to baseline (UA) of an eight-GPU, 1 MB All-Gather was measured for a sweep of matrix sizes. speed over a sweep of GEMM sizes for unaware (UA) and aware (NAP+NAR) to see when or if the collective performance degrades. Figure 14 summarizes the result. Significant on-chip GEMM traffic congests the NoC and causes the collective communication execution speed to degrade sharply. The NAP+NAR configuration saturates at a higher GEMM size, implying that the NUNA-aware configuration is more robust to concurrent execution. NUNA techniques localize the access to the I/O ports and minimize the traffic in the middle of the NoC. Therefore, the solution is more robust to the NoC congestion. The collective performance only degrades at a GEMM size two times larger than the baseline.

Sensitivity to NAR Granularity

We also measured the effect of NAR granularity on the NAR routing technique. We evaluated 1 MB All-Gather collectives with two configurations: one with high parallelization (1,008 threadblocks per GPU) and the other with low parallelization (7 threadblocks per GPU). The speedups of 2–12 NAR domains over baseline routing are plotted in Figure 13. As expected, when there are a sufficient number of threadblocks to enable effective load balancing, the configuration with the finest granularity (12 NAR domains) achieves the lowest latency and performs the best. However, when there are too few threadblocks to distribute across many NAR domains, traffic becomes unequally routed, resulting in slowdowns. This trade-off between low latency and load balancing is implicitly considered in the algorithmic NAP allocation, which determines the best number of domains based on the workload.

6.6

6.7

End-to-End LLM Execution

Finally, we evaluate the effect of NUNA-aware techniques to the end-to-end scenario. We apply the full NAP+NAR scheme on 12 models across 2–64 GPUs. Figure 15 specifically captures the decomposed runtime for 64-GPU results, where the communication is most exposed. Averaged (i.e., geometric mean) over all models and GPU counts, NUNA-aware execution reduces prefill TTFT by 6% (up to 11%) and decode TPOT by 7% (up to 28%). These end-to-end reductions follow directly from faster collectives: NAP+NAR improves communication time by a geomean of 1.32× for prefill collectives and 1.56× for decode collectives. The larger decode benefit reflects its smaller, latency-bound messages. We note Amdahl’s law bounds the end-toend gain by the compute-to-communication ratio. Because compute is unchanged, exposed communication ratio is approximately 4:1 for prefill and 5:1 for decode, respectively.

Interactions with Normal Execution

While these optimization techniques work well in isolation, they might be susceptible to forming hotspots or under-utilization when used with other kernels. To measure this effect, we evaluated the performance of NUNA techniques with an independent general matrix-multiply (GEMM) operation with 100 tiles. Its threadblocks were naively allocated across the GPU. They load the required data from input matrices, perform a computation as a static delay based on data size, and store the result to the output matrix. An eight-GPU, 1 MB All-Gather was concurrently run. We measured the collective 10

threadblock allocation to reduce inter-GPU communication [18] or improve cache locality [19, 54]. Khairy et al. [47] propose localityaware data management (LADM) to manage memory and threadblock scheduling to optimize for multi-die GPUs. LADM allocates compute and memory to reduce the amount of off-chip traffic. However, reducing inter-GPU traffic by co-locating chunks and threadblocks is not applicable to collectives, as collective communications are inherently inter-GPU. Furthermore, topology-specific information, critical for NUNA mitigation, is under-specified and cannot be extended. Kim et al. [50] use static analysis to schedule threadblocks, which improves sharing, but the work is only applicable to convolutions. Zhu et al. [108] propose spatially programmed GPU (SPGPU) to allow programmer hints on threadblock and data placement. However, it is based on tiled and strided access patterns, lacking generality, and appears to be a simplified version of others.

(a) Prefill. Lowest latency batch/parallelism per model.

8

(b) Decode. Lowest latency batch/parallelism per model.

Figure 15: End-to-end time (lower is better) of 64-GPU configuration relative to baseline. Compute (gray) includes compute and local memory operations. Baseline (UA, dark blue) and NUNA (NAP+NAR, light blue) are collective communication.

In prefill (Figure 15a), collectives occupy a substantial share of runtime but are comparatively large, so the per-collective (and end-to-end) gains are modest. In decode (Figure 15b), collectives are small and latency-bound and see the largest reductions. As a representative point, the latency-optimal Llama-70B configuration (eight-way tensor parallelism with 64 GPUs) achieves 8% and 16% end-to-end speedups for prefill and decode, respectively. All-Reduce of partial activations recovers the most time, at a per-collective geomean of 1.54×. All-to-All expert token routing dominates MoE communication, which NUNA improves by 1.48× on average. Consistent with our microbenchmarks, small messages get more benefit from NUNA: collectives up to 1 MB speed up 1.5–1.9×, decaying to 1.1× by 64 MB.

7

Conclusions

Physically large, multi-die GPUs ensure their integration into future AI systems. However, we should address the elongated wire delays to fully achieve their potential. Prior works focused on computememory locality within a socket. We emphasize that optimizing inter-GPU network operations, such as latency-sensitive collectives, requires the same attention to the increasing spatial effects. In this paper, we introduce the concept of NUNA. We demonstrate how NUNA-aware routing and NUNA-aware placement techniques speed up collectives up to 1.8×. For latency-sensitive collectives running on next-generation GPU devices, we propose that hardware designers and programmers should consider NUNA effects to maximize the performance of these impressive systems.

Acknowledgments We thank our colleagues Moumita Dey, Eris Furkan, Vinay Ramakrishnaiah, and Ruchi Shah for their revision advice and helpful discussions. We also thank Ganesh Dasika and Gabriel Loh for their reviews to improve the paper. AMD, the AMD Arrow logo, AMD Infinity Fabric, AMD Instinct, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.

Related Work

Dynamic or Reactive. Arunkumar et al. [13] introduce NUMAaware threadblock scheduling with a first-touch physical page allocation policy. However, the co-location policy neither explicitly controls the physical locations of the threadblocks on the compute die nor the physical locations of the memory chunks they access. Therefore, they cannot optimize for NUNA, as the NUNA effect arises from the physical distances between CUs, memory, and I/O ports. Milic et al. [63] propose dynamically changing link direction and caching policies to improve phased, asymmetric inter-GPU traffic. This approach is inapplicable to collectives, which are lowlatency and symmetric. Reactive solutions may incur prohibitive penalties for page migrations of up to 20–50 𝜇s [107], which can be avoided if the pages were statically allocated. Static Analysis. Kim et al. [49] perform static analysis to co-locate threadblocks and memory chunks to reduce off-chip traffic. Several works introduce compiler and static profiling techniques for

References [1] [n. d.]. GitHub - ROCm/rccl: ROCm Communication Collectives Library (RCCL) — github.com. https://github.com/ROCm/rccl. [Accessed 31-07-2025]. [2] [n. d.]. NVLink & NVSwitch for Advanced Multi-GPU Communication — nvidia.com. https://www.nvidia.com/en-us/data-center/nvlink/. [Accessed 31-07-2025]. [3] [n. d.]. SPCL - ATLAHS — spcl.inf.ethz.ch. https://spcl.inf.ethz.ch/Research/ Scalable_Networking/ATLAHS/. [Accessed 10-06-2026]. [4] 2024. NVIDIA GB200 Interconnect Architecture Analysis: NVLink, InfiniBand, and Future Trends - NADDOD Blog — naddod.com. https://www.naddod.com/blog/nvidia-gb200-interconnect-architectureanalysis-nvlink-infiniband-and-future-trends. [Accessed 31-07-2025]. [5] 2025. NVIDIA Collective Communications Library (NCCL). https://developer. nvidia.com/nccl. [Accessed 31-07-2025]. [6] Advanced Micro Devices, Inc. [n. d.]. AMD InstinctTM MI250 microarchitecture — ROCm Documentation. Online documentation page. [7] Advanced Micro Devices, Inc. 2021. Introducing AMD CDNATM 2 Architecture. White Paper. AMD. https://www.amd.com/content/dam/amd/en/documents/ instinct-business-docs/white-papers/amd-cdna2-white-paper.pdf Copyright notice shows 2021; document includes performance notes as of Jan. 27, 2022. 11

[8] Advanced Micro Devices Inc. 2025. HIP documentation; HIP 6.4.43484 Documentation. [Accessed 01-08-2025]. [9] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/ conference/osdi24/presentation/agrawal [10] AMD. [n. d.]. AMD InstinctTM MI350X GPUs. https://www.amd.com/en/ products/accelerators/instinct/mi350/mi350x.html. [Accessed 31-07-2025]. [11] AMD. 2025. AMD Advancing AI 2025. https://www.amd.com/en/corporate/ events/advancing-ai.html. [Accessed 31-07-2025]. [12] Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, and Sridhar Ramaswamy. 2022. NVIDIA Hopper Architecture In-Depth. https://developer.nvidia.com/blog/nvidia-hopper-architecture-indepth/. [Accessed 31-07-2025]. [13] Akhil Arunkumar, Evgeny Bolotin, Benjamin Cho, Ugljesa Milic, Eiman Ebrahimi, Oreste Villa, Aamer Jaleel, Carole-Jean Wu, and David Nellans. 2017. MCM-GPU: Multi-chip-module GPUs for continued performance scalability. ACM SIGARCH Computer Architecture News 45, 2 (2017), 320–332. [14] Mohammad Banikazemi, Vijay Moorthy, and Dhabaleswar K Panda. 1998. Efficient collective communication on heterogeneous networks of workstations. In Proceedings. 1998 International Conference on Parallel Processing (Cat. No. 98EX205). IEEE, 460–467. [15] Jeff Barr. 2019. Amazon ec2 update–inf1 instances with AWS inferentia chips for high performance cost-effective inferencing. AWS News Blog (2019). [16] Bradford M Beckmann and David A Wood. 2004. Managing wire delay in large chip-multiprocessor caches. In 37th International Symposium on Microarchitecture (MICRO-37’04). IEEE, 319–330. [17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [18] Javier Cabezas, Lluís Vilanova, Isaac Gelado, Thomas B Jablin, Nacho Navarro, and Wen-mei W Hwu. 2015. Automatic parallelization of kernels in sharedmemory multi-GPU nodes. In Proceedings of the 29th ACM on International Conference on Supercomputing. 3–13. [19] Li-Jhan Chen, Hsiang-Yun Cheng, Po-Han Wang, and Chia-Lin Yang. 2017. Improving GPGPU Performance via Cache Locality Aware Thread Block Scheduling. IEEE Computer Architecture Letters 16, 2 (2017), 127–131. https: //doi.org/10.1109/LCA.2017.2693371 [20] Jaehong Cho, Hyunmin Choi, and Jongse Park. 2025. LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure . IEEE Computer Architecture Letters 24, 02 (July 2025), 361–364. https://doi.org/10.1109/LCA.2025.3628325 [21] Jack Choquette. 2022. Nvidia hopper gpu: Scaling performance. In 2022 IEEE Hot Chips 34 Symposium (HCS). IEEE Computer Society, 1–46. [22] Charles Clos. 1953. A study of non-blocking switching networks. Bell System Technical Journal 32, 2 (1953), 406–424. [23] Patrick H Coppock, Brian Zhang, Eliot H Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C Mowry, and Dimitrios Skarlatos. 2025. LithOS: An operating system for efficient machine learning on GPUs. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 1–17. [24] David Correa. 2025. Artificial Intelligence Chip Market Expected to Reach $460.9 Billion by 2034. https://www.einpresswire.com/article/866225614/artificialintelligence-chip-market-expected-to-reach-460-9-billion-by-2034. [Accessed Nov. 11, 2025]. [25] Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. 2023. Mscclang: Microsoft collective communication language. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 502–514. [26] William James Dally and Brian Patrick Towles. 2004. Principles and practices of interconnection networks. Elsevier. [27] Preyesh Dalmia, Rajesh Shashi Kumar, and Matthew D Sinclair. 2024. CPElide: Efficient multi-chiplet GPU implicit synchronization. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 700–717. [28] DeepSeek-AI. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. Technical Report. DeepSeek AI. https://huggingface.co/deepseekai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf [29] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin

Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437 [30] Nikoli Dryden, Naoya Maruyama, Tim Moon, Tom Benson, Andy Yoo, Marc Snir, and Brian Van Essen. 2018. Aluminum: An asynchronous, GPU-aware communication library optimized for large-scale training of deep neural networks on HPC systems. Technical Report. Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States). [31] Ege Erdil. 2025. Inference economics of language models. arXiv preprint arXiv:2506.04645 (2025). [32] Amel Fatima, Yang Yang, Yifan Sun, Rachata Ausavarungnirun, and Adwait Jog. 2025. NetCrafter: Tailoring Network Traffic for Non-Uniform Bandwidth Multi-GPU Systems. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 1064–1078. [33] Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. 2024. Rdma over ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference. 57–70. [34] Krishnan Geeyarpuram. [n. d.]. Building Custom AI Infrastructure with NVLink Fusion. Presentation slides. https://hoti.org/assets/slides/2025_08_21_day2_ Invited_talk_NVIDIA_NVLink_Fusion.pdf [35] Raja Gond, Nipun Kwatra, and Ramachandran Ramjee. 2025. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference. arXiv preprint arXiv:2505.11329 (2025). [36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [37] Fei Gui, Kaihui Gao, Li Chen, Dan Li, Vincent Liu, Ran Zhang, Hongbing Yang, and Dian Xiong. 2025. Accelerating design space exploration for LLM training systems with multi-experiment parallel simulation. In Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (Philadelphia, PA, USA) (NSDI ’25). USENIX Association, USA, Article 25, 16 pages. [38] Nikos Hardavellas, Michael Ferdman, Babak Falsafi, and Anastasia Ailamaki. 2009. Reactive NUCA: near-optimal block placement and replication in distributed caches. In Proceedings of the 36th annual international symposium on Computer architecture. 184–195. [39] Ben Hawks, Gregor von Laszewski, Matthew D Sinclair, Marco Colombo, Shivaram Venkataraman, Rutwik Jain, Yiwei Jiang, Nhan Tran, and Geoffrey Fox. 2025. An MLCommons Scientific Benchmarks Ontology. arXiv preprint arXiv:2511.05614 (2025). [40] Kevin Hsieh, Eiman Ebrahimi, Gwangsun Kim, Niladrish Chatterjee, Mike O’Connor, Nandita Vijaykumar, Onur Mutlu, and Stephen W Keckler. 2016. Transparent offloading and mapping (TOM) enabling programmer-transparent near-data processing in GPU systems. ACM SIGARCH Computer Architecture News 44, 3 (2016), 204–216. [41] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. 2023. Tutel: Adaptive mixture-of-experts at scale. arXiv:2206.0338 (2023). https://doi.org/10.48550/arXiv.2206.03382 [42] Ziyang Jia, Laxmi N Bhuyan, and Daniel Wong. 2024. Pccl: Energy-efficient llm training with power-aware collective communication. In 2024 IEEE 42nd International Conference on Computer Design (ICCD). IEEE, 84–91. [43] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou 12

Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024). [44] Zhixian Jin, Christopher Rocca, Jiho Kim, Hans Kasan, Minsoo Rhu, Ali Bakhoda, Tor M Aamodt, and John Kim. 2024. Uncovering Real GPU NoC Characteristics: Implications on Interconnect Architecture. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 885–898. [45] Myoungsoo Jung. 2025. Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure. arXiv preprint arXiv:2507.07223 (2025). [46] Nathan Kalyanasundharam. 2025. Introducing UALink 200G 1.0 Specification. https://ualinkconsortium.org/wp-content/uploads/2025/04/UALink-1.0White_Paper_FINAL.pdf [47] Mahmoud Khairy, Vadim Nikiforov, David Nellans, and Timothy G Rogers. 2020. Locality-centric data and threadblock management for massive GPUs. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 1022–1036. [48] Changkyu Kim, Doug Burger, and Stephen W Keckler. 2002. An adaptive, nonuniform cache structure for wire-delay dominated on-chip caches. In Proceedings of the 10th international conference on Architectural support for programming languages and operating systems. 211–222. [49] Hyojong Kim, Ramyad Hadidi, Lifeng Nai, Hyesoon Kim, Nuwan Jayasena, Yasuko Eckert, Onur Kayiran, and Gabriel H Loh. 2017. CODA: Enabling Colocation of Computation and Data for Near-Data Processing. arXiv preprint arXiv:1710.09517 (2017). [50] Hyeonjin Kim and William J Song. 2023. LAS: Locality-aware scheduling for GEMM-accelerated convolutions in GPUs. IEEE Transactions on Parallel and Distributed Systems 34, 5 (2023), 1479–1494. [51] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. https://doi.org/10.1145/3600006.3613165 [52] Christoph Lameter. 2013. NUMA (Non-uniform memory access): An overview: NUMA becomes more common because memory controllers get close to execution units on microprocessors. Queue 11, 7 (2013), 40–51. [53] Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R Tallent, and Kevin J Barker. 2019. Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect. IEEE Transactions on Parallel and Distributed Systems 31, 1 (2019), 94–110. [54] Ang Li, Shuaiwen Leon Song, Weifeng Liu, Xu Liu, Akash Kumar, and Henk Corporaal. 2017. Locality-aware CTA clustering for modern GPUs. ACM SIGARCH Computer Architecture News 45, 1 (2017), 297–311. [55] Zonghang Li, Wenjiao Feng, Mohsen Guizani, and Hongfang Yu. 2024. TPILLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices. arXiv:2410.00531 [cs.DC] https://arxiv.org/abs/2410.00531 [56] Tim Luhnen, Tobias Marschner, and Sohan Lal. 2024. Benchmarking Thread Block Cluster. In 2024 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1–7. [57] Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, and Xiaowen Chu. 2024. Benchmarking and dissecting the nvidia hopper gpu architecture. In 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 656–667. [58] Clemens Lutz, Sebastian Breß, Steffen Zeuch, Tilmann Rabl, and Volker Markl. 2020. Pump Up the Volume: Processing Large Data on GPUs with Fast Interconnects. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (Portland, OR, USA) (SIGMOD ’20). Association for Computing Machinery, New York, NY, USA, 1633–1649. https: //doi.org/10.1145/3318464.3389705 [59] Xiaoyu Ma and David Patterson. 2026. Challenges and Research Directions for Large Language Model Inference Hardware. arXiv preprint arXiv:2601.05047 (2026). [60] Brandi Martina and Liz Stine. [n. d.]. AMD Unveils Strategy to Lead the $1 Trillion Compute Market and Accelerate Next Phase of Growth. Advanced Micro Devices, Inc. https://ir.amd.com/news-events/press-releases/detail/1266/amdunveils-strategy-to-lead-the-1-trillion-compute-market-and-acceleratenext-phase-of-growth Press release. [61] Philip K McKinley, Yih-jia Tsai, and David F Robinson. 1995. Collective communication in wormhole-routed massively parallel computers. Computer 28, 12 (1995), 39–50. [62] AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta. com/blog/llama-4-multimodal-intelligence/, checked on 4, 7 (2025), 2025. [63] Ugljesa Milic, Oreste Villa, Evgeny Bolotin, Akhil Arunkumar, Eiman Ebrahimi, Aamer Jaleel, Alex Ramirez, and David Nellans. 2017. Beyond the socket: NUMA-aware GPUs. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture (Cambridge, Massachusetts) (MICRO-50 ’17). Association for Computing Machinery, New York, NY, USA, 123–135. https:

//doi.org/10.1145/3123939.3124534 [64] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie Amy Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komuravelli, Ching-Hsiang Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng, Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, K. R. Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Ajit Mathews, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao. [n. d.]. Software-hardware codesign for fast and scalable training of deep learning recommendation models. https://doi.org/10.48550/arXiv.2104.05158 arXiv:2104.05158 [cs] NVIDIA Fabric Manager NVIDIA Fabric Manager — [65] NVIDIA. [n. d.]. docs.nvidia.com. [Accessed 31-07-2025]. [66] NVIDIA Corporation. 2016. NVIDIA Tesla P100: The Most Advanced Datacenter Accelerator Ever Built — Featuring Pascal GP100, the World’s Fastest GPU. Whitepaper WP-08019-001 v01.1. NVIDIA. https://images.nvidia.com/content/ pdf/tesla/whitepaper/pascal-architecture-whitepaper.pdf [67] NVIDIA Corporation. 2022. The NVLink Network Switch. Presentation slides, Hot Chips 34 (HC34). https://hc34.hotchips.org/assets/program/conference/ day2/Network%20and%20Switches/NVSwitch%20HotChips%202022%20r5.pdf [68] NVIDIA Corporation 2023. NVIDIA NVLink SGXLS10 Switch Systems User Manual: Introduction. NVIDIA Corporation. https://docs.nvidia.com/networking/ display/sgxh100/introduction Last updated Dec. 13, 2023. [69] NVIDIA Corporation. 2026. NVIDIA CUDA C++ Programming Guide. NVIDIA Corporation. Version 13.3. [70] NVIDIA Corporation 2026. NVIDIA Multi-Instance GPU User Guide. NVIDIA Corporation. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latestn [71] OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] https://arxiv.org/abs/2508.10925 [72] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, 13

[83] Gabin Schieffer, Ruimin Shi, Stefano Markidis, Andreas Herten, Jennifer Faj, and Ivy Peng. 2024. Understanding Data Movement in AMD Multi-GPU Systems with Infinity Fabric. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 567–576. [84] David Schor. 2021. 5th Gen CoWoS-S Extends 3 Reticle Size. https://fuse. wikichip.org/news/6031/5th-gen-cowos-s-extends-3-reticle-size/. [Accessed 31-07-2025]. [85] Aashaka Shah, Abhinav Jangda, Binyang Li, Caio Rocha, Changho Hwang, Jithin Jose, Madan Musuvathi, Olli Saarikivi, Peng Cheng, Qinghua Zhou, et al. 2025. MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications. arXiv preprint arXiv:2504.09014 (2025). [86] Jaya Shankar, TJ Xu, and Tejash Shah. 2025. Optimizing for Low-Latency Communication in Inference Workloads with JAX and XLA. NVIDIA Technical Blog. https://developer.nvidia.com/blog/optimizing-for-low-latencycommunication-in-inference-workloads-with-jax-and-xla/ Blog post. [87] Siyuan Shen, Tommaso Bonato, Zhiyi Hu, Pasquale Jordan, Tiancheng Chen, and Torsten Hoefler. 2025. ATLAHS: An Application-centric Network Simulator Toolchain for AI, HPC, and Distributed Storage. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 349–367. https://doi.org/10.1145/3712285.3759838 [88] Siddharth Singh, Mahua Singh, and Abhinav Bhatele. 2025. The Big Send-off: High Performance Collectives on GPU-based Supercomputers. arXiv preprint arXiv:2504.18658 (2025). [89] Alan Smith, Eric Chapman, Chintan Patel, Raja Swaminathan, John Wuu, Tyrone Huang, Wonjun Jung, Alexander Kaganov, Hugh McIntyre, and Ramon Mangaser. 2024. 11.1 AMD InstinctTM MI300 series modular chiplet package–HPC and AI accelerator for exa-class systems. In 2024 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 67. IEEE, 490–492. [90] Alan Smith, Gabriel H Loh, John Wuu, Samuel Naffziger, Tyrone Huang, Hugh McIntyre, Ramon Mangaser, Wonjun Jung, and Raja Swaminathan. 2024. AMD Instinct™ MI300X Accelerator: Packaging and Architecture Co-Optimization. In 2024 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits). IEEE, 1–2. [91] Srinivas Sridharan, Taekyung Heo, Louis Feng, Zhaodong Wang, Matt Bergeron, Wenyin Fu, Shengbao Zheng, Brian Coutinho, Saeed Rashidi, Changhai Man, et al. 2023. Chakra: Advancing performance benchmarking and co-design using standardized execution traces. arXiv preprint arXiv:2305.14516 (2023). [92] Andrew Tee, Nicholas Curtis, Noah Wolfe, and Daniel Wong. 2025. The MALL is Open: Exploring Shared Caches and Latency in AMD CDNA™ 3 GPUs. In Proceedings of the SC’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1110–1116. [93] Ajay Tirumala and Raymond Wong. 2024. Nvidia blackwell platform: Advancing generative ai and accelerated computing. In 2024 IEEE Hot Chips 36 Symposium (HCS). IEEE Computer Society, 1–33. [94] Walker J Turner, John W Poulton, John M Wilson, Xi Chen, Stephen G Tell, Matthew Fojtik, Thomas H Greer, Brian Zimmer, Sanquan Song, Nikola Nedovic, et al. 2018. Ground-referenced signaling for intra-chip and short-reach chip-tochip interconnects. In 2018 IEEE Custom Integrated Circuits Conference (CICC). IEEE, 1–8. [95] Ben Verghese, Scott Devine, Anoop Gupta, and Mendel Rosenblum. 1996. Operating system support for improving data locality on CC-NUMA compute servers. SIGPLAN Not. 31, 9 (Sept. 1996), 279–289. https://doi.org/10.1145/248209.237205 [96] Nandita Vijaykumar, Eiman Ebrahimi, Kevin Hsieh, Phillip B Gibbons, and Onur Mutlu. 2018. The locality descriptor: A holistic cross-layer abstraction to express data locality in GPUs. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 829–842. [97] Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, Yang Liu, Pengcheng Zhang, Kun Qian, Kunling He, Jiaqi Gao, Ennan Zhai, Dennis Cai, and Binzhang Fu. 2025. SimAI: unifying architecture design and performance tuning for large-scale large language model training with scalability and precision. In Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (Philadelphia, PA, USA) (NSDI ’25). USENIX Association, USA, Article 29, 18 pages. [98] William Won, Jinsun Yoo, Tuan Ta, Moumita Dey, Andy Balogh, Pradosh Datta, Furkan Eris, Conor Green, Winston Liu, Changhai Man, Kingshuk Mandal, Amos Rai, Vinay Ramakrishnaiah, Ruchi Shah, David Sidler, Harsh Sikhwal, Hanjiang Wu, Tushar Krishna, and Bradford M. Beckmann. 2026. ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling. arXiv:2606.10440 [cs.DC] https://arxiv.org/ abs/2606.10440 [99] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of machine learning and systems 4 (2022), 795–813. [100] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025).

Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774 [73] Muhammad Osama, Ryan Swann, Karthik Sangaiah, Sonali Singh, and Ganesh Dasika. [n. d.]. Deep dive into the MI300 compute and memory partition modes; ROCm Blogs — rocm.blogs.amd.com. https://rocm.blogs.amd.com/softwaretools-optimization/compute-memory-modes/README.html. [Accessed 31-072025]. [74] Myle Ott, Sam Shleifer, Min Xu, Priya Goyal, Quentin Duval, and Vittorio Caggiano. [n. d.]. Fully Sharded Data Parallel: faster AI training with fewer GPUs. https://engineering.fb.com/2021/07/15/open-source/fsdp/ [75] Saptadeep Pal, Daniel Petrisko, Matthew Tomei, Puneet Gupta, Subramanian S Iyer, and Rakesh Kumar. 2019. Architecting waferscale processors-a gpu case study. In 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 250–263. [76] Dylan Patel, Xie Myron, Daniel Nishball, Ivan Chiam, Patrick Zhou, Doug OLaughlin, and Wega Chu. [n. d.]. NVIDIA GTC 2025 – Built For Reasoning, Vera Rubin, Kyber, CPO, Dynamo Inference, Jensen Math, Feynman. https://semianalysis.com/2025/03/19/nvidia-gtc-2025-built-for-reasoningvera-rubin-kyber-cpo-dynamo-inference-jensen-math-feynman/. [Accessed 31-07-2025]. [77] Dylan Patel, Myron Xie, and Gerald Wong. 2023. AI Capacity Constraints—CoWoS and HBM Supply Chain. SemiAnalysis.[Online]. Available: https://www. semianalysis. com/p/ai-capacity-constraints-cowos-and (2023). [78] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132. [79] Suchita Pati. 2024. Cross-Stack Optimizations for Sequence-Based Models on GPUs. The University of Wisconsin-Madison. [80] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. 2024. T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 1146–1164. https://doi.org/10.1145/3620665.3640410 [81] David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, LluisMiquel Munguia, Daniel Rothchild, David R So, Maud Texier, and Jeff Dean. 2022. The carbon footprint of machine learning training will plateau, then shrink. Computer 55, 7 (2022), 18–28. [82] Le Qin, Junwei Cui, Weilin Cai, Meng Niu, Yan Yang, and Jiayi Huang. 2025. Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 659–674.

14

[101] Jie Amy Yang, Jongsoo Park, Srinivas Sridharan, and Ping Tak Peter Tang. 2020. Training deep learning recommendation model with quantized collective communications. In Conference on Knowledge Discovery and Data Mining (KDD), Vol. 95. [102] Jialiang Zhang, Michael Swift, and Jing Li. 2022. Software-defined address mapping: a case on 3d memory. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 70–83. [103] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). [104] Xia Zhao, Magnus Jahre, Yuhua Tang, Guangda Zhang, and Lieven Eeckhout. 2023. NUBA: Non-uniform bandwidth GPUs. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 544–559. [105] Xuanlei Zhao, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. 2024. HeteGen: Efficient Heterogeneous Parallel Inference for Large

Language Models on Resource-Constrained Devices. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 162–172. https://proceedings.mlsys.org/paper_files/paper/2024/file/ 5431dca75a8d2abc1fb51e89e8324f10-Paper-Conference.pdf [106] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 62557–62583. https://doi.org/10.52202/079017-2000 [107] Tianhao Zheng, David Nellans, Arslan Zulfiqar, Mark Stephenson, and Stephen W Keckler. 2016. Towards high performance paged memory for GPUs. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 345–357. [108] Shizhuo Zhu, Illia Shkirko, Jacob Levinson, Zhengrong Wang, and Tony Nowatzki. 2024. SPGPU: Spatially Programmed GPU. IEEE Computer Architecture Letters (2024).

15

Related documents

• GPUBreach: Privilege Escalation Attacks on GPUs using Rowhammer#1110228
• WarpGuard: Towards Control-Flow Attestation for Heterogeneous CPU-GPU Execution#370276
• Taming Bitwise Behavior in GPU Kernels with Tensor Core: Black-Box Reconstruction, Compiler Enforcement, and Static Verification#673469
• DB-SpMSpV: Dual-View Blocked Sparse Matrix-Sparse Vector Multiplication for Dynamic GPU Workloads#620708
• DB-SpMSpV: Dual-View Blocked Sparse Matrix-Sparse Vector Multiplication for Dynamic GPU Workloads#619029