Characterizing CUDA Locality Domains on Blackwell GPUs

Characterizing CUDA Locality Domains on Blackwell GPUs Rocky Wang∗

Joyce Zhou∗

Keith Jiang

Jason Zhao†

Scitix AI

arXiv:2611.07535v1 [cs.DC] 5 Nov 2026

Abstract A Blackwell GPU contains two compute dies, and latest CUDA provides locality domains (LDs) to control memory placement and restrict computation to a single die. Despite the introduction of these controls, their performance implications and underlying costs remain largely unexplored. We study LDs from three perspectives: memory management and access, computation, and communication. Across eight B300 GPUs, we use a controlled four-state comparison of stock execution, memory placement alone, compute binding alone, and both controls together. We also use mismatched configurations as negative controls and hardware counters to identify the causes of performance changes. Our measurements show that the benefit of locality depends on how data is accessed. A cross-die dependent HBM access costs 193.4 ns, while cross-die streaming achieves only about 27% of the baseline bandwidth. In contrast, cross-die reads add little latency once data is resident in L2, because the data can be cached in the accessing die’s local L2. These differences explain why memory placement alone improves remote reads by 2.08× for a 1 MiB buffer, while placing a 256 MiB streaming buffer on one die makes performance 47% worse. The results also show that different operations require different controls: compute binding alone speeds up SM-issued collectives by 1.495×, while resident-data GEMMs require both memory placement and compute binding to achieve a 1.37× speedup. Neither control provides a consistent benefit to a bandwidth-bound attention kernel once each configuration is allowed to select its own tile size. We also identify a substantial overhead in localized memory allocation that emerges beyond 32 MiB. The mapping loop of driver processes localized memory one 2 MiB page at a time, with 77–87% of the cost spent on full-chip barriers and TLB invalidations. By coalescing physically contiguous mappings and deferring synchronization, we reduce the time for a 1 GiB cuMemCreate operation by 7.02×. Together, our results show that LD performance depends on the interaction between memory placement, compute placement, and access patterns: locality can accelerate remote reads and computation on resident data, but can also reduce streaming performance, and its benefits must be weighed against both hardware and driver costs.

1

Introduction

A Blackwell GPU package contains two compute dies, each with its own L2 cache slices and memory controllers [9, 6]. The two dies communicate through an on-package fabric, so an access that crosses the die boundary can cost more than one that stays within a die. CUDA 13.4 exposes this hardware topology through locality domains (LDs), giving programmers more control over where data resides and where computation runs. LDs provide two independent controls. Memory can be placed on a selected die instead of being striped across both dies, using virtual memory management (VMM) or an LD-tagged memory pool [9]. Computation can be restricted to the 70 SMs on a selected die by partitioning SM resources and creating a green context on that partition [11, 1]. These controls give programmers a way to make data and ∗ †

Equal contribution. Corresponding author: jason@scitix.ai

1

computation local to each other. However, the performance effects of these controls have received little systematic study, and it is not clear when using them is worthwhile. Existing guidance on LDs generally points to performance benefits from improved locality [9, 14]. Our measurements, however, show that the results depend strongly on how data is accessed. Placing a 1 MiB buffer read by a remote GPU increases transfer bandwidth from 75.7 to 157.2 GB/s, more than doubling throughput. The same memory placement produces no measurable improvement for a 1 MiB buffer used by a local collective. For a local kernel streaming a 256 MiB buffer, it makes performance 47% worse. Thus, placing data on one die does not necessarily make an application faster. To understand why these results differ, we need to look at which component accesses the data, where that component runs, and how often data must cross the die boundary. We study LDs on eight B300 GPUs, covering memory management and access, computation, and communication. Across these experiments, we use the same four configurations: stock execution, memory placement alone, compute binding alone, and both controls together. This comparison lets us separate the effects of the two controls and determine whether they work independently or need to be combined. We also use mismatched placement as a negative control, hardware counters to examine memory behavior, and SM-count controls to distinguish locality benefits from the effects of using fewer SMs. To interpret the measurements, we develop a cost model that separates the cost of a cross-die access into two parts: the latency of each cross-die hop and the bandwidth cost of moving data. The first matters when an operation depends on each access completing before the next can proceed; the second matters when large amounts of data must be transferred. The benefit of LD therefore depends on both the access pattern and which placement control changes the path of an access. For example, placing a buffer on the appropriate die can help a remote GPU read it without changing where the copy engine runs. A local collective can instead benefit from binding its computation to one die, even when changing memory placement has little effect. A kernel that repeatedly reads resident data, such as GEMM weights, may need both controls to work together. The model turns these observations into testable predictions, which we evaluate using dependent-load and streaming microbenchmarks, hardware-counter analysis, and controlled workload experiments. Our measurements reveal several important differences in LD behavior. Cross-die dependent HBM accesses cost 193.4 ns, and cross-die streaming achieves only about 27% of baseline bandwidth. Yet cross-die reads add little latency once the data is resident in L2: the accessing die can retain a local copy, while writes continue to cross the fabric on each access. This difference helps explain why memory placement benefits some access patterns but not others. We also find that the two placement controls can complement each other. For resident-data GEMMs, memory placement alone achieves only 0.75× the baseline performance, and compute binding alone achieves 1.04×, while using both achieves 1.37×. For SM-issued collectives, compute binding alone delivers a 1.495× speedup, whereas a configuration using a comparable number of SMs across both dies achieves only 0.999×. In contrast, an apparent 1.27× improvement for a bandwidth-bound attention kernel disappears when each configuration is allowed to select its own tile size. These results show why LD performance must be evaluated with controlled comparisons rather than inferred from locality alone. We also examine the cost of allocating localized memory. Beyond a 32 MiB threshold, allocation overhead becomes substantial. We trace this cost to the open-source GPU driver’s mapping loop, which processes localized memory one 2 MiB page at a time. Full-chip barriers and TLB invalidations account for 77–87% of the cost of each call, rather than the page-table updates themselves. We modify the driver to coalesce physically contiguous mappings and defer synchronization. This reduces the time for a 1 GiB cuMemCreate operation by 7.02×, although some overhead remains. This result shows that the cost of using LDs comes not only from data access and placement, but also from the memory-management path used to establish that placement.

2

Our contributions are as follows: 1. A cost model for cross-die accesses. We develop a two-component model that separates per-hop latency from data-transfer bandwidth cost. The model predicts when memory placement, compute binding, or both should improve performance based on the accessor and the location of the data. 2. A controlled measurement methodology. We evaluate LDs across memory management and access, computation, and communication on eight B300 GPUs. By comparing stock execution, memory-only placement, compute-only binding, and full placement, and by using negative controls, hardware counters, and SM-count controls, we isolate the effects of locality from other factors. 3. A characterization of LD performance. We show that LDs can improve remote-GPU reads and SM-issued collectives, that resident-data GEMMs benefit from combining memory and compute placement, and that localized placement can substantially hurt streaming performance. We also identify read–write asymmetry, the limited effect of cross-die reads on L2-resident data, and the importance of controlling for kernel configuration when measuring speedups. 4. A driver fix for localized allocation overhead. We identify an allocation overhead that emerges beyond 32 MiB and trace it to repeated mapping calls and synchronization in the open-source driver. By coalescing contiguous mappings and deferring synchronization, we reduce the time for a 1 GiB cuMemCreate operation by 7.02×.

2

Background

2.1

Blackwell Architecture and Locality-Domain Controls

A Blackwell GPU contains two compute dies connected by an on-package fabric. Each die has its own L2 cache slices and memory controllers [6]. Accessing data on the other die therefore requires communication across this fabric. Each B300 GPU exposes two locality domains (LDs), each with 70 SMs out of 148 SMs in total [9]. The GPU has 126 MiB of L2 cache in total, or approximately 63.2 MiB per die. SM indices do not correspond to the two locality domains in a simple, contiguous split. For example, the domain-0 green context includes SMs {0–3, 6–9, 16–19, . . . , 142–147}, while domain-1 includes {4–5, 10–15, . . . , 130–135}. A range of SM indices therefore does not, by itself, tell us which die an SM belongs to. Throughout this paper, we determine compute placement from the green-context resource configuration and verify where kernels execute by sampling \%smid. CUDA provides two ways to control locality. The first is memory placement. With virtual memory management (VMM), an application can pass an LD location to cuMemCreate. It can also use a memory pool tagged with an LD. Both mechanisms use the location type CU_MEM_LOCATION_TYPE_DEVICE_ LOCALITY_DOMAIN to specify the target domain [9]. We check the resulting placement by reading back LOCALITY_DOMAIN_ORDINAL. On all eight GPUs in our experiments, this attribute reports the requested domain. By contrast, cudaMalloc does not provide an explicit way to select an LD, and its LOCALITY_DOMAIN_ORDINAL attribute reads back as −1. Our measurements show that memory allocated this way is striped across both dies. The second control is compute placement. CUDA’s cuDevSmResourceSplit API can split SM resources into groups that correspond to the two locality domains [1]. These groups can then be used to create green contexts and their associated streams, restricting kernel execution to the selected SM resources [11]. Memory placement determines where data resides, while compute placement determines which SMs execute the work. Since these controls are independent, we vary them separately and together throughout our experiments. 3

2.2

Locality Domains and GPU Partitioning Mechanisms

LDs are not the only way to control how GPU resources are used. NVIDIA also provides mechanisms such as MIG, MPS, green contexts, and MLOPart. Although some of these mechanisms can expose or use per-die resource configurations, they serve different purposes and do not provide the same placement controls. • Multi-Instance GPU (MIG). MIG divides a GPU into separate instances with dedicated resources. For instance, MIG separate a GPU into two instances, each with 70 SMs and 89 GB of memory, matching the SM count of the two locality domains [13]. The key difference is that MIG creates separate GPU instances, whereas LDs let an application control memory and compute placement without requiring a separate instance for each domain. • Multi-Process Service (MPS). MPS allows multiple processes to share a GPU and provides controls over the resources available to each client [12]. These controls determine how GPU resources are allocated among clients, but resource allocation alone does not specify which locality domain executes a client’s work. • Green contexts. Green contexts let applications create execution contexts backed by selected SM resources [11]. They provide the mechanism we use to restrict computation to one locality domain, but they do not determine where memory is allocated. • MLOPart. MLOPart exposes per-die partitions as devices that applications can target [14]. Unlike LD placement, which controls memory and compute placement within a GPU, MLOPart uses partitioned devices as the unit of resource allocation. We separately evaluate the runtime and driver overheads associated with concurrent partitions in this paper. The main difference between these mechanisms is what they control. MIG and MLOPart expose partitioned GPU resources, while MPS manages resource sharing among processes. Green contexts restrict execution to selected SM resources, and LDs provide separate controls for memory placement and compute placement. Recent work further shows that die-level structure affects GPU scheduling, and that thread block clusters let SMs cooperate within a kernel [3, 10, 7, 4]. These approaches change which SMs cooperate, but they do not by themselves control where data resides, which is the placement axis we study. These differences matter when comparing LDs with other approaches: a performance change may come from improved locality, but it may also come from changes in resource capacity or isolation. We return to this distinction in Section 8, where we use our cross-die cost model to interpret the experimental results.

3

A Cost Model for Cross-Die Access

3.1

Cost Model

To understand when locality domains improve performance, we separate execution time into the cost of individual accesses and the cost of moving bulk data. Consider a workload that transfers S bytes and performs n dependent accesses to data on the remote die. We use the following first-order model:
T = t0 + n thop + δ +

S , Beff

(1)

Here, t0 represents overheads that do not depend on placement. The term thop is the additional latency of accessing data across dies rather than locally, and δ captures the additional cost or savings associated with how the access is served after the initial remote access. The effective bandwidth, Beff , determines 4

how long it takes to transfer the remaining S bytes. This model does not capture every detail of GPU execution, such as overlapping accesses or hiding latency, but it lets us examine two distinct sources of cost: the number of accesses and the amount of data transferred. We can use the model to compare the default placement, which distributes memory across both dies, with a placement that localizes memory to one die. Let Tplain denote the execution time under the default placement and Tlocal the time under localized placement. The time saved by localization is ∆ = Tplain − Tlocal  ≈ n ∆taccess + S

1 Bplain

−

1 Blocal

 ,

(2)

where ∆taccess is the per-access cost removed by localization, and Bplain and Blocal are the effective bandwidths under the two placements. A positive ∆ means localization reduces execution time; a negative value means it makes the workload slower. The two terms can favor different placements. The first term grows with the number of accesses whose cost is reduced. The second depends on the amount of data transferred and the bandwidth available under each placement. If localization reduces access latency without sacrificing bandwidth, both terms favor localization. But if placing data on one die reduces the available bandwidth, the second term can become negative and outweigh the latency savings. This is why localization may help a workload with frequent dependent accesses but hurt a kernel that streams a large amount of data. The model gives us a way to make predictions before examining each workload. Its two key platformdependent quantities are the effective per-access cost associated with crossing dies and the bandwidth under localized placement relative to the default placement. We measure both experimentally in Section 5, and use hardware counters to investigate why the costs differ across access patterns.

3.2

Placement Hypotheses

The model estimates the benefit of removing cross-die access costs, but it does not tell us which LD control can remove them. Memory placement changes where data resides; compute placement changes where the kernel executes. Which control matters therefore depends on who issues the access and where the data is located. We use this distinction to formulate four hypotheses: • Remote accessors. When a remote GPU or copy engine reads a buffer on the local GPU, placing the buffer on the appropriate die may reduce cross-die traffic. Binding local SMs to a die should have little effect if those SMs do not issue the relevant accesses. We therefore expect memory placement to provide the main benefit. • Local SM-based collectives. When SMs issue accesses as part of a collective operation, binding execution to one die may reduce cross-die traffic. Changing memory placement alone may not help if the main cost comes from where the SMs execute and which cache or memory resources they access. We therefore expect compute placement to matter more than memory placement. • Repeated accesses to resident data. A kernel such as GEMM may repeatedly read weights that remain resident in memory or cache throughout execution. In this case, performance depends on both where the weights reside and where the SMs access them from. We expect memory placement and compute placement to work together when neither control alone fully aligns the data with its accessor. • Bandwidth-dominated streaming. A local kernel that streams a large working set may be limited mainly by memory bandwidth rather than access latency. If localization reduces the bandwidth

5

Table 1: Predicted placement effects and contrary evidence across accessor classes. Accessor class

Prediction

Contrary evidence

Remote reader

Memory placement helps; compute placement alone is neutral. Compute placement helps; memory placement alone does not. Combined placement outperforms either control alone. Localization hurts at large working-set sizes.

Compute placement alone provides a comparable gain, or memory placement does not help.

Local SM collective or copy Local SM reading resident data Bandwidth-bound streaming

Memory placement alone provides a comparable gain, or the gain is explained by SM count rather than die affinity. Either control alone achieves a comparable gain, or the combination adds no benefit. Localization continues to help at large sizes without a corresponding bandwidth explanation.

available to the kernel, that loss may outweigh the benefit of avoiding cross-die accesses. We therefore expect the default distributed layout to perform better in some bandwidth-bound workloads. These hypotheses are predictions, not assumptions that localization will help. Each predicts which control should matter for a particular access pattern, and can be rejected if the measurements show otherwise. Table 1 summarizes the predictions and the observations that would contradict them.

3.3

Alternative Explanations and Controls

A measured speedup does not necessarily mean that locality is responsible. Changing an LD configuration can also change the number of SMs available to a kernel, its launch configuration, or the effective memory bandwidth. We design the experiments in Section 4 to separate these effects. Resource allocation rather than placement. An LD configuration may use only 70 SMs, compared with 148 SMs on the full GPU. A speedup or slowdown could therefore result from changing the number of SMs rather than from keeping computation on one die. To test this possibility, we compare configurations with approximately the same SM count, placing the selected SMs either on one die or across both dies. Kernel configuration rather than placement. Different placement configurations may use different generated code or launch settings, which can affect performance independently of locality. We therefore compare configurations using the same binary where possible and check their launch configurations. For workloads where tile selection affects performance, we also compare configurations after allowing each to choose its own tile size. Bandwidth effects rather than per-access costs. A speedup at one working-set size does not, by itself, show that localization has reduced access latency. The result could instead reflect a change in effective bandwidth. We therefore sweep working-set sizes and compare the results with the model’s two terms. If the benefit changes as the amount of data transferred grows, this helps determine whether bandwidth loss offsets the savings from avoiding cross-die accesses. These controls help us determine whether the observed performance changes are consistent with the model or better explained by other factors. Section 7.4 presents the SM-count control results, while the remaining experiments examine the effects of access pattern, working-set size, and placement choice.

6

4

Measurement Design

4.1

Platform and Software Stack

All measurements run on the same platform: eight B300 SXM6 AC GPUs (compute capability 10.3), with driver 615.71.09, CUDA 13.4, NCCL 2.32.3, and nccl-tests 2.20.0. We use Nsight Compute 2026.3.1 for profiling. MIG cannot be enabled inside our container, so we do not report direct measurements of MIG performance. Instead, we use NVIDIA’s documentation and a substitute experiment to examine the relevant resource-partitioning question [13], and distinguish these results from direct MIG measurements. The experiments test the four placement hypotheses from Section 3. We first compare the effects of memory placement and compute placement, then use targeted workloads and additional controls to check whether the results are consistent with the model or can be explained by other factors.

4.2

The Four States

We compare four placement states and one negative control, summarized in Table 2. Each comparison changes the placement controls while keeping the other experimental conditions fixed as far as possible. Table 2: The four-state switch model used throughout. Any pair of numbers in this report differs only in the state applied, never in the binary or the protocol. State

Memory

Compute

Stock Memory-only Compute-only

cudaMalloc (striped across both dies) VMM or pool allocation placed in a chosen domain striped across both dies

Full-LD

chosen domain

Mismatch

domain X

all 148 SMs all 148 SMs green context pinned to one domain (70 SMs) pinned to the same domain pinned to domain Y ̸= X

Two comparisons are particularly important. First, the compute-only configuration uses 70 SMs, matching the SM count of the other LD configurations. Comparing compute-only with memory-only therefore lets us compare the two placement controls at the same SM count. This comparison alone does not rule out every effect of resource distribution, so we also run the SM-count control described below. Second, the mismatch configuration places data on one die but binds computation to the other. It keeps the kernel, memory footprint, and SM count unchanged while deliberately separating data placement from compute placement. If a performance benefit depends on bringing data and its accessor together, this configuration should behave differently from one in which they are placed consistently.

4.3

Workloads Chosen to Isolate Accessor Classes

We use one representative workload for each accessor class, together with microbenchmarks that measure the underlying access costs (Table 3). The microbenchmarks measure the two quantities in Equation (2) separately. A dependent pointer chase measures access latency with little memory-level parallelism, allowing us to compare local and cross-die accesses. Separate streaming kernels measure effective bandwidth and distinguish read and write behavior, which a combined copy operation could hide.

7

Table 3: Experimental workloads, why each was chosen, and the dimensions swept. Layer

Workload and rationale

Dimensions

Primitive

Self-written pointer chasing plus separate streaming read/write kernels; the only measurement that directly separates accessor die from data die Per-API timing of seven VMM calls, plus a memory pool nvbandwidth, which exercises copy engines and SMs as distinct accessors nccl-tests (general library) and flash-ar/flash-ag (a library that can delete algorithm phases) DeepGEMM FP8/FP4 masked and normal grouped GEMM, plus decode-attention kernels used as the bandwidth-bound counterexample MPS at 25/50/75/100%, MLOPart, MIG (documentation only)

accessor {d0, d1, unbound} × data {d0, d1, interleaved} × 64 KB–256 MB × read/write

Management Copy Communication Compute

Comparison

1 KiB–1 GiB, 30–66 MiB in 2 MiB steps, cold and warm pools, 1–8 GPUs × 1–8 threads local D2D and remote × CE/SM × 64 B–256 MiB × four states AR/AG/A2A × 2/4/8 ranks × R0/R2 × 1 KB–64 MB × four states M=1–16, hot/cold × four states plus mismatch; decode sweeps S, B and tile size \%smid landing, side-by-side comparison, skewed load

Each application workload tests a different placement hypothesis. For remote accessors, we use the copy-engine (CE) cases in nvbandwidth. In these tests, the copy engine initiates the transfer rather than an SM. They let us examine whether changing the location of the source buffer helps a remote GPU read it, without relying on compute placement to change the access path. For SM-issued communication, we use nccl-tests in symmetric-memory registration mode. These tests exercise the relevant collective communication path at different rank counts. We inspect the launch configurations and compare placement states to determine whether binding computation to one die improves performance. For repeated accesses to resident data, we use DeepGEMM’s grouped GEMM. Its eight MoE groups can be divided evenly across the two dies, allowing us to compare memory placement and compute placement separately and together when the kernel repeatedly accesses its weights. We also test a case where the model predicts little benefit. Decode attention is bandwidth-bound, and the split configuration studied here has only one cross-die stage: the LSE reduction. Once the kernel approaches the bandwidth limit, removing this cross-die stage may have little effect on total execution time. Section 6.7 tests this prediction. We also allow each configuration to select its own tile size. The apparent 1.27× gain disappears under this comparison, showing that the initial difference came from tile selection rather than a consistent benefit from locality.

4.4

Controls for Competing Explanations

Resource count. All LD configurations use 70 SMs, but matching the SM count does not by itself show that a speedup comes from die affinity. We therefore use MPS at 50% and 25% resource allocations to test configurations with a comparable number of SMs distributed across both dies. Before running the experiment, we set the decision rule in the analysis script. If the MPS control recovers at least the predefined 1.25× speedup threshold, the result favors the explanation that resource allocation accounts for the gain. If performance remains near 1.0×, the result supports the explanation that keeping computation on one die matters. This comparison helps distinguish the two explanations, although it does not eliminate every possible difference in resource distribution. Kernel and launch configuration. A placement comparison can be misleading if the configurations also use different kernels or launch settings. For comparisons using the same binary, we record the grid

8

and block dimensions in each state and check whether they change. Matching launch configurations rules out differences in grid and block geometry as the direct explanation for a speedup, although it does not rule out every difference in execution behavior or resource allocation. Bandwidth effects. We vary message size or batch size in the application experiments to see how the placement benefit changes with the amount of data transferred. The model predicts that the latencyrelated benefit depends on the number of affected accesses, whereas the bandwidth cost grows with the volume of data. If increasing the transfer size eventually reduces or reverses the benefit, and the crossover agrees with the measured latency and bandwidth terms, this supports the model. A trend that these measurements cannot explain would count against it.

4.5

Threats to Validity

Our experiments have two main limitations. Limited access to collective-path counters. Both Nsight Compute replay modes fail when used with NCCL’s memory management in our container. We therefore cannot directly use fabric-traffic counters to establish why the collective-path performance changes. Instead, we rely on the SM-count control and launch-configuration comparisons. We treat these results as indirect evidence and distinguish them from mechanisms that we confirm directly with hardware counters. MIG cannot be instantiated. We cannot enable MIG in our experimental environment. Our comparison with MIG therefore relies on NVIDIA’s documentation [13] and a substitute experiment that tests the relevant resource-partitioning premise. We do not present the substitute experiment as a direct measurement of MIG performance. We also take several steps to keep measurements reproducible. A global lock serializes experiments because concurrent bandwidth probes interfere with one another. We bind each GPU to its corresponding NUMA node, using node ⌊i/2⌋ for GPU i. For every LD allocation, we read back the locality-domain ordinal and discard the allocation if the reported domain does not match the requested one.

5

Memory Placement and Access

5.1

Research Questions

Equation (2) describes two ways in which memory placement can affect execution time. It can reduce the cost of individual accesses, but it can also change the effective bandwidth available to a workload. Which effect matters more depends on the access pattern. This section measures both costs, uses hardware counters to examine the difference between reads and writes, and tests whether memory placement helps remote accessors as predicted by our model. We also measure the cost of allocating localized memory. Unlike the cost of accessing data, allocation overhead is paid when memory is created or released. We measure this overhead separately and determine how much reuse is needed to recover it.

5.2

Method

An end-to-end copy time does not tell us how much of the cost comes from access latency and how much comes from bandwidth. It also combines the behavior of the accessor with that of the memory being accessed. We therefore use two microbenchmarks to measure these costs separately. Dependent-load latency. We use a pointer chase in which each load depends on the result of the previous load. This dependency 9

prevents the kernel from issuing many independent memory requests at once, making the result more sensitive to access latency than to aggregate throughput [6]. The kernel uses cache-global loads that bypass L1, and we sweep working-set sizes from 64 KB to 256 MB. A green context binds the accessor to a selected die, while VMM places the data on a selected die. We also use interleaved allocations from cudaMalloc as a baseline. By testing all combinations of accessor and data placement, including working sets both smaller and larger than one die’s L2 capacity, we can distinguish accesses served from cache from those that reach remote DRAM. Streaming read and write. We use separate kernels to stream data in each direction. This avoids combining reads and writes in a single memcpy, where one direction’s behavior could obscure the other. Each kernel processes a 256 MB buffer, well beyond L2 capacity, to measure the steady-state effective bandwidth Beff in Equation (1). We report the median across three independent process runs. For every LD allocation, we read back the locality-domain ordinal and discard the allocation if it does not match the requested domain.

Results

Dependent-load latency (ns, median)

5.3

accessor die0 → data die0 accessor die0 → data die1

6 × 102

whole device (interleaved)

+193 ns 4 × 10

2

3 × 102

2 × 102

102 64 KiB

256 KiB

1 MiB

4 MiB

16 MiB

64 MiB 256 MiB

Working set

Figure 1: Dependent-load latency (load.global.cg, bypassing L1; median of three runs). The three curves coincide while the working set fits in one die’s L2; only at 256 MB, where every access must come from HBM, does the cross-die curve separate.

Table 4: Primitive-level key points, accessor pinned to die 0; “interleaved” is the stock cudaMalloc layout. Latency in ns, bandwidth in GB/s; each cell is a median of three independent processes. Metric

Working set

Same die

Cross die

Interleaved

Read latency

8 MB (fits L2) 256 MB (HBM)

135.8 295.3

138.8 488.8

137.3 466.1

Read bandwidth

8 MB 256 MB

7592.6 3628.1

7549.7 2645.7

7612.2 5229.8

Write bandwidth

8 MB 256 MB

3699.4 3219.5

2439.7 2351.6

3678.8 3795.2

Table 4 and Figure 1 show three main results. 10

Cross-die latency increases sharply when accesses reach remote DRAM. For the 256 MB working set, dependent-load latency is 295.3 ns for same-die access and 488.8 ns for cross-die access, a difference of 193.4 ns. At 8 MB, the difference is only 3.0 ns. Across working sets from 64 KB to 32 MB, the measured difference stays between 2.9 and 3.4 ns. The result is similar when we reverse the accessor and data dies: the additional latency is 191.5 ns. Across all eight GPUs, the measured increment ranges from 191.4 to 193.3 ns (Section 7.6). The large penalty therefore appears when dependent accesses reach remote DRAM; crossing a die boundary does not add the same cost to every access. Reads and writes respond differently to cross-die placement. For L2-resident data, cross-die reads add little measured latency. Writes behave differently: cross-die writes reduce bandwidth from 3699.4 to 2439.7 GB/s, a 34.0% decrease. A single direction-independent cost cannot explain both results. We use hardware counters below to examine where the difference comes from. Restricting memory to one die can reduce streaming bandwidth. For the 256 MB streaming workload, the interleaved layout reaches 5229.8 GB/s, compared with 3628.1 GB/s when memory is localized to one die. Interleaving allows the workload to use memory channels on both dies, while localization restricts the available bandwidth to one die. The bandwidth loss can outweigh the savings from avoiding cross-die accesses, particularly when the workload transfers large amounts of data. Streaming read (256 MiB)

Streaming write (256 MiB)

6.98

7

6.49

Bandwidth (TB/s)

6 5 4

3.63

3.22 2.65

3

2.35

2 1 0 same die

cross die

whole device

same die

cross die

whole device

Figure 2: Steady-state bandwidth at 256 MiB with the accessor pinned to die 0. Cross-die read and write both fall to about 73% of the same-die figure, while an interleaved layout lets a single-die accessor use both dies’ HBM channels.

5.4

Read and Write Traffic Across Dies

The hardware counters help explain why reads and writes behave differently. The B300 does not expose a counter explicitly named for locality domains or dies. We therefore use lts__t_sectors_srcunit_ ltcfabric_op_read.sum and its write counterpart, which count sectors received from the LTC fabric. Across the four accessor–data placement combinations, these counters show different patterns for reads and writes.

11

Fabric sectors (log scale)

108

33.55M

33.54M

107 106 105 104 103 102 101

8

8

100 same-die read

cross-die read

same-die write

cross-die write

Figure 3: Cross-die fabric sectors for a single 256 MiB launch, on a log axis. Same-die traffic is not zero: it is 8 sectors against 33.5M for cross-die traffic, seven orders of magnitude apart, which is why the same-die bars are invisible on a linear scale. All four same-die combinations read exactly 8 and write exactly 0, so the residue is a fixed per-launch overhead rather than data movement. Cross-die traffic equals the full byte count (33.54M sectors × 32 B = 1.073 GB ≈ 1 GiB, i.e. one fabric crossing per byte transferred).

Cross-die reads can be served from the accessor’s local L2 after the initial access. For an 8 MiB cross-die read, we observe only 13,625 L2 misses and eight fabric sectors, despite a reported access count of 33.56M. The kernel takes 146.9 µs, compared with 144.5 µs for same-die access. This small difference agrees with the nearly flat L2-resident latency curve in Figure 1. Together, these measurements are consistent with remote data being brought into the accessing die’s local L2 and reused on subsequent accesses. The low measured overhead does not mean that a fabric hop is inexpensive. Rather, once the data is cached locally, the dependent access chain no longer needs to pay that cost on every access. Cross-die writes continue to generate fabric traffic. An 8 MiB write produces 33.55M write sectors, nearly doubling the total sector count from 33.6M to 67.1M. Runtime also increases from 292 to 441 µs. In this workload, repeated reads can reuse cached lines, but writes continue to generate cross-die traffic. The counter and runtime changes are consistent with the lower bandwidth measured for cross-die writes. DRAM traffic remains nearly unchanged. All eight read configurations transfer 1.073 GB from DRAM, with a reported spread of 0.001%. The write configurations transfer approximately 1.004 GB. The performance differences therefore do not come with a corresponding change in measured DRAM traffic. Instead, the counters point to differences in L2 and fabric traffic. This supports separating access latency from effective bandwidth in our model, rather than attributing the results to increased DRAM demand. These results also explain why the effect of memory placement depends on the accessor. A local kernel that repeatedly reads resident data can reuse its local L2 copy even when the original data is placed on the other die. A remote reader does not necessarily benefit from the same cache relationship. The next experiment tests whether placing the source data near a remote reader improves transfer performance.

5.5

Memory Placement for Remote Accessors

Prediction. When a remote GPU reads data from the local GPU, placing the source buffer on the appropriate 12

die should reduce the cost of the transfer. Compute binding should have little effect if local SMs do not initiate the relevant accesses. Method. We use nvbandwidth to measure transfers initiated by both copy engines and SMs, covering local and remote access patterns. We test the four placement states and add a one-sided placement experiment in which only the source or only the destination buffer is localized. This lets us distinguish the effect of moving the read source from that of changing the receiving buffer’s placement. Results. Table 5: nvbandwidth in four states, speedup against the same-binary stock baseline (three repetitions, median). Bold marks the best state in the row. Path

Size

Stock (GB/s)

Memory-only

Compute-only

Full-LD

Remote read, CE

64 B 1 MiB 256 MiB

7.4 75.7 744.7

1.72 2.08 1.08

1.00 1.15 1.00

1.72 2.08 1.08

Remote read, SM

1 MiB 4 MiB

114.0 314.5

1.29 1.19

1.53 1.23

1.41 1.24

Remote write, CE Remote write, SM

1 MiB 1 MiB

204.6 143.4

1.01 1.00

1.00 1.55

1.00 1.43

Local D2D, CE

64 B 1 MiB 256 MiB

45.5 663.7 3236.6

1.00 1.00 0.53

1.15 1.05 0.72

1.32 1.18 0.52

Local D2D, SM Local D2D, SM

1 MiB 256 MiB

599.2 3136.1

1.00 0.53

1.11 1.01

1.23 0.50

For a remote copy-engine read of a 1 MiB buffer, memory placement alone delivers a 2.08× speedup, while compute binding alone delivers a 1.15× speedup. The one-sided experiment shows where the larger gain comes from. Placing only the source buffer yields 157.7 GB/s, while placing only the destination yields 75.3 GB/s, close to the stock baseline of 74.46 GB/s. The source-only result supports the prediction that the main benefit comes from placing the data near the remote reader, rather than changing the receiving side. It also shows why memory placement and compute binding should be evaluated separately: the copy engine initiates the transfer, so restricting local SM execution does not directly control the relevant access. The other transfer cases show a different pattern. For D2D transfers initiated by SMs, compute binding is the effective control, while memory placement alone provides little benefit. For large local copies involving 256 MiB, configurations that pin memory achieve only about 0.53× the baseline bandwidth. This agrees with the streaming measurements: when bandwidth dominates, restricting memory to one die can cost more than localization saves. The results therefore depend on which component issues the access. Remote copy-engine reads benefit primarily from placing the source data near the reader, whereas SM-initiated transfers respond differently to compute placement. The one-sided experiment further shows that placing the destination alone does not explain the remote-read improvement.

5.6

The Cost of Localized Allocation

Motivation. Localized memory delivers benefits only when its performance gains outweigh the overhead of 13

placement setup. VMM allocation and release involve driver-side work that is not included in the GPU bandwidth measurements above. For workloads that frequently create and destroy allocations, this overhead can offset the savings during execution. Method. We measure the full create-to-free cycle across seven VMM APIs, with allocation sizes from 1 KiB to 1 GiB and multiple concurrency levels. To locate the allocation-size threshold, we also sweep the 30–66 MiB range in 2 MiB increments, taking 30 measurements per point. We then break down the 1 GiB cost by API and compare cold allocation with allocation from a warmed memory pool. Results.

Cycle cost ratio (LD / baseline)

6 × 100

locality domain 0 locality domain 1

4 × 100 3 × 100

2 × 100 32 MiB

60 MiB baseline outlier removed (did not reproduce)

32 MiB

64 MiB

Allocation size

Figure 4: Full create-to-free VMM cycle, LD divided by baseline. 30–66 MiB is scanned in 2 MiB steps with n = 30 per point; the two domains differ by less than 2% at every point.

Table 6: VMM full allocation/free cycle (P50, one GPU, one thread) and the extra cost of LD. The 60 MiB baseline point is a known non-reproducing outlier and is excluded. Size

Baseline (µs)

LD (µs)

Ratio

Extra cost

2 MiB 32 MiB 34 MiB 64 MiB 512 MiB 1 GiB

123.1 135.9 134.9 147.9 507.6 877.7

144.8 158.3 492.0 814.0 5876.5 11543.4

1.18× 1.16× 3.65× 5.50× 11.58× 13.15×

+21.7 µs +22.4 µs +357.1 µs +666.1 µs +5368.9 µs +10665.7 µs

The overhead changes sharply around 32 MiB rather than increasing at a constant rate with allocation size. Below 32 MiB, the additional cost is approximately 22 µs, giving a ratio of 1.16–1.18× relative to ordinary allocation. At 32 MiB, the additional cost jumps to about 357 µs and then grows by approximately 10 µs per MiB. Both locality domains show similar behavior, with the transition occurring between 32 and 34 MiB. We find that the driver’s allocation path is a major source of the overhead through API breakdown. With one thread and a 1 GiB allocation, cuMemCreate increases from 194.0 to 5532.7 µs (28.5×), while cuMemRelease increases from 475.4 to 5817.3 µs (12.2×). In the same setting, cuMemMap, cuMemSetAccess, and cuMemAddressReserve change relatively little. Concurrency makes the driver overhead more pronounced. With eight GPUs and eight threads, 14

cuMemSetAccess reaches 44.8 times its baseline cost. The 1 GiB allocation ratio reaches 12.10×, with a P99 ratio of 17.22×. Two further experiments help narrow down the cause. Enabling a green context changes the 1 GiB result only slightly: 11.54 ms with a green context versus 11.58 ms without one. This suggests that greencontext use is not responsible for the allocation overhead. Running both locality domains concurrently also fails to amortize the cost: the measured time is 80.71 ms, compared with a 6.64 ms ordinary baseline. 8 LD pool (die0)

alloc + free, P50 (μs)

7

ordinary pool

6 5 4 3 2 1 0 4 MiB

16 MiB

64 MiB

256 MiB

1 GiB

Allocation size

Figure 5: A warmed pool (release threshold = ∞) makes cuMemAllocFromPoolAsync+cuMemFreeAsync indistinguishable between an LD pool and an ordinary pool across 2 MiB–1 GiB.

Implications. The overhead is concentrated in allocation and release, rather than in each subsequent GPU access. Reusing memory from a warmed pool largely removes the difference: LD and ordinary pools differ by only −0.06 to −0.01 µs across sizes from 2 MiB to 1 GiB, with zero allocation-ordinal read-back errors. A pool that returns physical memory on every free, using a threshold of zero, still incurs a 14–18% overhead, similar in scale to direct VMM allocation. These results suggest that localized memory is practical when allocations are created once and reused. Repeatedly creating, freeing, or trimming localized allocations can make driver overhead a substantial part of end-to-end execution time.

5.7

Tracing and Reducing the Management Overhead

The sharp increase in allocation cost around 32 MiB led us to examine the driver’s mapping path. Three observations helped narrow down the cause. First, the threshold coincides with PMA_LOCALIZED_ MEMORY_ALLOC_STRIDE, which is 32 MiB and defines the granularity at which PMA guarantees physical contiguity for localized memory. Second, each call to the mapping routine costs about 13 µs, whether it handles a 2 MiB range or a much larger one. Third, the green-context experiment shows that compute placement has little effect on allocation time. Together, these observations suggest that the overhead comes mainly from repeatedly calling the mapping routine, rather than from writing the page-table entries themselves. We traced the problem to the static BAR1 mapping path and made two changes to reduce the cost.

15

5.7.1

Root cause

BAR1 is the GPU’s second PCIe Base Address Register. It exposes VRAM through a system address range for CPU and peer-DMA access. During initialization, the driver builds a static translation table using a mapping of the form BAR1 VA = VRAM physical address + startOffset, with a 2 MiB page size. This table needs to be updated only for allocations that are not of the default kind, exceed the default page size, or use localized memory. The relevant code is \_kbusUpdateStaticBar1VAMapping_TU102. Its early-exit condition handles ordinary allocations, but localized allocations do not meet that condition and proceed to the mapping loop. The loop advances by memdescGetContiguity(...) ? mapSize : pageSize. Because the memdesc for a localized allocation is not marked PHYSICALLY_CONTIGUOUS, the contiguity check returns false and the loop falls back to a 2 MiB step. As a result, mapping a 1 GiB allocation requires 512 iterations. Each iteration issues a separate driver call and carries its own DMA_TLB_INVALIDATE. The time breakdown explains why this is expensive. Page-table writes account for only 13–22% of each call. The remaining 77–87% is spent on a full-chip memory barrier (kbusFlush_HAL), which makes the page-table updates visible to the GPU, together with TLB invalidation. These operations account for 86.9% of the cost at 2 MiB and 77.5% at 1 GiB. Since synchronization costs roughly the same amount per call, repeating it for every 2 MiB page makes the total overhead depend heavily on the number of iterations. 5.7.2

Two complementary changes

Our fix makes two changes to the mapping path. The first change combines physically contiguous pages into larger mapping runs. Instead of assuming that adjacent logical offsets correspond to adjacent physical addresses, the driver checks each page’s physical address and extends a run only while physical contiguity holds. It then passes the whole run to the HAL in a single call. This reduces the number of calls from O(size/2 MiB) to O(size/32 MiB). For a 1 GiB allocation, the count falls from 512 to 32. The physical-address check is necessary because the optimization bypasses the existing contiguity check; merging pages without verifying their physical addresses could produce an incorrect mapping. The second change defers synchronization until the loop finishes. Each mapping call uses DMA_DEFER_TLB_INVALIDATE, followed by a single memory barrier and TLB invalidation after all runs have been processed. This avoids repeating synchronization for every run. The two changes address different parts of the cost. Coalescing reduces the number of mapping calls, while deferred synchronization reduces the number of barriers and invalidations. The second change provides no benefit when there is only one run, as below 32 MiB, because there is no repeated synchronization to eliminate. We have submitted a pull request to Nvidia’s open-gpu-kernel-modules repository ∗ † . ∗ †

https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1499 https://github.com/NVIDIA/open-gpu-kernel-modules/pull/1500

16

Table 7: Due to the static-BAR1 fix: cuMemCreate P50 (one GPU, one thread) under each patch combination, and the gain of A+B over the unpatched LD path. Patch B alone acts on sync groups and Patch A alone on call count, so the four columns separate their contributions. Below 32 MiB there is a single run and both patches are no-ops.

5.7.3

Size

LD-off

LD-on

B only

A only

A+B

A+B vs LD-on

2 MiB 32 MiB 64 MiB 128 MiB 256 MiB 512 MiB 1 GiB

8.9 13.3 9.0 9.8 10.3 119.1 145.6

20.4 26.2 353.1 684.7 1359.8 2850.2 5745.3

20.3 26.0 115.2 205.6 388.7 914.0 1863.4

20.6 25.8 56.7 95.5 178.0 485.6 1018.8

20.4 26.2 50.0 77.8 132.9 395.3 818.1

1.00× 1.00× 7.07× 8.80× 10.23× 7.21× 7.02×

Performance gains and remaining overhead

Table 7 shows the effect of these changes on cuMemCreate. The improvement is substantial for allocations above 32 MiB: 7.07× at 64 MiB, 10.23× at 256 MiB, and 7.02× at 1 GiB. The fix does not remove the entire overhead. At 1 GiB, localized allocation still takes 818.1 µs, compared with 145.6 µs for the non-localized path, leaving a 5.6× gap. This remaining cost is consistent with PMA providing physically contiguous runs of at most 32 MiB. Removing that limit would require changes to PMA, which are outside the scope of this fix. Below 32 MiB, the fix has no effect. The mapping is handled as a single run, so neither combining multiple runs nor deferring synchronization reduces the work. The roughly 2× overhead for small allocations therefore remains. We also had to ensure that the optimization preserved the original synchronization behavior. An earlier version used PTE_UPGRADE for deferred invalidation and appeared to improve allocation times at every size, including below 32 MiB. That result was misleading because this type also skips the system memory barrier, so the change did more than batch synchronization. Using PTE_DOWNGRADE, the type selected by the original routine, removes this unintended effect. The apparent improvement for small allocations disappears, as expected. All results in Table 7 use the corrected implementation. Measurement scope. Table 7 measures cuMemCreate alone on a B200 with CUDA 13.4 and driver 615.71.09. Tables 6 and 8 report the full seven-step allocation cycle on a B300. These measurements are not directly comparable and should not be subtracted from one another. The mapping mechanism and the 32 MiB threshold are consistent across the two platforms, but the reported latencies apply to their respective experimental setups. Correctness checks and limitations. We performed three checks: reading back BAR1 contents page by page and comparing them with per-page tags written through CUDA; releasing and immediately reallocating memory to test reuse; and injecting failures into the mapping path. All three passed. The BAR1 check is necessary because CUDA virtual addresses are translated through the GMMU page table and do not exercise the static BAR1 mapping. A successful CUDA readback alone therefore cannot verify the BAR1 contents. The first and third checks required driver instrumentation that is absent from the clean patch, so their results apply strictly to the instrumented build. The patch also changes error handling. In the original code, a later successful run could overwrite an earlier error, allowing the caller to receive success even when part of the requested range had not been mapped. The patch preserves and returns the first error instead. Neither version rolls back PTEs already written, so a failed operation can leave residual entries. In the original path, these entries could be overwritten incidentally during a subsequent release. The patch makes partial mapping failures explicit, 17

but does not provide rollback. Adding rollback would require changes to the release path and remains outside the scope of this fix. 5.7.4

Implications for allocation and amortization

The measurements show that much of the overhead for large localized allocations comes from repeated synchronization in the driver’s mapping path, rather than from locality domains themselves. The fix recovers a substantial part of this cost above 32 MiB, but leaves the small-allocation overhead and the residual cost associated with PMA’s contiguity granularity unchanged. The fix also changes how we should interpret the allocation-amortization results below. The overhead is still paid when memory is allocated, rather than on every GPU access, so a warmed pool can amortize it. For large allocations, the fix also makes cold allocation substantially cheaper. The repayment counts in Table 8, which use the original driver path, should therefore be treated as upper bounds when the fix is applied to allocations above 32 MiB. We do not report rescaled counts because the original and patched measurements were collected on different platforms.

5.8

Amortization: When Placement Costs Are Repaid

Question. Localized memory has an upfront allocation cost, while its performance benefit is earned during subsequent accesses. The relevant question is how many accesses or transfers are needed to recover that cost. Method. We divide the measured additional allocation cost by the measured saving per access or transfer. We use only cases in which the experiments in Section 5.5 identify memory placement as the effective control. An earlier version of this analysis used the saving from a collective, even though that improvement came from compute binding rather than memory placement. We exclude that case because it would overstate the benefit attributable to localized memory. Results. Table 8: Number of accesses needed to repay one LD allocation. Basis: a cross-die HBM dependent access saves 193.4 ns; one remote CE read of 1 MiB saves 7.18 µs (75.7→157.2 GB/s). Allocation

Extra cost (µs)

Repaid by dependent accesses

Repaid by 1 MiB CE reads

2 MiB 32 MiB 64 MiB 512 MiB 1 GiB

21.7 22.4 666.1 5 368.9 10 665.7

112 116 3 444 27 758 55 143

3 3 93 748 1 486

The results of three different cases are shown in Table 8. For a 2 MiB buffer repeatedly read by a remote engine, the measured allocation cost is recovered after three transfers. For a 1 GiB allocation, the reported cost of 10.7 ms is equivalent to 55,143 dependent cross-die accesses. Recovering this cost therefore requires sufficient reuse over the allocation’s lifetime. For a large streaming workload, placement does not repay its cost through access savings because localization itself reduces bandwidth. Here, the bandwidth penalty outweighs the potential latency benefit. These results give a practical way to decide when to use LDs. Frequently reused buffers accessed by a remote reader can benefit from localization. Large allocations need enough reuse to recover their setup

18

cost. Bandwidth-bound streaming workloads, by contrast, should generally retain an interleaved layout when it provides access to more memory bandwidth. The amortization counts above use the original, unpatched management path. With the static-BAR1 fix, the additional allocation cost for sizes above 32 MiB decreases by the factors reported in Table 7, so the corresponding repayment counts should also decrease. Counts below 32 MiB remain unchanged because the fix does not modify that path. We retain the original measured counts rather than reporting rescaled estimates, since the original and patched measurements were collected on different platforms and rescaling would imply more precision than the data supports.

6

Compute Placement

6.1

Motivation and Experimental Challenge

The third case is a local kernel that repeatedly reads data already resident in memory, such as the weights of a GEMM. Section 5.4 showed that cross-die reads can be served from the accessor’s local L2 after the first remote access. Repeated reads may therefore pay little additional cost for crossing the fabric. However, when a kernel runs on SMs across both dies, its data cannot be local to every SM at the same time. Local-L2 reuse reduces repeated fabric traffic, but it does not make a split execution equivalent to one where data and compute are placed on the same die. Our hypothesis is that memory placement and compute placement work best together. Memory placement alone leaves some executing SMs remote from the data, while compute placement alone leaves some data remote from the executing SMs. Enabling both should avoid these two problems. Testing this hypothesis is not straightforward, because the two controls also change resource availability: memory placement restricts where data resides, and compute placement restricts the available SMs. We use four placement states and a mismatch control to separate these effects.

6.2

Method

Workload. We use DeepGEMM’s masked grouped GEMM, which lets us control where the data resides and where the computation runs. A mixture-of-experts layer issues eight independent group GEMMs, which we divide evenly between the two dies. The weights stay fixed across iterations, so we can place them once and reuse them. We focus on small batch sizes, where execution time is short and placement effects are easier to observe. All variants run inside a single CUDA graph and use prepacked weight scales, so graph construction and input preparation are outside the measured interval. Placement states. We evaluate the four states listed in Table 2. The compute-only state uses the same 70-SM resource count and memory footprint as the full-placement state, but does not explicitly place the weights on the execution die. Comparing these two states shows what memory placement adds when compute is already restricted to one die. Mismatch control. The mismatch state places the weights on one die and runs the kernel on the other. The workload, kernel, and SM count remain unchanged; only the data and execution locations are deliberately misaligned. Comparing mismatch with full placement shows the cost of this misalignment. It may also change fabric traffic and contention, so we examine branch timings and concurrency before attributing the slowdown to a particular mechanism. Mechanism instrumentation. We record CUDA events on both execution streams without inserting a profiler into the measured loop. This lets us check whether the branches still overlap and measure each branch’s execution span. Profiler traces substantially perturb the overlap. We also found that a five-state counter collection came from a reference GEMM rather than DeepGEMM. We therefore do not use either

19

the profiler trace or those counters to explain the measured slowdown. Boundary conditions. We also flush 256 MB of L2 before each kernel and repeat the experiments with FP8 and FP4. These tests show whether the gains persist when the weights are not already resident in cache and when the amount of data transferred changes.

6.3

Results

Table 9: DeepGEMM under the r2 protocol (eight groups split 4+4 across dies, one CUDA graph, prepacked weight scales). Hot wall clock in µs, median of three repetitions; speedups are relative to the same-protocol stock. Precision

M

Stock

Memory-only

Compute-only

Full-LD

Full speedup

FP8

1 2 8 16

12.14 29.36 29.62 29.41

11.79 (1.03×) 39.08 (0.75×) 39.46 (0.75×) 38.91 (0.76×)

11.72 (1.04×) 28.41 (1.03×) 28.39 (1.04×) 27.90 (1.05×)

11.52 (1.05×) 21.46 (1.37×) 21.64 (1.37×) 21.41 (1.37×)

1.05× 1.37× 1.37× 1.37×

FP4

8 16

15.84 16.06

19.04 (0.83×) 18.74 (0.86×)

15.20 (1.04×) 15.23 (1.05×)

14.94 (1.06×) 14.89 (1.08×)

1.06× 1.08×

compute-only

Full-LD

14.9

18.7

15.2

16.1

14.9

15.8

19.0

21.4

20

15.2

27.9

38.9 29.4

28.4 21.6

30

memory-only

39.5

40 29.6

Hot wall clock (μs, median)

stock

10

0 FP8 M=8

FP8 M=16

FP4 M=8

FP4 M=16

Figure 6: DeepGEMM masked hot wall clock. Memory placement alone is a net loss (0.75–0.86×), compute binding alone is nearly neutral (1.03–1.05×), and only the combination reaches 1.37×.

The four states behave quite differently. With memory placement alone, performance falls by 25% at M ≥ 2: runtime increases from 29.62 to 39.46 µs. The weights reside on one die, but execution still uses all 148 SMs. SMs on the other die must therefore access remote weights, even if repeated reads can later hit in their local L2. Compute placement alone is close to the stock configuration, at 1.04×. The kernel runs on one die, but some of the weights remain remote. With both controls enabled, runtime drops from 29.62 to 21.64 µs, a 1.37× speedup. The gain largely disappears when we flush L2 before each kernel. Under cold-cache conditions, speedup ranges from 1.01 to 1.05× for FP8 and from 0.94 to 1.05× for FP4. Even in the hot-data case, FP4 achieves only 1.06–1.08×. The benefit is therefore not a general improvement from restricting execution to one die. It depends on how much data can be reused and whether the saved access latency is large enough to offset the loss of bandwidth and other fixed costs. 20

6.4

Interaction Between Placement Controls

Memory placement alone achieves 0.75× the stock performance, and compute placement alone achieves 1.04×. With both enabled, performance reaches 1.37×. The combined result is much better than either control on its own. For comparison, if the two controls had independent multiplicative effects, their combined speedup would be 0.75 × 1.04 = 0.78×. The measured speedup of 1.37× is about 1.76× this value. The two controls therefore interact strongly rather than contributing independent gains. The reason is straightforward: memory placement alone does not move all execution closer to the weights, and compute placement alone does not move all the weights closer to execution. Applying both aligns the data with the SMs that use it, while still allowing the workload to use both dies. The single-domain results show why this alignment matters. Pinning the entire GPU to one domain, without distributing work across both dies, reaches only 0.53–0.67× the stock performance. This configuration keeps compute and data local, but also gives up SMs and memory-channel capacity. Locality alone is not enough to make up for the lost parallelism.

6.5

The Cost of Misaligned Placement

The mismatch control increases runtime by 1.74×, from 21.39 to 37.22 µs. Because the kernel, workload, and SM count are unchanged, the difference cannot be explained by a change in these factors. It does not, however, identify a single physical cause: misalignment can also change fabric traffic and contention. We examine the branch timings and overlap to narrow down the explanation. Both branches take longer when they run together. branch alone

Wall clock (μs)

40

overlap 0.92

branch span, two concurrent

overlap 0.96 37.4

overlap 0.94

28.2

30

20

19.8

21.1

20.4

18.4

10

0 Aligned

Mismatched

Compute-only

Figure 7: Event-instrumented branches (median of three independent processes). Bars are the mean span of the two branches; the label above each pair is their interval overlap. Under mismatch the overlap does not fall; each branch simply becomes slower.

As shown in Figure 7, under mismatch, both branch spans increase from about 21 µs to 37.4 µs. When each branch runs alone, however, its runtime changes only from 19.5/20.0 µs to 20.2/20.6 µs, an increase of roughly 2–4%. Remote weight access has only a small effect on an isolated branch, but a much larger effect when both branches run concurrently. This points to contention when both branches access weights across dies. The independent primitive 21

measurements found that cross-die read bandwidth is 73% of same-die bandwidth (Section 5.4), which provides a plausible explanation for the slowdown. The branch timings also make reduced concurrency an unlikely primary cause. However, without valid DeepGEMM fabric counters, we cannot directly confirm that the remote-access path is saturated.

6.6

Sensitivity to Data Residency

The placement benefit changes substantially when the weights are not resident in L2. After flushing 256 MB of L2 before each kernel, speedup falls to 1.01–1.05× for FP8 and 0.94–1.05× for FP4. The gain seen with resident weights therefore does not carry over to the cold-cache case. When the working set exceeds the effective L2 capacity, more accesses reach DRAM. In this regime, restricting memory to one die also limits the bandwidth available to the kernel, making it harder for the latency savings to compensate for the bandwidth loss. The cold-cache results put a clear limit on the benefit of placement: it helps most when the kernel can reuse resident data.

6.7

Decode Attention Under Bandwidth Pressure

Workload and experimental setup. We next examine decode attention, which repeatedly scans historical K/V data. Unlike grouped GEMM with resident weights, this workload continuously moves through a large sequence and puts more pressure on memory bandwidth. We implement a chunked decode-attention kernel in CUDA C++. Each decode CTA scans part of the K/V sequence, computes a partial attention result, and writes an intermediate softmax state and output. A separate merge kernel combines these intermediate results. Our initial GQA implementation uses 64 query heads and eight KV heads, but shows less than 1% sensitivity to placement at a logical K/V scan rate of about 265 GB/s. We therefore build a higherthroughput scan workload with 64 independent output rows per request. Each row owns a separate bf16 K/V sequence with head dimension 128. For batch size B, sequence length S, and KV tile size t, the decode grid contains 64B⌈S/t⌉ CTAs. The stock configuration divides the output rows evenly between two ordinary CUDA streams, uses interleaved K/V allocation, and has access to all 148 SMs. The LD-aware configuration keeps the same row partition, kernel, and decode grid, but binds the two branches to separate 70-SM Green Contexts and places their K/V and intermediate buffers on the corresponding dies. We evaluate 30 combinations of sequence length and batch size on B300. Each CTA uses eight warps, and the runs use warm-cache, pipelined execution. We warm up each configuration for five iterations and measure 20 iterations, excluding compilation and initialization. Both implementations use the same input data, arithmetic, and correctness checks.

22

a

Matched Tile

b

Independently tuned

10 representative shapes 64

128

256

512

All 30 workload shapes

1024

2048

4096

256/1

Best / best speedup

2,048/2 4,096/1

S/B

1.107

1.10

512/1

4,096/2 8,192/1 16,384/8 32,768/4

1.063

1.05 1.00 0.95 0.90

65,536/8 131,072/8

B=1

0.85 256 Stock faster .85

LD faster .95

1.0 1.05

1.15

1.25

1K

4K

B=2 16K

B=4 65K

B=8 131K

Sequence length S Left: matched final Stage-B candidates; grey = not confirmed. Right: stock and LD choose their own best confirmed tile.

Figure 8: Decode attention under matched and independently tuned KV tiles. (a) Same-tile speedup for ten representative workload shapes from the confirmed candidate set; gray cells were not selected for final confirmation. (b) Best-vs-best speedup across all 30 workload shapes, allowing stock and LD-aware execution to select their fastest measured tile independently. Values above unity favor LD-aware execution. Table 10: Pre-registered equal-tile comparison at the eight points whose default-tile gain exceeded 1.20×; the decision rule was frozen before the run. Each form chooses the tile that minimises its own median time (in parentheses), so R is best versus best. Classes use the pre-registered boundaries: SURVIVES R > 1.05, SHRINKS 0.95 ≤ R ≤ 1.05, REVERSES R < 0.95. Hot wall clock in ms, median of three repetitions. S

B

Baseline best (tile)

Row split best (tile)

R

2048 2048 4096 4096 4096 8192 8192 16384

2 4 1 2 4 1 2 1

0.0362 (128) 0.0680 (256) 0.0366 (128) 0.0681 (256) 0.1304 (512) 0.0684 (256) 0.1304 (512) 0.1309 (512)

0.0412 (128) 0.0725 (512) 0.0422 (128) 0.0729 (512) 0.1361 (512) 0.0732 (512) 0.1365 (512) 0.1379 (1024)

0.878 0.938 0.867 0.935 0.958 0.934 0.955 0.949

Class REVERSES REVERSES REVERSES REVERSES SHRINKS REVERSES SHRINKS REVERSES

Fixed-configuration performance. We first hold the sequence length S, batch size B, and KV tile size t constant, and compare stock with LD-aware execution. Figure 8(a) shows same-tile speedups for representative workload shapes and tile sizes. With the original default tile, LD-aware execution is more than 5% faster on 15 of the 30 workloads, including eight with gains above 20%. The maximum speedup is 1.27×. The largest gains occur when the stock decode grid exposes relatively little parallel work, with approximately 0.9–3.7 decode CTAs per SM at the strongest gain points. Changing the tile size can change the result substantially. Across the matched-tile sweep, LD-aware execution generally does better with coarser KV tiles, while finer tiles often favor stock. Intermediatesized workloads are particularly sensitive to this choice. For example, at S = 4096 and B = 1, LD-aware execution reaches 1.051× with tile 64 but falls to 0.867× with tile 128. With the original tile size of 2048, the same workload reaches 1.27×. The placement configuration has not changed; changing the amount of K/V work assigned to each CTA is enough to turn a large gain into a slowdown. The four placement states help explain some of these differences. For long sequences, memory place-

23

ment and compute binding can work together. At S = 131072, B = 8, and t = 2048, memory placement alone reaches 0.793× stock performance, and compute binding alone reaches 0.936×. Together, they reach 1.063×. Short sequences are also sensitive to the intermediate buffers. At S = 256 and S = 512, localizing the partial/output allocation bundle improves the already aligned compute-and-K/V configuration by another 3.6–3.8%. The gain is negligible for the longest sequence. Since this bundle also changes allocator details, we cannot attribute the difference to physical locality alone. The results show that the useful placement choices vary with workload shape. Independent tile tuning. A same-tile comparison does not necessarily show how much benefit placement can deliver after tuning. The best tile for stock need not be the best tile for LD-aware execution. Tile size determines how much K/V data each CTA scans, how many independent CTAs the grid exposes, and how much work the merge kernel must do. Since the two configurations use different compute resources and memory paths, they may favor different tile sizes. In our measurements, their best tiles differ for 19 of the 30 workload shapes. We therefore compare the best measured configuration of each variant for every workload. We first test all legal power-of-two tiles from 64 to 4096, then select leading candidates independently for stock and LD-aware execution. We remeasure the union of these candidates in ten fresh processes per workload, balancing the two execution orders. For each variant, we choose the tile with the lowest median latency across the repeated processes and compare the two selected configurations. Best-vs-best results. Figure 8(b) shows the results after independent tuning. Only four workload shapes retain more than 5% speedup with LD-aware execution: S = 256, B = 1 (1.107×); S = 512, B = 1 (1.063×); S = 65536, B = 8 (1.050×); and S = 131072, B = 8 (1.063×). The remaining gains are concentrated in two regions: short sequences with small batches, and long sequences with large batches. For the 16 intermediate-sequence workloads (S = 1024–16384), median speedup is 0.948×, and none exceeds 1.05×. For the longest sequence, LD-aware execution fares better as batch size increases: at S = 131072, speedup rises from 1.001× at B = 1 to 1.063× at B = 8. S = 4096, B = 1 0.25

Wall clock (ms, median)

(R = 0.867)

S = 8192, B = 2

base hsplit

(R = 0.955)

0.26 0.24

0.20

0.22 0.15

0.20 0.18

0.10 0.16 0.14

0.05 128

256

512

1024

2048

Tile size

128

256

512

1024

2048

Tile size

Figure 9: Tile sweep at two of the eight pre-registered points; stars mark each form’s own best tile. The baseline’s time falls faster as the tile shrinks, so across the eight points it gains 2.07–7.11× from choosing its own tile against 1.59–4.87× for the row split. The ordering measured at the default tile therefore reverses once both forms are tile-matched.

The workloads with the largest gains under the default tile do not remain the strongest beneficiaries after tuning. None of the eight workloads that initially showed more than 1.20× speedup retains a positive best-vs-best result. Consider S = 4096, B = 1. Reducing the tile from 2048 to the best measured setting cuts stock latency from 260.0 to 36.6 µs. LD-aware latency also falls, from 205.3 to 42.2 µs, but the initial 1.27× 24

advantage becomes a ratio of 0.867×. Finer K/V partitioning helps stock much more in this case, removing the apparent benefit of placement. Understanding the remaining gains. A KV-mismatch control helps show what die alignment changes. Deliberately placing each branch’s K/V on the opposite die increases long-sequence decode latency by about 32%. The penalty is also present when branches run individually and is concentrated in the K/V-scanning stage. Keeping K/V data on the same die as the branch that scans it therefore avoids a substantial remote-access cost. That does not mean the aligned configuration must outperform stock by the same amount. Stock already uses the memory resources of both dies and retains more freedom in how computation is scheduled. For short sequences, K/V scanning takes less time, so the configuration of partial and output buffers becomes more visible in the overall result. Implications. Grouped GEMM and decode attention respond differently to placement. With resident weights, grouped GEMM reaches 1.37× speedup when both controls are enabled. For decode attention, after independent tile tuning, gains above 5% remain for only four of the 30 workload shapes, with speedups of 1.05–1.11×. Tile tuning gives decode attention another way to improve performance: changing the K/V chunk size can expose more CTA parallelism without restricting execution to individual dies. The benefit of locality must therefore be compared with what ordinary kernel tuning can already achieve, as well as with the SMs and memory bandwidth lost through partitioning. In a serving system, scheduling and kernel selection introduce further choices, so die-aware placement is one part of the overall optimization rather than a standalone fix. Three findings emerge from these experiments. First, memory placement and compute placement can reinforce each other: neither control alone delivers the full grouped-GEMM gain. Second, the useful configuration depends on workload shape. Short decode sequences are more sensitive to auxiliary-buffer placement, while long sequences benefit from aligned K/V access at sufficiently large batch sizes. Third, independent tile tuning substantially changes the results and removes the largest apparent gains under the default configuration. What matters is not just whether data and compute are placed together, but whether that placement still helps after accounting for available resources and kernel tuning.

6.8

Summary

For grouped GEMM with resident weights, memory and compute placement work best together. Neither control alone delivers the full gain, while the combined configuration reaches 1.37× speedup. The mismatch control also shows that separating data and execution placement can be costly even when the branches continue to overlap. Branch timings point to contention during concurrent remote weight access, although the lack of valid DeepGEMM fabric counters means we cannot directly confirm the mechanism. The cold-cache GEMM tests and decode-attention experiments show where this benefit ends. Placement gains depend on data residency, memory bandwidth, and the parallelism exposed by the kernel configuration. For decode attention, the 1.27× speedup under the default tile disappears after independent tile tuning for the workloads that initially showed the largest gains. Locality should therefore be evaluated together with resource availability and kernel tuning, rather than assumed to improve performance in every configuration.

25

7

Communication Placement

7.1

Attributing the Benefit of Compute Placement

The second accessor class is a local SM that issues a transfer, as in a collective whose TMA engine and participating threads run on the same GPU. Our hypothesis is that compute placement matters most for this case, while memory placement alone may do little or even hurt when it reduces available bandwidth. There is, however, an important confounding factor: every LD configuration uses 70 SMs instead of 148. A collective might run faster with fewer SMs for reasons unrelated to which die they occupy. We therefore use a separate control to distinguish SM count from placement. We first measure the gain and how it changes with message size, then compare LD against MPS configurations with a similar number of SMs but execution spanning both dies. Finally, we examine the execution timings to narrow down the possible mechanism. We also test flash-ar, a second collective workload in which the critical path involves remote writes and local polling. Comparing these two workloads lets us test whether the useful placement control depends on the access pattern, rather than on communication as a whole.

7.2

Method

General workload. We use nccl-tests with symmetric-memory registration to exercise the same collective path across 2, 4, and 8 ranks, with message sizes from 1 KB to 64 MB. We test the four placement states using both All-Reduce and All-Gather. Our main case is 2-rank All-Gather at 1 MB. In the reference implementation, this operation includes an out-of-place self-copy. The hypothesis is that compute placement makes this copy cheaper by keeping the issuing SMs and the relevant L2 resources on the same die. SM-count control. To separate placement from SM count, we compare the 70-SM compute-only state with MPS configurations that expose a similar number of SMs while allowing execution to span both dies. MPS at 50% and 25% provides the comparison points. We set the decision rule before running the experiment: a speedup of at least 1.25× under MPS would support the explanation that fewer SMs are enough to produce the gain. A result near 1.0× would instead suggest that confining execution to one die matters. Launch geometry. We also run the same binary in the stock and compute-only states under a tracing profiler and compare the collective kernel’s grid and block dimensions. If these dimensions are unchanged, the gain cannot be explained by a change in launch geometry, although this comparison does not rule out every possible difference in code generation. Second workload. Finally, we measure custimized all-reduce and all-gather kernels, named flash-ar and flahs-ag, at world sizes of 2, 4, and 8. These implementations have distinct algorithmic phases that we can selectively remove. By measuring the affected phase separately, we can determine whether placement changes the communication itself or another part of the collective.

26

7.3

Results

Table 11: NCCL 2-rank R2 All-Gather in four states (µs, median of three repetitions). Bold marks the best state per row. Size

Stock

Memory-only

Compute-only

Full-LD

Compute speedup

1 KB 64 KB 1 MB 4 MB 16 MB 64 MB

2.93 9.98 15.52 19.57 30.63 75.46

3.05 10.56 15.96 19.49 30.35 74.96

2.93 9.99 10.78 14.80 25.78 70.30

3.05 10.57 11.32 15.60 26.43 71.99

1.00× 0.99× 1.44× 1.32× 1.19× 1.07×

stock memory-only compute-only Full-LD

Latency (μs, median)

6 × 101 4 × 101 3 × 101 2 × 101

101 6 × 100 4 × 100 3 × 100 1 KiB

4 KiB 16 KiB 64 KiB 256 KiB 1 MiB 4 MiB 16 MiB 64 MiB

Message size (per rank)

Figure 10: 2-rank R2 All-Gather in four states. Memory-only coincides with stock; compute-only alone captures the gain and is slightly better than full placement.

For 2-rank symmetric-memory All-Gather at 1 MB, stock takes 15.52 µs. Memory placement alone takes 15.96 µs (0.97×), compute placement alone takes 10.78 µs (1.44×), and full placement takes 11.32 µs (1.37×). In this case, moving the data alone does not help; restricting execution to one die does. Memory placement can also make a collective slower. For 8-rank R0 All-Reduce at 16 MB, performance falls to 0.86×, with runtime increasing from 85.47 to 99.86 µs. Restricting data to one die limits the bandwidth available to the operation, and the cost outweighs any benefit from locality. The compute-only gain becomes smaller as the message grows: 1.44× at 1 MB, 1.32× at 4 MB, 1.19× at 16 MB, and 1.07× at 64 MB. We also found an average absolute saving of 4.52 µs, with a standard deviation of 0.24 µs, over 512 KB–16 MB. The absolute saving stays nearly constant while the relative speedup shrinks. This trend is consistent with Equation (2): a roughly fixed per-operation benefit becomes a smaller fraction of total runtime as the transfer time grows.

27

7.4

Separating Placement from SM Count

Table 12: SM-count control: equalising the available SM count while changing die ownership (NCCL 2-rank R2 All-Gather, median of three repetitions). Size

Stock

Compute-only one die, 70 SMs

MPS 50% both dies

MPS 25% both dies

Compute speedup

512 KB 1 MB 4 MB 16 MB

16.43 15.76 18.84 30.01

11.19 10.54 13.91 24.97

16.43 15.77 18.92 30.04

16.70 15.81 18.77 30.25

1.468× 1.495× 1.354× 1.202×

1.6 compute-only (one die, 70 SM) MPS 50% (spans both dies) MPS 25% (spans both dies)

Speedup vs. stock

1.5 1.4 1.3 1.2 1.1 1.0 0.9 512K

1M

4M

16M

Message size

Figure 11: MPS 50% and compute-only expose a similar number of SMs (about 88 versus 70), but MPS spans both dies while compute-only sits in one, and the outcomes are 0.999× versus 1.495×.

The MPS control does not reproduce the LD speedup. At 1 MB, MPS at 50% reaches 0.999×, and MPS at 25% reaches 0.997×. Compute-only execution on one die’s 70 SMs reaches 1.495×. Reducing the available SM count therefore gives essentially no improvement when execution still spans both dies. The launch-geometry comparison points in the same direction. Both states use a 512-block grid with identical block dimensions, yet median runtime falls from 20.26 µs to 13.52 µs. The gain is not explained by a change in grid size or block dimensions. With SM count and launch geometry controlled, the main remaining difference is where the 512 CTAs execute. In the compute-only state, they run on one die and use that die’s L2 slices and scheduling resources. In the stock state, execution spans both dies and their interconnect. The measurements show that confining the collective’s CTAs to one die improves performance, but they do not tell us exactly how much comes from L2 behavior, scheduling, or fabric traffic. We could not obtain usable kernel-level counters for this path (Section 8.6).

7.5

Memory Placement for Communication Control Data

The previous experiment shows that compute placement helps an SM-issued collective. Flash-ar provides a different case. Its slabs, mailboxes, and counters are managed by the backend and can be placed locally, while the user payload remains striped across the dies.

28

Table 13: flash-ar / flash-ag small-message latency gains (P50, relative to the same-binary baseline; values parsed from the recorded benchmark output). Operation All-Reduce (16 B–64 KB) All-Gather (≤128 KB)

world 2

world 4

world 8

−10 to −14% ≈ −21%

−6 to −10% ≈ −11%

−3 to −5% −2 to −4%

Mechanism ablation (world 2, all-reduce) CTX=0 (memory only) vs. CTX=1 < 1% difference at every size Spin-wait phase 1.18 → 0.78 µs (w2); 1.41 → 1.34 µs (w8) Pure transport phase send-only / recv-local change ±0.05 µs

At world size 2, placing only the slabs and mailboxes locally captures the entire gain to within 1% of the configuration that also binds compute. The phase-deletion experiment shows where the gain comes from: the spin-wait phase drops from 1.18 to 0.78 µs, while pure transport changes by no more than ±0.05 µs. The localized objects are the mailbox and counter that a remote rank writes and the local rank polls. In other words, the critical path includes a remote write followed by local polling. Moving these control objects closer to the polling side helps, even though the user payload remains striped. This matches the remote-access pattern discussed in Section 5.5. The result changes for large messages. Beyond 32 MB, localization causes a 1–6% slowdown. In the ablation, localizing memory while keeping all SMs available produces results within 0.2% of those obtained by localizing memory and binding compute; both are slower than baseline. The slowdown therefore follows memory placement, not the reduction in SM count. At these sizes, the loss of bandwidth from restricting data to one die outweighs the benefit for the control objects. Interpretation. The two collectives respond to different controls. The SM-issued All-Gather benefits mainly from compute placement, while flash-ar benefits from placing the backend objects used in remote writes and local polling. Neither result supports a blanket rule that one placement control is always best for communication. The useful control depends on which access lies on the critical path.

7.6

Asymmetry Between Locality Domains

Table 14: flash-ar All-Reduce at world 2, split by GPU pair (64 KB point, µs). die0’s advantage is concentrated in one pair; die1 is slightly slower than stock everywhere. GPU pair

Stock

die0

die1

die0 speedup

GPU 0/1 GPU 2/3 GPU 4/5 GPU 6/7

3.15 3.17 3.17 3.14

2.68 2.95 2.93 2.94

3.22 3.24 3.23 3.19

1.18× 1.07× 1.08× 1.07×

Repeating the primitive matrix on all eight GPUs gives a cross-die latency increment of 191.4–193.3 ns, a spread of only 1.9 ns. The additional latency for L2-resident accesses ranges from 2.7 to 3.3 ns, and cross-die bandwidth falls by about 27% in either direction. At this level, neither die is consistently faster. The collective results are less symmetric. On GPU pair 0/1, choosing die0 gives 1.18× speedup at 64 KB, while the other three pairs reach only 1.07–1.08×. Die1 is slightly slower than stock on every pair, at 0.95–0.98× across the measured sizes. Reversing the run order and repeating the measurements preserves both patterns, making warm-up and run-order effects less likely explanations. Since the primitive measurements are symmetric to within about 1 ns, the collective difference is 29

unlikely to come from a large intrinsic latency difference between the dies. One possibility is that the world-size-2 rank mapping gives the two dies different roles in the collective, including different uses of their L2 slices. We could not verify this explanation without the kernel-level counters unavailable in our environment, so it remains a hypothesis. The practical implication is that there is no hardware-based rule to always choose one die. Die0 is a reasonable starting point for the setup we measured, but deployments should benchmark both choices with their actual rank mapping and workload.

7.7

Summary

For SM-issued collectives, compute placement is the effective control in our measurements. The speedup reaches 1.495×, compared with 0.999× for an MPS configuration with a similar SM count that still spans both dies. The explicit resource-count control is important: a comparison between stock and LD alone would not show whether the gain came from placement or from using fewer SMs. The gain also shrinks as message size grows, while the absolute saving stays nearly constant over the earlier sweep. Memory placement helps a different part of communication. In flash-ar, the gain comes from localizing the mailbox and counter involved in remote writes and local polling, not from placing the user payload locally. For large messages, the bandwidth loss can outweigh this benefit. Together, the two workloads show that the useful placement control depends on the access on the critical path, not simply on whether the workload is a collective. Finally, the primitive measurements show no consistent preference between the two dies, even though the collective results differ by die. The difference is more likely related to the collective’s mapping than to a large physical latency gap between the dies, but we cannot confirm the mechanism with the available counters.

8

Discussion: Locality Domains and Alternative GPU-Sharing Mechanisms

8.1

Placement, Capacity, and Isolation

Sections 5–7 measured both the benefits of locality domains (LDs) and the costs of using them. We now compare LD with other ways to share a GPU. The main question is whether these alternatives can provide the same performance benefit, and what they give up in doing so. The mechanisms control different aspects of the GPU. MPS limits the resources available to a process [12], while MIG provides hardware partitioning and isolation [13]; both have been evaluated for spatial co-execution [15]. MLOPart exposes per-die partition devices [14]. Spatial GPU-sharing systems such as MuxFlow [17], ParvaGPU [5], and VUDA [16] divide capacity among concurrent jobs without controlling placement. LD instead controls where memory resides and where computation runs. These capabilities overlap, but they are not interchangeable: limiting the number of SMs does not necessarily put the remaining SMs on the desired die, and isolating a workload does not by itself align its data with its execution.

30

Table 15: Capability matrix. “SM split” asks whether SMs can be partitioned. Evidence tags: M measured, D documented, I inferred, N not measurable. Mechanism

SM split

Pick die

Memory by die

Pool shrinks

Change at run

— yes (89 GB) yes (SM halved) yes (70/148) no yes

Stock MIG

no Dyes

no Dyes

MPS

Mquota

Mno

no Iwith instance Myes

green context

Mexact

Myes

no

memory-only MLOPart

no Myes

no Myes

native Full-LD

Myes

Myes

Myes Iwith device Myes

yes (70/148)

Isolation cost

Source

— no

1.00× N

M D

Myes

M0.893×

M

no

M1.021×

M

— no

M0.740× M1.23– 2.13× M1.367×

M M

no

M

The experiments below compare these mechanisms along three dimensions: which placement choices they expose, how they affect the available resources, and what overhead they introduce. This distinction matters because the best configuration in our measurements is not always the one with the fewest active SMs or the strongest partitioning.

8.2

MPS: Controlling Capacity Without Controlling Placement

MPS is the most widely deployed alternative we evaluate. It works in our environment: the daemon starts, clients attach, and SM limits take effect. However, it does not provide the same control over placement as LD. MPS does not select a die explicitly. The SMs exposed by MPS form a contiguous range starting at SM 0, but this range spans both dies. Across settings from 25% to 100%, the observed distribution remains approximately 55:45 across the two dies. The driver’s reported SM count also differs from the number that can actually be scheduled. At the 50% setting, for example, the driver reports 74 SMs, while 88 are observed to be schedulable. MPS therefore behaves more like a resource quota than an exact partition of the SMs on a particular die. Reducing the SM count does not reproduce the LD gain. For grouped GEMM, MPS at 50% takes 32.51 µs, achieving 0.893× the stock performance. Full placement without MPS takes 21.22 µs and achieves 1.367×. Combining MPS with full placement reaches only 0.911× the stock performance. Restricting the resources also affects memory throughput. For a 256 MiB buffer, read bandwidth falls from 6947 to 6253 GB/s, while write bandwidth falls from 6414 to 4164 GB/s. MPS can limit how much of the GPU a process uses, but that limit does not guarantee that data and execution are placed together. In these experiments, reducing capacity alone does not recover the full LD benefit. MPS shares unused capacity rather than enforcing a fixed split. We also run two processes performing 40963 bf16 GEMMs, with one process assigned one quarter of the work. Stock execution takes 592 ms, compared with 573 ms under MPS at 50/50. The shorter job does not have to wait for the longer job to finish before unused resources can be used by the other client. This behavior helps utilization under skewed loads, but it is different from reserving a fixed share of the GPU for each workload. Multiplexing complementary workloads over the same GPU has similarly been used to improve inference goodput at the serving layer [2]. This distinction helps explain the collective results in Section 7.4. MPS reduces the available resource count without explicitly selecting a die, but does not reproduce the collective speedup. Fewer SMs alone 31

are therefore not enough to explain the benefit of compute placement.

8.3

MIG: Isolation Does Not Guarantee the Same Performance

MIG is the most relevant alternative when hard isolation is required. Performance isolation for co-located inference processes has been studied in edge GPU systems [8]. Vendor documentation [13] describes a Blackwell profile with two instances, each exposing 70 SMs and 89 GB of memory, matching the measured 2 × 70 + 8 SM structure. We could not run MIG directly because it is disabled in our container. Instead, we test the performance assumption behind a proposed MIG-based bound using a substitute configuration. The results below should not be read as direct MIG measurements. Our substitute is deliberately favorable to the partitioned configuration. We place all eight groups on one die to emulate a single instance, then estimate the time for two such instances to run concurrently under a zero-interference assumption. This assumption is what the experiment tests. Table 16: The MIG-style split, measured and bounded (DeepGEMM FP8 masked M=8, hot, µs, three repetitions). The 1.43× figure assumes two instances with zero interference. Configuration Stock: 148 SMs, eight groups One die (domain0 / domain1), four groups One die, eight groups (uneven load) Same process, 4+4, compute-only Same process, 4+4, memory + compute (Full-LD)

Wall clock (µs)

vs. full stock

29.00 20.22 / 20.44 46.25 28.44 21.14

1.00× 1.43× (upper bound) 0.63× 1.02× 1.37×

The estimated runtime is 20.33 µs per instance, compared with 29.00 µs for the full GPU, giving a nominal 1.43× speedup. But the corresponding measured configuration, which uses the same 4+4 split in one process with compute binding but without memory placement, takes 28.44 µs, or 1.02× the stock performance. The difference comes from concurrent execution. As shown in Section 6, an isolated branch takes 18.1–18.6 µs, whereas two concurrent branches take 28.44 µs. The measured configuration therefore does not behave as though the two branches run independently. This is consistent with interference when data and execution are not aligned, although the substitute is not a direct measurement of MIG. The second column of Table 16 shows another cost of partitioning. Running all eight groups on one die takes 46.25 µs, or 2.28× the 20.33 µs required by four groups. Reducing the available SMs without reducing the workload accordingly reverses the apparent benefit. The trade-off is different for the two mechanisms. MIG provides hardware isolation, but a workload must fit the resources assigned to its instance. LD allows memory and compute placement to be adjusted without necessarily shrinking the resource pool, although it does not provide the same isolation as a dedicated hardware partition. These measurements do not establish that MIG itself achieves 1.02×, nor that its performance matches our substitute. They show a narrower point: the nominal 1.43× estimate depends on the assumption that concurrent instances incur no interference. The measured substitute does not support that assumption.

8.4

MLOPart: Per-Die Placement and Its Management Cost

MLOPart exposes per-die partition devices [14], so it can select placement in a way that MPS cannot. Its measured cost is mainly in driver-side memory management. In prior measurements, VMM allocation and release for a single partition cost 0.98–1.28× the native-

32

memory-only

compute-only

Full-LD 2.08

2.08

Speedup vs. stock

2.0

1.44

1.5

1.0

1.37

1.37 1.04

0.97

1.15

0.75

0.5

0.0 NCCL AG, 2 ranks, 1 MB

DeepGEMM FP8, M=8

nvbandwidth remote CE read, 1 MiB

Figure 12: Three states over three accessor shapes, all relative to the same-binary stock. NCCL and DeepGEMM gain most from compute-only; the remote copy-engine read gains most from memory-only; only DeepGEMM needs both, and its 1.37× exceeds the product of the two single states.

LD baseline at the P50 median. With two concurrent partitions, the ratios increase to 1.23–2.13×. P90 medians range from 1.27 to 3.47×, with individual points reaching 4.89–6.54×. With four threads per partition, we also observed CUDA_ERROR_INVALID_DEVICE and sticky errors. These measurements come from a prior study, not a rerun in our current experimental environment, so the numerical comparison should be treated with care. They nevertheless show that providing per-die placement is only part of the problem: the cost and reliability of managing those partitions also matter. MLOPart and LD can express overlapping placement choices, but their operating costs are not the same.

8.5

Summary

Figure 12 summarizes the main finding: no single placement state is best for every accessor. The useful control depends on which part of the access path is expensive. The observations below summarize the results and findings. 1. Cross-die access has different costs for different operations. [measured] A dependent HBM access incurs an additional 193.4 ns when it crosses dies, with measurements agreeing to within about 2 ns across eight GPUs. For L2-resident reads, the measured increment is only about 3 ns. Cross-die writes still incur a substantial penalty when data is L2-resident, and cross-die streaming bandwidth is about 27% lower than the interleaved baseline in the measured configuration. These results show that the cost depends on the operation and where its data is served from. The exact numbers may differ on other platforms, but the measurements here do not support treating every cross-die access as having the same cost. 2. Locality can reduce latency while also reducing bandwidth. [measured] For the measured streaming workload, a single-die layout reaches 3628 GB/s for reads and 3219 GB/s for writes, compared with 5230 and 3795 GB/s for the striped layout. Restricting memory to one die can therefore reduce throughput even when it improves locality. Decode attention shows why this trade-off matters in a real kernel. With the default tile, LDaware execution reaches up to 1.27× speedup. After stock and LD-aware configurations are tuned 33

independently, none of the eight workloads with default-tile speedups above 1.20× retains a positive best-vs-best result. At the continuously batched decode operating point, with 14.6 blocks per SM, the default-tile ratio is already 0.984–1.003×. A locality comparison must therefore account for the kernel configuration and the available resources. Moving data closer is not enough if the resulting layout limits bandwidth or if ordinary kernel tuning removes the original performance gap. 3. The useful placement control depends on the accessor. [measured] A remote reader benefits from memory placement alone, reaching 2.08× the baseline for a 1 MiB transfer. SM-issued collectives benefit mainly from compute binding, reaching 1.495×, while memory placement is neutral or harmful in the measured cases. For resident-data GEMM, neither control alone delivers the full gain; enabling both reaches 1.37×. The results support a simple rule: place data near the side that accesses it, and place compute near the data when the executing SMs issue repeated accesses. A controlled experiment showing that the opposite control consistently wins for the same accessor, with other factors held fixed, would challenge this interpretation. 4. The collective speedup is not explained by fewer SMs alone. [measured] For the measured collective, one-die compute placement reaches 1.495×, while an MPS configuration with a comparable SM count but execution spanning both dies reaches 0.999×. Both states use the same 512-block grid and block dimensions. The comparison argues against reduced SM count or changed launch geometry as the explanation for the speedup. This conclusion would need to be reconsidered if a comparable resource count spanning both dies reproduced the same gain. 5. Memory and compute placement reinforce each other for resident-data GEMM. [measured] For the measured GEMM, memory placement alone achieves 0.75× the stock performance, compute placement alone achieves 1.04×, and full placement reaches 1.37×. If their effects were independent and multiplicative, the expected combined speedup would be 0.75 × 1.04 = 0.78×. The measured result is approximately 1.76× this baseline. This comparison shows a strong positive interaction for this workload. It does not imply that combining locality controls will have the same effect for every kernel. 6. The mismatch slowdown appears when branches run concurrently. [measured] with a mechanism caveat. Under mismatch, branch overlap remains high at 0.957, while each branch’s execution span grows from about 21 µs to 37.4 µs. When a branch runs alone, its runtime increases by only 2–4%. The slowdown is therefore consistent with contention when concurrent branches access remote weights, rather than simply a loss of concurrency. However, we did not obtain valid DeepGEMM fabric counters, so the measurements do not establish a unique physical cause. 7. LD–VMM management overhead changes sharply with allocation size. [measured] With the original allocation path, the measured VMM cycle costs about 22 µs below 32 MiB, then rises sharply around the 32–34 MiB threshold and grows by about 10 µs per MiB above it. Most of the additional cost is in object creation and release. A warmed allocation pool removes the measured difference in the tested configurations, showing that reusing allocations can amortize much of this overhead. 34

This does not mean every allocation path is free of overhead, nor that pooling eliminates all management costs. 8. The primitive measurements show no consistent preference for either die. [measured] Across eight GPUs, the measured cross-die latency increment varies by only 1.9 ns. The collective results nevertheless show a recurring difference between die choices for particular GPU pairs. The source of this difference remains unknown; the current evidence does not establish an intrinsic physical advantage for either die. 9. The MIG-style speedup estimate depends on a zero-interference assumption. [measured] for the substitute; [documented] for documented MIG capabilities. The nominal 1.43× estimate falls to 1.02× in the measured substitute configuration, where concurrent branches do not behave as independent workloads. This challenges the estimate under the measured execution conditions, but it is not a direct measurement of MIG performance. 10. Capacity, isolation, and placement are different controls. [measured] + [documented] MPS primarily controls resource quotas, while MIG provides hardware partitioning and isolation. Neither reproduces arbitrary data–execution placement in the experiments considered here. MLOPart can express per-die placement, but its measured concurrent management costs are substantial. LD exposes memory and compute placement without necessarily reducing the resource pool, but does not provide the same isolation as MIG. These mechanisms should therefore be compared by the control they provide, the resources they restrict, and the overhead they introduce, rather than treated as interchangeable ways to divide a GPU.

8.6

Limitations of the Mechanism Analysis

Two parts of the analysis have weaker evidence than the performance results themselves. These limitations restrict how precisely we can explain the mechanisms, but they do not remove the measured performance differences. We could not obtain kernel-level counters for the collective path. The profiler’s kernel-replay mode fails with a memory-save error (rc=9), while application replay times out (rc=124) against NCCL’s memory management. We did not pursue an injection-based alternative because it requires rebuilding the plugin configuration, and the container has no network access. As a result, the attribution in Section 7.4 rests on the SM-count control and launch-geometry comparison. The explanation for why the self-copy becomes cheaper under compute placement remains an inference, not a mechanism directly confirmed by kernel-level counters. One counter dataset was attributed to the wrong kernel. The five-state counter rows were later found to belong to a reference GEMM (nvjet\_sm103\_\*, grid 1312), not DeepGEMM’s kernel (grid 70). The recorded values have not changed, but they cannot be used to explain DeepGEMM’s behavior. The conclusions in Section 6 therefore rely on event-level timings and concurrency measurements, not on those counter rows. The distinction is important throughout this discussion: we directly measure the performance changes and how they depend on placement, workload, and resource configuration. For some results, we can narrow down the likely explanation, but we cannot yet identify a single hardware mechanism with certainty.

35

9

Conclusion

This paper studies how CUDA Locality Domains affect GPU workloads on Blackwell B300, focusing on which placement control reduces the cross-die cost on a workload’s critical access path. We use a two-term cost model to distinguish the latency saved by avoiding cross-die accesses from the bandwidth lost through localized memory placement, and derive predictions based on the accessor. The measurements show that these two costs affect workloads differently. A dependent HBM access incurs a cross-die latency increment of 193.4 ns, while the increment for L2-resident reads is only about 3 ns. Cross-die streaming bandwidth is approximately 27% lower than the interleaved baseline. The read/write asymmetry is consistent with remote reads being backfilled into the accessor’s local L2, while writes continue to traverse the fabric; DRAM traffic remains unchanged in the measured comparison. The effective placement control also depends on the accessor. Memory placement alone improves remote reads of a 1 MiB buffer by 2.08×. For SM-issued collectives, compute binding achieves 1.495× the stock performance, compared with 0.999× for a comparable SM count spanning both dies. For resident-data GEMM, neither control is sufficient on its own: memory placement reaches 0.75× stock performance, compute placement reaches 1.04×, and combining both reaches 1.37×. Decode attention provides a contrasting case: the apparent 1.27× gain with a fixed default tile disappears at the eight strongest measured points when the stock and LD-aware variants are tuned independently. These results support accessor-based predictions, but do not imply uniform behavior across all workloads in a class. The experiments also expose two practical complications. LD–VMM management overhead rises sharply around 32–34 MiB rather than acting as a uniform per-allocation cost. A warmed allocation pool can amortize this overhead when memory is reused, but does not remove it from every allocation path. In addition, partitioning work into one single-die instance per die does not guarantee the benefit predicted by a zero-interference model. Our substitute experiment challenges the assumption behind the nominal 1.43× MIG-style bound, but does not measure MIG directly. The main lesson is that placement should follow the accessor’s critical path, not a general expectation that locality is beneficial. Its value depends on the cross-die accesses it removes, the bandwidth it sacrifices, and the compute and memory-system resources left available to the workload.

9.1

Practical Guidance

Table 17 summarizes how to choose between memory and compute placement. The decision starts with the accessor that pays the cross-die cost, then considers whether that cost is large enough to outweigh the bandwidth and resource trade-offs. Choose placement based on the accessor, not the workload name. Workloads with similar names can respond to different controls. SM-issued collectives benefit primarily from compute binding, whereas a collective whose critical path includes a remotely written, locally polled mailbox can benefit from memory placement. Application categories alone are therefore not a reliable guide to which control will help. Do not assume that localized memory provides more bandwidth. For large streaming workloads, localizing memory can reduce throughput relative to striped placement. When accesses generate substantial DRAM traffic, the bandwidth penalty can outweigh the reduction in cross-die latency. Striped memory may then be preferable, while compute binding should be evaluated separately against the loss of available SMs and memory-system parallelism. Working-set size alone does not determine which configuration will be faster. Verify placement and benchmark the target configuration. An allocation API should not be assumed to guarantee a particular die placement. In our remote-read benchmark, uncontrolled placement

36

of cudaMalloc buffers produces a bimodal baseline of 74–101 GB/s, making results sensitive to the actual placement. Likewise, the primitive measurements show no consistent intrinsic preference between dies, yet some collective configurations retain a performance asymmetry. Deployments sensitive to die choice should verify allocation and rank placement, then benchmark the actual workload rather than assume that one die is always preferable. Table 17: Switch selection by accessor and data volume. Accessor / shape

Recommendation

Evidence

CE or peer GPU reading this GPU’s data (small to medium, ≤4 MiB) Flag or counter written remotely and polled locally Communication issued by local SMs (small messages, TMA path) Local SMs repeatedly reading resident data (GEMM weights)

Place memory, do not bind compute Place memory only

Remote CE read at 1 MiB: memory-only 2.08×, compute-only 1.15× flash-ar at world 2: CTX=0 captures the full gain (< 1% difference) NCCL 1 MB compute-only 1.44–1.50×; 8-rank AR 16 MB compute-only 1.33× vs. full 0.94× DeepGEMM 1.37×; single-domain placement 0.53–0.67×

Any large streaming access (> L2 per die) Bandwidth-bound kernel (decode attention) Allocating on a hot path

Objects localised once (≤32 MiB) Large objects (>32 MiB) on a driver with the static-BAR1 fix

Small objects (≤32 MiB), patched or not Multi-tenant or multi-process isolation Which die to choose

9.2

Bind compute; this also holds for large messages Place memory and bind compute to the same die, and split work by die Do not enable LD; bind compute only if needed Do not enable; a gain at a default tile does not survive an equal-tile comparison Build a localized pool at startup, allocate and free inside it, never trim Direct VMM allocation is fine; it costs about 22 µs Object creation is 7–10× cheaper than unpatched, but still 5.6× the non-localized path; pre-allocate, do not allocate per request Unchanged by the fix; treat exactly as before MIG on bare metal, or MLOPart; do not expect MPS to control die assignment Default to die0, but measure per deployment

Local D2D 256 MiB: memory-only 0.53×, full 0.52× Pre-registered equal-tile comparison: 0/8 survive, R = 0.867–0.958; a tile-mismatched baseline shows 1.27× Warmed pool LD vs. ordinary < 0.1 µs; cold pool still 14–18% slower A 2 MiB buffer repays itself in three CE reads 1 GiB cuMemCreate 5745.3 → 818.1 µs; residual gap is PMA’s 32 MiB run limit

Fix is a no-op below 32 MiB (single run) MPS spans both dies at every setting; MIG is documented Hardware is symmetric; GPU pair 0/1 shows a clear die0 gain, others 7%, die1 slightly slower

Limitations

1. One hardware platform. The model’s two platform-specific inputs—the cross-die latency increment of approximately 193 ns and the relative bandwidth— were measured on eight B300 GPUs with CUDA 13.4. Other Blackwell configurations may differ in die count, fabric topology, and driver behavior. Applying the model elsewhere requires measuring these quantities on the target platform. 2. No kernel-level counters for the collective path. Both profiler replay modes fail against NCCL’s memory management in our environment, and we did not attempt the injection-based alternative. The attribution of the communication gain therefore rests on the SM-count control and launch-geometry comparison. Why the self-copy becomes cheaper remains an interpretation of the placement results, not a mechanism directly verified by kernel-level counters. 3. One counter dataset was misattributed. The five-state kernel-counter dataset belongs to a reference GEMM, not the DeepGEMM kernel under study. The recorded values themselves are unchanged, but 37

they cannot support claims about DeepGEMM’s execution. Our conclusions for that kernel instead rely on event-level timing and concurrency measurements. 4. MIG was not measured directly. The container does not permit us to instantiate MIG. Whether a MIG instance’s SM set matches a locality domain, and whether LD allocation composes with a MIG instance, therefore remain unverified. The 1.43× figure is a substitute bound, not a measured MIG speedup. Our experiment challenges its zero-interference assumption but does not establish MIG’s actual performance. 5. Cold working sets show limited gains. After flushing L2, the compute-side gain for the measured GEMM falls to approximately 1–5% for FP8, while FP4 ranges from a small slowdown to a similarly modest gain. Workloads with limited data reuse should not be expected to reproduce the gains measured with hot, resident data. 6. The attention study covers a limited design space. The attention experiments use one GPU and software stack, three repetitions per point, synthetic KV data rather than an end-to-end serving system, and tile sizes restricted to powers of two in [128, 2048]. The independent-tuning comparison covers the eight points with the largest default-tile gains, not every workload and kernel configuration. SM clocks were measured at 2032 MHz at both session endpoints, with within-point drift at or below 1.14%. The results characterize the tested configurations; they do not establish that decode attention cannot benefit from LD placement. 7. The die asymmetry remains unexplained. Primitive measurements show no substantial intrinsic difference between the dies, but some collective configurations exhibit a persistent performance asymmetry. We did not instrument the rank-to-die mapping and kernel-level behavior needed to identify its cause. A mapping-related explanation remains a hypothesis. 8. Some comparisons use earlier measurements. The MLOPart and one-sided-copy results were collected in earlier studies rather than rerun under the current experimental setup. They provide qualitative comparisons, but differences in platform and measurement conditions limit the precision of direct numerical comparisons. Further work should prioritize kernel-level measurements of the collective path. These could distinguish among L2-slice contention, scheduling effects, fabric traffic, and other possible explanations for the self-copy result. They could also establish whether the residual die1 performance deficit comes from rank-to-die mapping or another source. More broadly, evaluating LD on additional platforms and resident-data workloads would test how widely the measured latency–bandwidth trade-off and the interaction between memory and compute placement generalize.

References [1] Joshua Bakita and James H Anderson. Hardware compute partitioning on nvidia gpus. In 2023 IEEE 29th Real-Time and Embedded Technology and Applications Symposium (RTAS), pages 54– 66. IEEE, 2023. Best Paper Award; Library: http://rtsrv.cs.unc.edu/cgit/cgit.cgi/ libsmctrl.git. [2] Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. Towards high-goodput llm serving with prefill-decode multiplexing. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 2030–2047, 2026.

38

[3] Xiaoze Fan, Jianhao Wang, Weihao Cui, Han Zhao, Zhuobin Huang, Yangjie Zhou, Yuxian Qiu, Shixuan Sun, Bingsheng He, Quan Chen, et al. Dissecting how die scaling breaks gpu fine-grained scheduling. arXiv preprint arXiv:2609.24270, 2026. [4] Jingkai He, Guangda Sun, Tianjian Li, Dong Du, Yubin Xia, and Haibo Chen. Taming dynamism on gpus: Cross-sm kernel fusion via sm cooperation and just-in-time reduction. In Proceedings of the ACM SIGOPS 32nd Symposium on Operating Systems Principles, pages 1070–1090, 2026. [5] Munkyu Lee, Sihoon Seong, Minki Kang, Jihyuk Lee, Gap-Joo Na, In-Geol Chun, Dimitrios Nikolopoulos, and Cheol-Ho Hong. Parvagpu: Efficient spatial gpu sharing for large-scale dnn inference in cloud environments. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14. IEEE, 2024. [6] Changxi Liu, Yun Chen, and Trevor E Carlson. Dgna: Dissecting gpu numa architecture through microbenchmarking and data analysis. ACM Transactions on Architecture and Code Optimization, 2026. [7] Xinhao Luo, Zihan Liu, Yangjie Zhou, Shihan Fang, Ziyu Huang, Yu Feng, Chen Zhang, Shixuan Sun, Zhenzhe Zheng, Jingwen Leng, et al. Clusterfusion: Expanding operator fusion scope for llm inference via cluster-level collective primitive. Advances in Neural Information Processing Systems, 38:38790–38817, 2026. [8] Juan José Martı́n, José Flich, and Carles Hernández. Performance isolation for inference processes in edge gpu systems. arXiv preprint arXiv:2601.07600, 2026. [9] NVIDIA. Locality domain. CUDA Programming Guide, Special Topics, 2025. Official technical documentation. [10] NVIDIA. Thread block cluster. CUDA Programming Guide, Appendix C: C++ Language Extensions, 2025. Official technical documentation. [11] NVIDIA. Green contexts. CUDA Programming Guide, Special Topics, 2026. Official technical documentation; Driver API reference also available. [12] NVIDIA. Multi-process service. NVIDIA CUDA Toolkit Documentation, 2026. Official technical documentation. [13] NVIDIA. NVIDIA multi-instance GPU user guide, 2026. Official technical documentation, Release r580. [14] NVIDIA Developer Blog. Boost GPU memory performance with no code changes using NVIDIA CUDA MPS, December 2025. Technical blog introducing MLOPart within MPS; MLOPart later renamed to Locality Domains in MPS v3. [15] Jorge Villarrubia, Luis Costero, Francisco D Igual, and Katzalin Olcoz. A comprehensive evaluation of spatial co-execution on gpus using mps and mig technologies: J. villarrubia et al. The Journal of Supercomputing, 82(4):175, 2026. [16] Bin Xu, Pengfei Hu, Wenxin Zheng, Jinyu Gu, and Haibo Chen. Vuda: Breaking cuda-vulkan isolation for spatial sharing of compute and graphics on the same gpu. arXiv preprint arXiv:2605.01352, 2026. [17] Yihao Zhao, Xin Liu, Shufan Liu, Xiang Li, Yibo Zhu, Gang Huang, Xuanzhe Liu, and Xin Jin. Muxflow: Efficient and safe gpu sharing in large-scale production deep learning clusters. arXiv preprint arXiv:2303.13803, 2023.

39

Related documents

• GPU-Initiated Discrete Simulated Bifurcation: Low-Latency Requests and Streaming Dense Couplings#1215718
• GPU-Initiated Discrete Simulated Bifurcation: Low-Latency Requests and Streaming Dense Couplings#1194383
• Empirical Analysis of GPU Frequency Behavior Under ML Workloads#353030
• An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU#1457231
• Latency-Aware Orchestration for Multi-Agent LLM Workflows on Heterogeneous GPUs#926263