CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems Tingkai Liu∗† , Muralidhar Andoorveedu† , Sanjoy Das† , Sanjay Patel† , Volodymyr Kindratenko∗ ∗ University of Illinois Urbana-Champaign † NVIDIA Corporation

arXiv:2607.07862v1 [cs.DC] 8 Jul 2026

{tingkai2, kindrtnk}@illinois.edu, {tingkail, mandoorveedu, sanjoyd, sanjpatel}@nvidia.com

Abstract—The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated sharedmemory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTApipelining reduces latency by up to 31.8% compared to microbatching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary. Index Terms—Graphics Processing Unit, Parallel programming, Distributed Computing, Machine Learning.

I. I NTRODUCTION Since the development of the Transformer architecture [1], [2], optimizing Large Language Model (LLM) inference for production has become a critical challenge [3]. Modern serving frameworks have two main objectives: maintaining high aggregate throughput for cost efficiency, and meeting latency Service-Level Objectives [4]. In highly interactive scenarios, the limiting factor becomes single-batch user input latency. Since GPUs have traditionally been designed as throughputoriented devices, minimizing latency for a single-batch request introduces new system-level requirements. Correspondingly, to support the rapid scaling of LLM workloads, modern multi-GPU hardware systems have evolved into tightly coupled architectures, calling for novel software paradigm to fully exploit them. Systems such as the NVIDIA GB200 NVL72 [5] utilize NVLink and NVSwitch [6] interconnects not only to provide high peer-to-peer bandwidth, but also to enable the multi-GPU cluster to function with a unified shared-memory space. Although recent advancements have introduced sophisticated serving frameworks to maximize multi-GPU deployment efficiency [7]–[11], as well as efforts to build systematic abstractions for programming multi-GPU workloads [12], [13], there remains untapped potential for novel execution paradigms that natively exploit these tightly coupled clusters as holistic shared-memory systems.

At the current stage, the standard paradigm for largescale LLM deployment on multi-GPU systems relies on hybrid parallelism strategies [14]–[17], primarily combining Pipeline Parallelism (PP) [18], [19] and Tensor Parallelism (TP) [20]. While emerging techniques such as Expert Parallelism (EP) [21] and disaggregated serving [22] offer further optimizations, they are highly workload-specific. Therefore, PP and TP remain the universal baselines. PP mainly focuses on improving overall serving throughput by operating at the inter-layer level to distribute transformer blocks across devices. TP provides both throughput improvement and latency reduction by spatially sharding computations at the operator level. However, TP introduces additional collective communication (e.g., AllReduce) to resolve data dependencies [15], establishing a hard ceiling on latency optimization. Between inter-layer PP and operator-level TP lies an opportunity for further latency reduction: the intra-layer, interoperator execution space. Existing efforts accelerate this space primarily through temporal optimizations to improve efficiency on single device, such as kernel fusion [23] or mega-kernels [24]–[26]. However, these require complex compiler toolchains or rigid rewrites [26]. Alternatively, localized micro-batching [22], [27] offers finer-grained pipelining, but introduces pipeline bubbles and degrades kernel efficiency at small chunk sizes. These limitations call for more efficient inter-operator spatial scaling techniques. To address the dual requirements of single-batch latency optimization for GPU workloads and the need for new software paradigms for shared-memory multi-GPU systems, we propose CTA-pipelining, a novel latency-oriented spatial scaling method. Operating at the inter-operator level, it leverages the unified NVLink memory domain to enable simultaneous execution of data-dependent kernels across GPUs, via dynamic spatial pipelining at the Cooperative Thread Array (CTA) granularity. The designed protocol is minimally invasive to existing GPU kernel implementation, by relying only on the addition of prologue and epilogue code snippets. This preserves the potential for automated integration across a wide variety of workloads. As an initial demonstration, we implement prototype and perform analysis using multi-layer general matrix-matrix multiplication (GEMM), a critical GPU workload, which also represents the multilayer perceptron (MLP) layers in transformerbased LLMs. Our implementation and evaluation are built upon the state-of-the-art NVIDIA libraries including CUT-

LASS [28], cuBLAS [29], and NCCL [30], on cutting-edge hardware including 8-GPU H200 NVLink system and 8-GPU B200 NVLink system. We evaluate CTA-pipelining via two angles: validating the fundamental mechanism and demonstrating its broader scaling capabilities. First, to validate the protocol’s overhead, we analyze the integration with both classical and warpspecialized multi-stage persistent CUTLASS kernels. Results show the overhead is minimal, and can even be mostly hidden within warp-specialized executions. Second, to establish CTA-pipelining as a generalized spatial scaling paradigm, we evaluate CTA-pipelining against traditional micro-batch chunk pipelining (achieving up to 31.8% latency reduction) and Tensor Parallelism (achieving up to 29.6% latency reduction), on setups representing MLP operations. We further demonstrate that it can be combined with TP as an orthogonal spatial scaling dimension, further pushing the latency scaling limit by providing benefit on both computation and communication. The contributions of this paper are summarized as follows: • We propose CTA-pipelining, a novel spatial scaling paradigm for multi-GPU shared-memory systems, aimed at optimizing single-batch latency for multi-GPU workloads such as LLM inference. • We demonstrate a prototype implementation of CTApipelining using the state-of-the-art GEMM library, NVIDIA CUTLASS, on various styles of GPU kernels. • We perform analysis on basic protocol overhead, and identify the benefit of integrating with warp-specialized multi-stage persistent kernels. • We compare CTA-pipelining against traditional microbatch chunk pipelining, yielding latency reductions of up to 31.8% on multi-layer GEMMs. • We compare CTA-pipelining against Tensor Parallelism on multilayer perceptron setup, providing latency reductions of up to 29.6%. • We show how CTA-pipelining serves as an orthogonal spatial scaling method to Tensor Parallelism, offering benefit on both computation and communication, further pushing the latency optimization frontier. • We hint potential hardware evolutions that could further benefit this execution model. II. BACKGROUND AND R ELATED W ORK A. GPU Execution model Basic understanding of the GPU execution model is essential to understand this work. CUDA [31] uses Single Instruction, Multiple Data (SIMD) execution model [32]. Each GPU kernel launches a global grid of threads to process data concurrently. These threads are grouped into blocks, also known as Cooperative Thread Arrays (CTAs). Each CTA utilizes a localized shared memory to cooperatively compute a specific tile of data. Recent architectures further introduce a abstraction layer called Thread Block Clusters (or Cooperative Group Arrays, CGAs), which groups multiple CTAs together, enabling cooperate via Distributed Shared Memory

(DSMEM) [33]. At the hardware level, CTAs are dispatched to Streaming Multiprocessors (SMs), where threads are bundled into groups of 32, known as warps. While individual warps are scheduled and operate independently, threads within the same warp execute the same instruction simultaneously. As GPU architectures evolve, core computational workloads are increasingly offloaded to specialized on-chip accelerators. General-purpose SMs are transitioning from performing the primary computation to orchestrating these specialized units. Mature accelerators include Tensor Core for MatrixMatrix Multiply-Accumulate (MMA), and Tensor Memory Accelerator (TMA). The recently announced integration with Language Processing Unit (LPU) for LLMs also reflects this trend [34]. To efficiently exploit this heterogeneous on-chip setup, warp specialization has emerged as a standard programming paradigm. Specific warps are dedicated to each stages of an execution pipeline, managing asynchronous calls to the hardware accelerators. High-performance GEMM implementations, such as those in the NVIDIA CUTLASS library [28], rely on this paradigm to optimize the overall computation. B. Inter-operator Optimizations Large amount of existing work focuses on improving GPU execution efficiency through the co-optimization of consecutive kernels (inter-operator optimization). The boundaries between discrete kernels often introduce performance bottlenecks, including kernel launch overheads [35], redundant global memory round-trips [36], and wave quantization effects, where the final wave of scheduled thread blocks fails to fully utilize the available resource. Furthermore, individual kernels have different resource utilization profiles, typically categorized as compute-bound, memory-bound, or communicationbound. By fusing or concurrently executing these operators, it is possible to overlap mismatched resource demands, achieving higher overall hardware utilization. To directly address inter-operator inefficiencies, kernel fusion approaches this by systematically merging multiple consecutive operations into a single kernel [37], [38]. Pushing this optimization paradigm to its extreme results in the mega-kernel approach. In a mega-kernel design, massive computational sub-graphs, ranging from complex attention mechanisms [23] to entire multi-GPU model inference pipelines [25], [26], are encapsulated within a single GPU kernel launch. However, this extreme form of fusion requires re-compilation of the whole workflow, as well as sophisticated in-kernel runtimes to manage intra-kernel synchronization, warp-level task scheduling, and decentralized hardware resource allocation to prevent SM underutilization [39]. Besides, inter-operator pipelining offers a complementary approach by overlapping consecutive execution stages. Unlike coarse-grained inter-layer parallelism, these techniques decompose individual operators into fine-grained micro-batches or tiles to enable concurrent processing [40]–[42]. This intralayer pipelining also serves as a primary mechanism for achieving communication-computation overlap [43]–[45].

However, these methods have primarily been deployed as temporal pipelining on single devices to improve overall hardware utilization. The performance benefits of some of those approaches also rely on processing multiple batches of data concurrently, for example, by overlapping the communication of a previous batch with the computation of the current one. The use of fine-grained inter-operator pipelining spatially across multiple devices for reducing single-batch input latency remains largely unexplored. Historically, this is mainly because pipelining introduces pipeline bubbles and requires frequent inter-stage synchronizations, limiting its effectiveness on latency reduction [15]. Very recently, hardware-centric studies such as Kitsune [46] have begun exploring the hardware architecture potential to support spatial pipelines. Our work comes from a different angle: we propose a pure software technique designed to exploit the capabilities of modern shared-memory multi-GPU systems, focusing on using fine-grain spatial pipelines to reducing single-batch query latency. III. CTA-P IPELINING P ROTOCOL : D ESIGN , I NTEGRATION WITH CUTLASS, AND OVERHEAD A NALYSIS In this section, we present the design and implementation of the CTA-pipelining protocol. We demonstrate its integration into the state-of-the-art NVIDIA CUTLASS library for GEMM operations and provide a detail execution trace analysis to understand and quantify the protocol’s overhead. As a software approach built upon current hardware and CUDA programming capabilities, our inter-kernel communication relies on atomic counters and queues, sharing structural similarities with latest literatures [41], [46]. However, we distinguish our work by elevating this mechanism into a crossdevice spatial scaling method, and provides the placement strategy and additional coherency-ensuring design for multiGPU setup. Moreover, we present novel implementation of integrating the protocol into multi-stage warp-specialized persistent kernels, and provide novel finding that such integration can effectively hide protocol overhead. A. Basic Protocol As its name suggests, CTA-pipelining is designed to execute dependent GPU kernels in a deeply pipelined manner at the finest architectural granularity: the CTA. In this model, each CTA consumes one or more input data tiles and produces a single output tile. Rather than enforcing strict kernel-level synchronization, our approach allows the consumer kernel to launch its CTAs as soon as the producer kernel generates the requisite data tiles. The kernel execution trace is similar to Figure 3b, where data dependent kernels are spatially distributed across GPU compute resources, getting launched simultaneously, and complete almost at the same time, except the tail at the consumer kernel for the last wave of pipelined CTA computation. In single-GPU context, this goal is conceptually similar to the Megakernel [25], [26] approach, which recompiles the entire workload to execute a tile-level dependency graph on a

Fig. 1: Setups of enabling CTA-pipelining across 2 GPUs. single device. However, CTA-pipelining aims to enable finegrained pipelined execution spatially across multiple GPUs while preserving the original kernel structure, without the need to recompile the overall workflow. To enable data-dependent kernels to launch concurrently across multiple GPUs while maintaining execution correctness, additional control-flow dependency organization is required. Figure 1 illustrates the overall execution process of CTApipelining, using a two-GPU setup to demonstrate how the original kernel interacts with the additional components. The underlying data structures that support this paradigm include dependency arrays, scoreboards, and inter-device workqueues, which we collectively refer to as dependency structure. To interface with the dependency structure and orchestrate the control flow, lightweight prologues and epilogues are injected directly into the original kernel code. We detail each of these core components below, followed by a step-by-step execution walkthrough. 1) Dependency Array: The dependency array indicates the control-flow dependencies between producer CTAs and consumer CTAs. Specifically, it is indexed by the producer CTA ID to identify which consumer CTAs if affects, including a contiguous list of consumer CTA IDs and a corresponding offset array that defines the index range for each producer. These dependencies can be derived directly from data dependencies, using static analysis, trial runs of the kernels, or even calculated dynamically at runtime if closed-form formula exist. As the dependency array is exclusively accessed by consumer CTAs, it is stored on the producer device memory. 2) Scoreboard: To resolve many-to-one data dependencies, where a consumer CTA relies on output from multiple producer CTAs, a scoreboard is used to track producer completion. The scoreboard consists of an array of atomic counters, with each consumer CTA assigned an entry, initialized to the total number of its prerequisite producer CTAs. A counter reaching zero serves as a readiness signal, indicating a consumer CTA is ready for execution. The scoreboard is initialized as part of the prior dependency analysis. Since the scoreboard is also exclusively modified by producer CTAs, it is stored on the producer device memory. 3) Inter-device Workqueue: The inter-device workqueue serves as a readiness signaling mechanism between the producer kernel and the consumer kernel. Implemented as a cyclic

ring buffer, it is managed by several atomic values including head, tail, and size to ensure coherency. Since the workqueue requires concurrent access from both kernels, cross-device NVLink is utilized. We place the workqueue in the consumer device’s memory to minimize critical-path latency. While cross-device writes are theoretically expensive, CUDA’s asynchronous write semantics allow the producer to issue ”fire-and-forget” memory operations without stalling for completion (unless there is explicit flush barrier). Conversely, consumer CTAs must continuously poll the workqueue, and forcing these frequent read operations across NVLink would incur latency penalties. Therefore, placing the workqueue on the consumer device is a more reasonable design choice. 4) Overall Workflow with Prologue and Epilogue Code Snippets: The data structures described above facilitate the CTA-pipelining control flow. The logic that utilizes these data structures consists of minimal prologue and epilogue code snippets. These are added to the beginning and end of the kernel code, leaving the core original kernel implementation untouched. (Warp-specialized kernels are treated slightly differently, as introduced later.) The overall control flow orchestrated by the injected prologue and epilogue is illustrated by the numbered arrows in Figure 1. First, the producer CTA writes its output directly to the consumer’s input memory across NVLink, as indicated by arrow (1). Its epilogue then issues a system-wide memory fence, ensuring subsequent dependency operations and workqueue updates do not become visible to other devices before the actual output data is visible. After that, it utilizes SIMD execution to atomically decrement dependency counters in scoreboard after querying dependency array (2, 3). When a counter reaches zero, the producer thread pushes the ready consumer CTA ID into workqueue (4). On the consumer side, the injected prologue uses a single thread to busy-poll this workqueue while the remaining threads wait at a barrier (5). Upon retrieving a ready ID, the polling thread broadcasts it via shared memory. The consumer CTA then remaps its ID to this fetched value and executes its standard, unmodified kernel payload. 5) Host-Side Organization: From the host side, each kernel is assigned a dedicated CUDA stream and bound to a specific device or a partition of devices. During execution, all kernels are launched simultaneously, with internal execution order dynamically guided by the CTA-pipelining protocol. In multilayer workloads, intermediate kernels act as both producers and consumers. These kernels feature both a prologue to fetch from an source workqueue and an epilogue to push to a destination workqueue. Note, this entire multi-kernel execution process can be captured by CUDA Graphs, reducing kernel launching bubbles. In actual workflows, the CUDA driver’s default CTA scheduling order might not allow the entire workload to execute in a smoothly pipelined manner. In such cases, altering the execution order, such as shifting from a column-major to a row-major pattern, can be beneficial. In this case, the

first kernel in the workflow can simply consume a pre-defined source workqueue to explicitly guide its CTA execution order. There is an optional micro-optimization to eliminate the SM resource waste of initial busy-polling, using cuStreamWaitValue32 API. This feature blocks the consumer’s execution stream until the workqueue contains at least one item, preventing immediate spin-waiting. However, this stream-level synchronization does incur a slight latency penalty for the kernel launch. B. Integration with CUTLASS GEMM Kernels In this subsection, we explain how the CTA-pipelining protocol can be integrated into two different styles of GPU kernels, using NVIDIA CUTLASS GEMM implementations [28] as examples. To illustrate the dependency mapping, we use a two-layer GEMM workload as our running example, where the output of the first GEMM is directly consumed by the subsequent GEMM. 1) Classical Kernels: As a straightforward example, we demonstrate how CTA-pipelining integrates with classical GEMM kernels, using the SM90 TMA kernel from the CUTLASS library. In this kernel, each CTA has all threads follow the same execution flow, producing one tile of the output matrix. At the end of its execution, the CTA terminates, and the next wave of CTAs is scheduled by the CUDA driver. This represents the most classical GPU kernel structure. In terms of control dependencies, each consumer CTA computes one output tile by consuming an entire row of input tiles, making it dependent on an entire row of producer CTAs. Furthermore, consumer CTAs in the same row share identical dependencies. While we recognize that this could be simplified to a row-to-row dependency rather than a strict CTA-to-CTA dependency, or calculated dynamically at runtime using matrix and tile dimensions, we present the most general case for illustrative purposes. Regarding modifications to the CUTLASS library, integrating CTA-pipelining protocol is minimally invasive. Beyond extending the parameter structure to accept the necessary dependency structures, the source code changes are restricted to injecting the prologue and epilogue snippets. Notably, while the prologue requires shared memory to broadcast the dynamically fetched CTA ID to all threads, it reuses the memory space already allocated for the kernel’s main execution. This does not affect the subsequent execution, as the prologue runs at the very beginning of the kernel and its shared memory usage is strictly one-time. In summary, integrating CTA-pipelining with classical kernels is highly straightforward. It does not require a detailed understanding or modification of the core execution logic, demonstrating that this approach is highly generalizable to other kernels. 2) Warp-Specialized Multi-Stage Persistent Kernels: The warp-specialized multi-stage persistent kernel is a representative example of programming models evolution and serves as the foundational design for modern high-performance libraries

like CUTLASS. As its structural paradigm differs from classical kernels, it is helpful to demonstrate how CTA-pipelining integrates with this kernel design, using SM100 TMA Warp Specialized kernel from CUTLASS as an example. This architecture relies on three important concepts. First, persistent kernels launch a fixed number of CTAs that remain active for the entire computation, dynamically fetching worktiles via explicit metadata rather than relying on standard CTA IDs. Second, a multi-stage design divides the workflow into an internal micro-pipeline. Third, warp specialization assigns these distinct micro-pipeline stages to independent warps of threads. In the SM100 CUTLASS implementation, these three concepts operate together. The resulting micro-pipeline includes discrete stages for scheduling, main data loading, Matrix Multiply-Accumulate (MMA), GEMM epilogue data loading, and GEMM epilogue operations. One dedicated warp handles each stage, except the GEMM epilogue stage might have multiple warps. The CTA-pipelining prologue is integrated directly into the scheduler warp. In the original SM100 CUTLASS kernel, information regarding the next work tile is obtained via a hardware query, known as a Cluster Launch Control (CLC) query, issued by a single thread within the scheduler warp to fetch the next ready work ID. Subsequently, the returned information is multicast across the CGA and stored in each CTA’s Distributed Shared Memory (DSMEM). After this broadcast, each active warp decodes the response to extract the index of its next work tile. Aligning with this native kernel structure, the CTApipelining prologue reuse this work-tile fetching and broadcasting pathway. Instead of issuing a hardware CLC query, the scheduling thread performs busy-polling on the added workqueue. Once it successfully fetches a work item from the queue, it issues remote memory stores to the DSMEM across the CGA, reusing the buffer originally allocated for the CLC response. Since the scheduler tracks the total number of tiles to be executed, it uses this information to determine the completion of the workload by comparing the current workqueue index against the total tile count. The other compute warps fetch the next work tile similar to before, but changing the decoding format for the new metadata structure. The CTA-pipelining epilogue is positioned at the final stage of the micro-pipeline: the GEMM epilogue. In the SM100 CUTLASS GEMM implementation, multiple warps are used to execute arithmetic operations, while only one warp performs the final output write to global memory. As the global threadfence that ensures consistent memory views is only effective for each calling thread, only the specific warp responsible for writing global output issues the threadfence, and only this warp is utilized for the CTA-pipelining epilogue operations. The specific operations performed on the dependency structure remain identical to before. A special case arises with Blackwell’s specific Tensor Core operations, where two CTAs cooperatively issue a single Tensor Core instruction. In the CUTLASS GEMM implementa-

tion, this is reflected as a change in the tile ID mapping: while each CTA still receives separate work-tile information, the total tile size is doubled along one dimension to accommodate the CTA-pair operation. To handle this special case, the worktile information fetched from the workqueue are decoded differently, and the scoreboard tracking values are adjusted accordingly. All other parts of the pipelining logic remain unchanged. C. Protocol Overhead on CUTLASS The first question that naturally arises is how the additional operations introduced by the CTA-pipelining protocol affect overall performance. To understand this impact, we investigate detailed execution phases and profile the latency of each operation by injecting timestamping code into the kernel, and perform analysis based on the collected timestamps. We present results for both classical kernels and warp-specialized multi-stage persistent kernels, as they exhibit different performance patterns. The evaluation of the classical SM90 CUTLASS kernel is performed on a 8-GPU H200 system, with GPUs connected by the 4th generation NVLink. The SM100 TMA Warp Specialized kernel is evaluated on a 8-GPU B200 system, where GPUs are connected by the 5th generation NVLink. For all evaluation on protocol overhead, all GEMM input sizes are 16384 by 8192, multiplying with weight matrix of 8192 by 8192. The input and output data types are set to BF16, and the Tensor Core accumulator type is set to FP32. The specific CUTLASS configuration is selected by running the CUTLASS Profiler [28] on the given input size, and selecting the configuration with the fastest execution time. For the classical kernel, the profile of the execution phases is shown in Figure 2a. For each CTA, the epilogue operations take about T1=6µs. Subsequently, it takes T2=120µs for this

(a)

(b)

Fig. 2: The detail execution trace of adding CTA-pipelining to (a) classical kernel, and (b) warp-specialized multi-stage persistent kernel. For illustration purpose, time is not at the correct scale.

update to become visible to the consumer device. On the consumer side, the prologue takes about T3=1.5µs to fetch the work queue value and broadcast the fetched information. Notice, this overhead accumulates for each wave of CTA execution. The warp-specialized persistent kernel exhibits dramatically different results. CTA-pipelining can utilize the idle time of specific warps, effectively hiding its protocol overhead. As introduced earlier, the warp-specialized kernel inherently functions as a micro-pipeline. Since the entire pipeline is typically dominated by the latency of the compute-heavy MMA operations, the other warps frequently need to wait at pipeline barriers. This presents an opportunity to perform additional work within these waiting warps, such as the CTApipelining operations, without affecting overall performance. The exact execution trace for the warp-specialized persistent kernel is illustrated in Figure 2b, including only the related warps for analysis. For each wave of execution, the epilogue operations, including the threadfence, take approximately T1=6µs. During this time, other warps remain active for the next wave of computation. By the time the next computational wave reaches the GEMM epilogue warp, the CTA-pipelining epilogue operations from the previous wave have already finished, allowing the warp to take on the next wave immediately, which hides the CTA-pipelining operation overhead. The cross-device NVLink data write takes roughly T2=5µs to become visible on the consumer device. On the consumer side, the prologue work queue fetching takes about T3=1.5µs, and it takes about T4=0.5µs to broadcast across the CGA by the scheduler warp. Similarly, since the scheduler warp is an stage of the micro-pipeline, this latency is also hidden. As a result, assuming no other memory or communication contention, the CTA-pipelining overhead is theoretically only visible once during the initial pipeline ramp-up. As the CTA-pipelining logic is confined to the injected prologue and epilogue, it also imposes minimal performance interference on the original kernel. It does not disrupt the main compute phase’s active register allocation or shared memory usage, avoiding register spilling or shared memory contention. For instance, in our evaluation of SM100 TMA Warp Specialized kernel, the baseline execution of two consecutive GEMMs takes an average of 1080µs each. When executed using CTA-pipelining, the producer completes in 1090µs, and the consumer finishes in 1165µs, counting in both the visible overhead and the latency of the final pipeline wave. In summary, the overhead introduced by the CTA-pipelining protocol is minimal, and it can even be largely hidden within warp-specialized persistent kernels, making it suitable for general fine-grain spatial pipelining. We demonstrate more scaling examples with performance analysis in the next section. IV. CTA-P IPELINING AS A S CALING M ETHOD : C OMPARING AGAINST M ICRO - BATCHING AND TP In this section, we demonstrate how CTA-pipelining can be used as a latency-oriented spatial scaling method, on optimizing single batch execution latency. The evaluation

consists of two parts. First, we perform a comparison against traditional static micro-batch chunk pipelining (also known as micro-batching), using a multi-layer GEMM as the primary workload. Second, we compare CTA-pipelining against Tensor Parallelism (TP), and demonstrate how it can be combined with TP by evaluating performance across different scaling setups and input sizes. The workload chosen for this section consists of multilayer GEMM operations, which simulate LLM MLP layers by omitting intermediate element-wise non-linear activations for simplicity. This abstraction assigns specific meanings to our matrix dimensions. For example, in the two-layer GEMM Y = XAB, X represents the input activation tensor, while A and B are the respective weight matrices. Because of this mapping, we fix the weight matrices A and B at 8192 by 8192 to reflect typical LLM architectures, while leaving the row dimension of X flexible for varying input sequence lengths. As before, the input and output data types are set to BF16, and the Tensor Core accumulator type is set to FP32, with specific CUTLASS configuration determined by the CUTLASS profiler. Our testbeds is an 8-GPU NVIDIA B200 system with 5th generation NVLink. All the CTA-pipelining implementation utilize the integration with SM100 TMA Warp Specialized kernel introduced in previous section. The baseline we compare against are built using state-of-the-art cuBLAS [29] and NCCL [30] libraries. All execution time we present are the average of 5 repeated runs, where data are collected using NVIDIA Nsight Systems [47]. A. Comparison Against Micro-Batch Chunk Pipelining Intuitively, the CTA-pipelining execution model is similar to micro-batching operating at the finest granularity. However, they are architecturally different, especially in terms of kernel efficiency and parallelism. Unlike traditional methods, CTApipelining does not explicitly pre-partition the input into static chunks, nor does it invoke multiple separate kernel launches for each stage of the pipeline. Instead, it preserves the original kernel structure and leverages the unified memory space within the NVLink domain for coordination. This approach preserves single-kernel efficiency while enabling cross-device parallelism, qualifying it as a general, latency-oriented spatial scaling method. In this subsection, we present experimental results that support these claims. To compare against static micro-batching, we performed experiments running a multi-layer GEMM workload across multiple GPUs using our prototype built with NVIDIA CUTLASS. As a baseline, static micro-batching is built with the NVIDIA cuBLAS library. For this specific scenario, cuBLAS is considered a stronger baseline than CUTLASS, as it selfadapts to different input sizes, providing the most robust comparison when varying static chunk sizes. Furthermore, the baseline is executed using CUDA Graph to minimize kernel launching overhead. For demonstration, the input sequence length is chosen to be 16384, while the weight dimensions are fixed to be 8192 by

8192, as introduced before. Each GPU is assigned one layer of the GEMM workload, and increasing number of GPUs also increases the layer of GEMM operations. Micro-batching is performed by splitting the input matrix in row-major order, where the ”chunk size” we refer to is the input sequence length (row dimension) of the split sub-matrix, and the number of pipeline stages is equal to the number of GEMM layers. This experimental setup is illustrated in Figure 3.

(a)

(b)

Fig. 3: The experiment setup for comparing executing multiple layers of GEMM with (a) micro-batch chunk pipelining, and (b) CTA-pipelining, using GPU=2 as an example. The experimental results are presented in Figure 4. We compare CTA-pipelining against static micro-batching across varying numbers of GPUs and a range of chunk sizes. As optimal chunk size selection is critical for the baseline’s performance, sweeping this parameter ensures a fair comparison. The results demonstrate that CTA-pipelining consistently outperforms static micro-batching across all evaluated configurations. Comparing against the optimal chunk size in the sweeping, CTA-pipelining reduces latency by 31.8%, 30.0%, and 23.4% in 2, 4, and 8 GPU setup, respectively. The more common 2-layer GEMM case, representing MLP layers, is further evaluated across different input sequence length in Figure 5, further proving CTA-pipelining’s effectiveness. The fundamental problem with traditional micro-batch chunk pipelining can be viewed as a dilemma between parallelism and kernel efficiency when selecting chunk sizes. When the chunk size is too large, the head and tail phases of the pipeline execution become longer, where most devices waiting for the previous pipeline stage to finish, thereby decreasing overall parallelism. If the chunk size is too small, the compute efficiency of the individual kernels suffers. GPU kernels require sufficiently large input sizes to maintain high hardware utilization, and small kernels often suffer from quantization effects. Smaller chunk sizes also implies larger number of kernel launches, accumulating more kernel launching bubbles. In contrast, CTA-pipelining achieves overlap at the finest possible granularity: the CTA level. At the same time, as it does not explicitly split the input into discrete chunks, the original compute efficiency of the kernel is preserved. This effectively solves the traditional chunking dilemma. Furthermore, executing micro-batching as a spatially across devices introduces additional overhead. It suffers not only from pipeline bubbles induced by repeated kernel launches,

Number of GPUs Latency Reduction

1 -

2 31.8%

4 30.0%

8 23.4%

Fig. 4: Multi-layer GEMM (16384×8192×8192) latency reduction, comparing CTA-pipelining against micro-batching implemented with cuBLAS. The number of GEMM layers increases with the number of GPUs. but also from cross-device write latency, as the final wave of TMA writes must post before the kernel can terminate. In contrast, our execution trace analysis in Figure 2b demonstrates that, while CTA-pipelining requires a system-wide memory fence to guarantee inter-device memory consistency, the warpspecialized kernel design effectively hides this overhead. Despite its significant benefits, CTA-pipelining does become less effective at extremely small input sizes. As shown in Figure 5, while protocol overhead is mostly hidden, it does stand out when kernel execution time drops to the 100µs range (e.g., sequence length 1024). In the most extreme case, where a kernel requiring only a single wave of CTAs, intra-kernel pipelining becomes physically impossible. Nevertheless, traditional micro-batching is also reaching its limit under such extreme scenarios. In summary, CTA-pipelining is a superior method than micro-batch chunk pipelining in most of the cases. It maximizes cross-device parallelism and preserves native kernel efficiency, while effectively hiding pipeline bubbles and global memory fence overheads. As an added practical benefit, the CTA-pipelining approach saves the tuning effort typically required to discover the optimal micro-batch chunk size. For workflows that already utilize micro-batch pipelining executing, CTA-pipelining can serve as a direct replacement. B. Comparison Against Tensor Parallelism and Integration with Tensor Parallelism for Spatial Scaling As discussed in the introduction section, Tensor Parallelism (TP) has emerged as the de facto standard for latencyoriented scaling in multi-GPU environments. We propose CTA-pipelining as an orthogonal spatial scaling technique that can be combined with TP. The evaluation is this section consists of two parts. First, we compare CTA-pipelining against TP on multi-layer GEMM. Second, we evaluates the integration of CTA-pipelining with TP, demonstrating how their

TABLE I: Multi-layer GEMM (16384×8192×8192) latency reduction, comparing CTA-pipelining against Tensor Parallelism. Number of GPUs Tensor Parallelism (µs) CTA-Pipelining (µs) Latency Reduction

Input R-MB R-TP

1024 1.3% 1.9%

2048 9.1% 6.3%

4096 18.3% 23.9%

8192 26.3% 25.1%

16384 31.8% 29.0%

32768 22.7% 29.6%

R-MB: Latency reduction against micro-batching R-TP: Latency reduction against Tensor Parallelism ∗ The weight matrices are fixed to be 8192×8192. ∗∗ The micro-batching execution latency is obtained by sweeping chunk sizes and selecting the optimal one.

Fig. 5: CTA-Pipelining latency vs. micro-batching and Tensor Parallelism on 2-layer GEMM, representing MLP layers, on 2 GPUs, across different input sequence length. combined application can further push the latency optimization frontier. The baseline TP implementation mirrors the standard Megatron-LM paradigm for LLM MLP layers [20]. Under standard TP for calculating Y = XAB, A is sharded in a column-parallel fashion, and B is sharded in a row-parallel fashion across all participating GPUs. An All-reduce collective operation is then required to sum the partial results. For the pure TP cases, cuBLAS kernels are again used to provide a strong baseline. The NCCL library is used for the All-reduce collective operations. Note that splitting the input tensor X while replicating the weight matrices A and B across devices to achieve Data Parallelism is not the standard practice for LLM deployments. In modern LLMs, transformer layers are repeated numerous times throughout the architecture, each containing multiple massive MLP weight matrices. As the total parameter size exceeds the memory capacity of a single GPU, replicating these weights across devices is impractical. Instead, weight matrices A and B are sharded across devices, while the input tensor X can be replicated. 1) Comparing CTA-Pipelining Against TP: We perform our multi-layer GEMM comparison using the same experimental setup as the micro-batching evaluation above, where the number of GEMM layers increases with the GPU count. Under this setup, following standard Megatron-LM practices, TP requires an All-Reduce operation every two layers. We note that while a 2-layer structure is a more common pattern in real-world workloads, we include one example of multi-layer GEMM here for evaluation completeness. The experiment results are shown in Table I. CTA-pipelining

1 1080 1080 -

2 1640 1165 29.0%

4 2636 1418 46.2%

8 4493 1844 59.0%

effectively reduces multi-layer GEMM latency by 29.0%, 46.2%, and 59.0% on 2, 4, and 8 GPUs, respectively. The primary source of this benefit is that, within this setup, CTA-pipelining completely avoids All-Reduce communication. Conversely, TP requires frequent communication for every two layers. This demonstrates the advantage of using CTA-pipelining as a scaling method for suitable workflows that can execute in pure CTA-pipelining manner end-to-end. As 2-layer GEMM represents a more realistic setup, similarly, we provide further evaluation across different input sequence length in Figure 5. Result demonstrates the general benefit of CTA-pipelining against TP by saving communication time, despite the limitation on extremely small input sizes, which has been discussed in previous subsection. 2) Combining CTA-pipelining with TP: As demonstrated above, while CTA-pipelining provides significant benefits when workflows can execute entirely within this paradigm, some workflows lack enough kernels to form sufficient pipeline stages across multiple GPUs. In such scenarios, we purpose that, CTA-pipelining can be combined with TP as an orthogonal scaling dimension to further reduce latency. We demonstrate this capability using 2-layer GEMMs, representing the MLP layers. To integrate CTA-pipelining with TP, we alter this sharding strategy. Rather than distributing A and B across all devices, we partition the hardware into 2-GPU groups. The weight matrices are then sharded evenly across these groups. Within

(a)

(b)

Fig. 6: The experiment setup for comparing executing 2 Layers of GEMM using (a) pure TP, and (b) the combination of TP and CTA-pipelining, using GPU=4 as an example.

(a)

(b)

(c)

Fig. 7: The execution latency breakdown of 2-layer GEMM with different input sequence lengths, comparing pure TP and the combination of TP and CTA-pipelining. any given group, the paired GPUs utilize CTA-pipelining to execute the local two-layer GEMM (Yi = XAi Bi ). In this setup, the number of ranks involved in the final Allreduce operation is equal to the number of groups, halving the collective communication world size compared to the pure TP case. Figure 6 provides an example comparing pure TP with this combined CTA-pipelining and TP approach. As described before, the weight matrices are fixed to be 8192 by 8192. We vary the input sequence length across 4096, 8192, and 16384 as representative user workload sizes. The results are shown in Figure 7, with the lines showing the trend of total execution latency, and bars explicitly separate the computation and communication times for each setup. These results demonstrate that combining CTA-pipelining with TP provides a beneficial deployment strategy for multi-GPU scenarios, by pushing the latency frontier even further compared to pure TP deployment. The most direct benefit comes from the reduction in the world size involved in the All-reduce operation, which directly reduces the overall communication time. In some cases, such as in Figure 7c, increasing pure TP degree can result in negative impact, since the communication time dominates, while combining with CTA-pipelining continues the latency reduction with more GPU resources. The other factor to consider is the pure computation time. When the TP degree is large, the input size per GPU becomes small, which can degrade kernel efficiency, similar to the micro-batching discussed earlier. By integrating CTApipelining, we effectively reduce the required TP degree, preserves larger matrix dimensions, and thereby benefits compute efficiency. However, executing via CTA-pipelining introduces additional pipeline’s ramp-up delay (the one-wave CTA execution latency), which counteracts some of these computational gains. As a result, whether the computation time get benefit from combining with CTA-pipelining varies across different matrix sizes. In summary, the above experiments demonstrates that CTApipelining is a qualified spatial scaling method that is orthogonal to TP, allowing it to be seamlessly combined with TP for multi-GPU deployments. Compared to pure TP, this combined approach provides clear communication benefits, along

with computational benefits in certain scenarios. Ultimately, this provides a powerful alternative for deploying multi-GPU workloads and holds the potential to further scale applications for lower latency, particularly when the whole workload can be executed in pure CTA-pipelining manner. V. D ISCUSSION A. Memory Consistency Using Lamport Synchronization As described in previous sections, to guarantee memory consistency during inter-kernel signaling, an explicit systemwide threadfence is issued after the output data write and before workqueue update. This ensures correct memory visibility ordering across the entire multi-GPU system. However, a system-wide threadfence is an expensive operation, especially when it must be issued for every single tile. Although we demonstrated that in warp-specialized multi-stage persistent kernels, this overhead can be hidden, classical kernels still suffer from this overhead. Furthermore, our empirical experiments show that even without an explicit threadfence, the probability of an out-of-order global write is low, motivating the exploration of further optimizations. To address this, we evaluated an alternative method to guarantee memory consistency, referred to as Lamport synchronization. Under this approach, explicit threadfence are no longer used. Instead, the dependent data buffer between kernels is pre-initialized with dummy sentinel values (such as negative zero) that are unlikely to be produced during normal computation. Upon retrieving a ready CTA ID from the workqueue, the consumer kernel directly reads the dependent data from global memory, and performs an additional validation check. If the read data matches the dummy value, the kernel continuously re-reads from global memory until valid data is observed. Despite its theoretical viability, Lamport synchronization introduces hardware and software implementation challenges. On Blackwell, data transfers directly to Tensor Memory (TMEM). As TMEM is only accessible to Tensor Cores, directly checking fetched data against dummy values is infeasible, while inferring correctness from final Tensor Core outputs instead dramatically increases programming complexity. Furthermore, Lamport synchronization might require tracking in-

termediate states (e.g., GEMM K-indices), which can increase shared memory pressure and degrades performance. Finally, the introduction of Lamport synchronization also breaks the simplicity of only adding prologue and epilogue code snippets to enable CTA-pipelining, creating challenge for automating the integration. As a result, we have excluded Lamport synchronization as the default configuration for our CTA-pipelining protocol. Nevertheless, we note that if Lamport synchronization were supported at the hardware level, where the memory subsystem natively checks sentinel values during global memory reads, this approach would become highly practical. Such hardwareassisted checking would benefit many applications far beyond CTA-pipelining. B. Tile-Based Models and Inter-Kernel Signaling It is natural to connect CTA-pipelining to the tile-based execution model, which has recently gained popularity with the introduction of frameworks like Triton [48] and cuTile [49]. If a GPU program is described using the tile abstraction, achieving tile-level inter-kernel pipeline execution becomes a highly intuitive process. The tile abstraction largely formalizes the input and output patterns of a kernel, thereby simplifying the dependency analysis between kernels. This demonstrates the potential of CTA-pipelining to serve as a widely adaptable technique for multi-kernel workflows in future GPU programming paradigms. Building on these insights, enabling more efficient CTAlevel inter-kernel signaling presents bigger potential impact. While our current CTA-pipelining protocol is fully functional on current hardware using strictly user-level CUDA code, it exposes a broader architectural paradigm shift. In modern GPUs, dense arithmetic is increasingly offloaded to specialized accelerators, frequently leaving the general-purpose Arithmetic Logic Units underutilized. Our methodology demonstrates how these idle general-purpose units can be effectively repurposed for inter-kernel orchestration. On the other hand, this also hints the potential for native driver-level support and future hardware integration for more efficient inter-kernel signalling. C. Communication Overlap and NVLink Topology The NVLink interconnect attempts to create the illusion of a unified multi-GPU system, where GPUs can access each other’s memory as if it were local. To a large extent, this abstraction is successful, and our CTA-pipelining technique effectively utilize this illusion. However, current hardware still possesses physical limitations that prevent it from fully supporting this seamless abstraction. Analyzing how communication actually occurs during CTA-pipelining execution demonstrates these limitations, while also highlighting how more advanced hardware could eventually resolve them. Within our 8-GPU B200 testbed, the devices are interconnected via NVLink and NVSwitch. While this fabric theoretically support a peer-to-peer bandwidth of 1.8 TB/s [50], the routing is physically realized through the centralized switch. In

contrast to traditional HPC topologies [51] that approximate non-blocking all-to-all connectivity, this star-like centralized architecture bottlenecks the maximum concurrent ingress and egress bandwidth per GPU. Under the CTA-pipelining execution paradigm, this centralized NVLink topology restricts the full potential for computecommunication overlap. As the producer kernel writes output data directly to the consumer’s device memory, leveraging the illusion of a shared-memory machine, it consumes NVLink bandwidth under the hood. Consequently, if the consumer device wants to achieve compute-communication overlap by communicating with a device other than the producer, its performance is still degraded by the producer’s ongoing memory writes. Even though the producer is not actively involved in the consumer’s secondary communication, both operations share the same physical interconnect links to the centralized NVLink Switch, resulting in contention. Theoretically, these bottlenecks can be resolved by more advanced NVLink topologies, similar to the complex fabrics that currently exist for HPC clusters. For instance, the GB200 NVL72 [5] system is capable to have more NVLink Switch. If we strategically place the workflow, such that a stack of kernels executing via CTA-pipelining is localized under a single NVLink Switch, while secondary communication is routed through a separate switch, compute-communication overlap becomes highly feasible. This would further increase the power of CTA-pipelining and hints at potential hardware-software co-design on upcoming large-scale multiGPU shared-memory systems beyond NVL72. VI. C ONCLUSION In conclusion, this paper introduces CTA-pipelining, a latency-oriented spatial scaling paradigm that exploits modern multi-GPU shared-memory systems, aiming to reduce the single batch processing latency required by latest LLM inference workload. Through a prototype integration with NVIDIA CUTLASS, we demonstrate our protocol has minimal overhead, and can benefit from the warp-specialized multi-stage kernel design. When being used for spatial scaling, CTApipelining outperforms traditional micro-batch chunk pipelining and Tensor Parallelism on the example MLP setup, across various input sequence lengths. Furthermore, we establish CTA-pipelining as a general scaling dimension that composes orthogonally with Tensor Parallelism to further push the multiGPU latency optimization frontier. As computation architectures evolves, CTA-pipelining offer a paradigm for future hardware-software co-design and next-generation multi-GPU execution model. R EFERENCES [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010. [Online]. Available: https://papers.nips.cc/paper files/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

[2] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361 [3] R. Zhen, J. Li, Y. Ji, Z. Yang, T. Liu, Q. Xia, X. Duan, Z. Wang, B. Huai, and M. Zhang, “Taming the titans: A survey of efficient llm inference serving,” 2025. [Online]. Available: https://arxiv.org/abs/2504.19720 [4] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” in Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024. [Online]. Available: https://dl.acm.org/doi/10.5555/3691938.3691945 [5] NVIDIA, “Nvidia gb200 nvl72: Rack-scale system built for the age of ai reasoning.” [Online]. Available: https://www.nvidia.com/en-us/ data-center/gb200-nvl72/ [6] ——, “Nvidia nvlink and nvlink switch.” [Online]. Available: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ [7] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165 [8] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “Sglang: efficient execution of structured language model programs,” in Proceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY, USA: Curran Associates Inc., 2024. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2024/hash/ 724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html [9] llm d, “llm-d: a kubernetes-native high-performance distributed llm inference framework.” [Online]. Available: https://llm-d.ai [10] T. A. Team, J. Shan, V. Gupta, L. Xu, H. Shi, J. Zhang, N. Wang, L. Xu, R. Kang, T. Liu, Y. Zhang, Y. Zhu, S. Jin, G. Lim, B. Chen, Z. Chen, X. Liu, X. Chen, K. Yin, C.-P. Chung, C. Jiang, Y. Lu, J. Chen, C. Lin, W. Xiang, R. Shi, and L. Xie, “Aibrix: Towards scalable, cost-effective large language model inference infrastructure,” 2025. [Online]. Available: https://arxiv.org/abs/2504.03648 [11] NVIDIA, “Nvidia dynamo: Scale and serve ai inference—fast.” [Online]. Available: https://www.nvidia.com/en-us/ai/dynamo/ [12] S. H. Sul, S. Arora, B. F. Spector, and C. Ré, “Parallelkittens: Systematic and practical simplification of multi-gpu ai kernels,” 2025. [Online]. Available: https://arxiv.org/abs/2511.13940 [13] S. Heldens, P. Hijma, B. Van Werkhoven, J. Maassen, and R. V. van Nieuwpoort, “Lightning: Scaling the gpu programming model beyond a single gpu,” in 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2022, pp. 492–503. [Online]. Available: https://doi.org/10.1109/IPDPS53621.2022.00054 [14] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro, “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” 2022. [Online]. Available: https://arxiv.org/abs/2201.11990 [15] L. Zheng, Z. Li, H. Zhang, Y. Zhuang, Z. Chen, Y. Huang, Y. Wang, Y. Xu, D. Zhuo, E. P. Xing, J. E. Gonzalez, and I. Stoica, “Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 559–578. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/zheng-lianmin [16] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3458817.3476209 [17] M. Qi, J. Peng, J. Zhang, J. Zhu, Y. Li, and H. Ma, “Synergistic

tensor and pipeline parallelism,” 2025. [Online]. Available: https: //arxiv.org/abs/2510.27257 [18] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2019/file/093f65e080a295f8076b1c5722a46aa2-Paper.pdf [19] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–15. [Online]. Available: https://doi.org/10.1145/3341301.3359646 [20] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053 [21] T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” in Proceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen, Eds., vol. 5. Curan, 2023, pp. 288–304. [Online]. Available: https://proceedings.mlsys.org/paper files/paper/ 2023/file/5a54f79333768effe7e8927bcccffe40-Paper-mlsys2023.pdf [22] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024. [Online]. Available: https://dl.acm.org/doi/10.5555/3691938.3691949 [23] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: fast and memory-efficient exact attention with ioawareness,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html [24] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He, “Deepspeedinference: Enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, 2022, pp. 1–15. [Online]. Available: https://doi.org/10.1109/SC41404.2022.00051 [25] M. Wu, X. Cheng, S. Liu, C. Shi, J. Ji, K. Ao, P. Velliengiri, X. Miao, O. Padon, and Z. Jia, “Mirage: A multi-level superoptimizer for tensor programs,” in 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). Boston, MA: USENIX Association, Jul. 2025. [Online]. Available: https://dl.acm.org/doi/10.5555/3767901. 3767914 [26] X. Cheng, Z. Zhang, Y. Zhou, J. Ji, J. Jiang, Z. Zhao, Z. Xiao, Z. Ye, Y. Huang, R. Lai, H. Jin, B. Hou, M. Wu, Y. Dong, A. Yip, Z. Ye, S. Wang, W. Yang, X. Miao, T. Chen, and Z. Jia, “Mirage persistent kernel: A compiler and runtime for mega-kernelizing tensor programs,” 2025. [Online]. Available: https://arxiv.org/abs/2512.22219 [27] A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills,” 2023. [Online]. Available: https: //arxiv.org/abs/2308.16369 [28] NVIDIA, “Cutlass library.” [Online]. Available: https://docs.nvidia.com/ cutlass/latest/ [29] ——, “cublas: Basic linear algebra on nvidia gpus.” [Online]. Available: https://developer.nvidia.com/cublas [30] ——, “Nvidia collective communications library (nccl).” [Online]. Available: https://developer.nvidia.com/nccl [31] ——, “Cuda programming guide.” [Online]. Available: https://docs. nvidia.com/cuda/cuda-programming-guide/index.html [32] E. Lindholm, J. Nickolls, S. Oberman, and J. Montrym, “Nvidia tesla: A unified graphics and computing architecture,” IEEE Micro, vol. 28, no. 2, pp. 39–55, 2008. [Online]. Available: https: //doi.org/10.1109/MM.2008.31 [33] NVIDIA, “Nvidia h100 tensor core gpu architecture,” 2022. [Online]. Available: https://resources.nvidia.com/en-us-tensor-core/ nvidia-hopper-architecture-whitepaper

[34] ——, “Nvidia groq 3 lpx: The inference accelerator for nvidia vera rubin.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ lpx/ [35] W. Kwon, G.-I. Yu, E. Jeong, and B.-G. Chun, “Nimble: lightweight and parallel gpu task scheduling for deep learning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020. [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/hash/5f0ad4db43d8723d18169b2e4817a160-Abstract.html [36] Y. Shi, Z. Yang, J. Xue, L. Ma, Y. Xia, Z. Miao, Y. Guo, F. Yang, and L. Zhou, “Welder: Scheduling deep learning memory access via tile-graph,” in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). Boston, MA: USENIX Association, Jul. 2023, pp. 701–718. [Online]. Available: https://www.usenix.org/conference/osdi23/presentation/shi [37] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “Tvm: an automated end-to-end optimizing compiler for deep learning,” in Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’18. USA: USENIX Association, 2018, p. 579–594. [Online]. Available: https: //dl.acm.org/doi/abs/10.5555/3291168.3291211 [38] A. Li, B. Zheng, G. Pekhimenko, and F. Long, “Automatic horizontal fusion for gpu kernels,” in Proceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization, ser. CGO ’22. IEEE Press, 2022, p. 14–27. [Online]. Available: https://doi.org/10.1109/CGO53902.2022.9741270 [39] L. Ma, Z. Xie, Z. Yang, J. Xue, Y. Miao, W. Cui, W. Hu, F. Yang, L. Zhang, and L. Zhou, “Rammer: Enabling holistic deep learning compiler optimizations with rTasks,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 881–897. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/3488766.3488816 [40] Z. Zheng, C. Oh, J. Zhai, X. Shen, Y. Yi, and W. Chen, “Versapipe: a versatile programming framework for pipelined computing on gpu,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO-50 ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 587–599. [Online]. Available: https://doi.org/10.1145/3123939.3123978 [41] A. Jangda, S. Maleki, M. M. Dehnavi, M. Musuvathi, and O. Saarikivi, “A framework for fine-grained synchronization of dependent gpu kernels,” in Proceedings of the 2024 IEEE/ACM International Symposium on Code Generation and Optimization, ser. CGO ’24. IEEE Press, 2024, p. 93–105. [Online]. Available: https://doi.org/10.1109/CGO57630.2024.10444873 [42] Z. Zhang, H. Wang, H. Xu, D. Yang, X. Zhou, and D. Cheng, “Hytis: Hybrid tile scheduling for gpu gemm with enhanced wave utilization and cache locality,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 1604–1618. [Online]. Available: https://doi.org/10.1145/3712285.3759771 [43] K. Zhu, Y. Gao, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, T. Tang, Q. Xu, Z. Ye, K. Kamahori, C.-Y. Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci, “Nanoflow: towards optimal large language model serving throughput,” in Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI ’25. USA: USENIX Association, 2025. [Online]. Available: https://dl.acm.org/doi/10.5555/3767901.3767942 [44] L.-W. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y. Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang, H. Lin, X. Jin, and X. Liu, “Flux: Fast software-based communication overlap on gpus through kernel fusion,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06858 [45] S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi, B. Hechtman, D. Chen, K. S. Murthy, M. Maggioni, Q. Zhang, S. Kumar, T. Guo, Y. Xu, and Z. Zhou, “Overlap communication with dependent computation via decomposition in large deep learning models,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2022, p. 93–106. [Online]. Available: https://doi.org/10.1145/3567955.3567959 [46] M. Davies, N. Crago, K. Sankaralingam, and S. Keckler, “Kitsune: Enabling dataflow execution on gpus with spatial pipelines,” ACM

Trans. Archit. Code Optim., vol. 22, no. 4, Dec. 2025. [Online]. Available: https://doi.org/10.1145/3777466 [47] NVIDIA, “Nvidia nsight systems.” [Online]. Available: https://developer. nvidia.com/nsight-systems [48] P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, ser. MAPL 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 10–19. [Online]. Available: https://doi.org/10.1145/3315508.3329973 [49] NVIDIA, “cutile python.” [Online]. Available: https://docs.nvidia.com/ cuda/cutile-python/ [50] ——, “Nvidia hgx platform.” [Online]. Available: https://www.nvidia. com/en-us/data-center/hgx/ [51] W. Dally, “Performance analysis of k-ary n-cube interconnection networks,” IEEE Transactions on Computers, vol. 39, no. 6, pp. 775–785, 1990. [Online]. Available: https://doi.org/10.1109/12.53599