When Does Heterogeneity Pay Off? Understanding GPU-WSE System Complementarity

arXiv:2610.39222v1 [cs.DC] 29 Oct 2026

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity Ziyu Hu

Zhijing Ye

Jiamin Wang

zhu31@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

zye25@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

jwang259@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

Zhiqing Zhong

Zhendong Mi

Shaoyi Huang

zzhong9@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

zmi2@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

shuang59@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

Weijian Zheng

Rajkumar Kettimuthu

Xiaodong Yu

wzheng@anl.gov Argonne National Laboratory Lemont, Illinois, USA

kettimut@anl.gov Argonne National Laboratory Lemont, Illinois, USA

xyu38@stevens.edu Stevens Institute of Technology Hoboken, New Jersey, USA

Abstract

to efficiently serve all workload. The complementary architectural characteristics of GPUs and WSEs provide an opportunity to address this challenge through heterogeneous integration. GPUs integrate a large number of highly parallel compute units and leverage SIMT execution to effectively utilize these resources, delivering high computational throughput. In contrast, WSEs leverage their massive chip area to integrate large amounts of distributed SRAM, allowing larger working sets to reside entirely on-chip and transforming much of the data movement that would otherwise rely on off-chip memory into on-chip accesses. Consequently, these distinct architectures exhibit different execution characteristics across AI workloads, creating opportunities for architectural complementarity in heterogeneous systems.This opportunity is also beginning to attract industrial interest, with emerging systems exploring the integration of AI accelerators with complementary architectural characteristics. For example, AWS and Cerebras have announced plans to integrate Trainium and WSE within a heterogeneous AI infrastructure[3]. Challenges in Exploiting GPU-WSE Complementarity: While the architectural differences between GPUs and WSEs create theoretical opportunities for complementarity, they also introduce substantial architectural asymmetry. Consequently, the feasibility boundaries of heterogeneous GPU–WSE execution have not been systematically studied. First, current AI system stacks are primarily designed for homogeneous accelerators or accelerators with similar architectures. Their parallelization strategies, system software, and interconnect mechanisms typically assume comparable hardware capabilities and execution characteristics, making them ill-suited to coordinating fundamentally different accelerators while fully exploiting their respective strengths.

As AI accelerator architectures become increasingly diverse, integrating different architectures into a unified system is emerging as a new heterogeneous computing paradigm. GPUs and WSEs differ substantially in compute, memory, and communication organization, creating both architectural complementarity and additional overheads, yet how to translate this heterogeneity into practical performance gains remains unclear. To address this gap, we propose a hierarchical benchmarking methodology to characterize GPU–WSE heterogeneous execution and the conditions for exploiting complementarity, together with an analytical performance model that identifies when heterogeneity becomes a bottleneck and when it yields system-level performance gains.

1

Introduction

GPUs have been remarkably successful in accelerating modern AI workloads due to their massive parallelism and mature software ecosystem, making them the de facto standard for both training and inference[16, 25]. However as AI workloads become increasingly diverse, the limitations of general-purpose GPU architectures have become more apparent, driving the emergence of specialized accelerator architectures optimized for AI workloads. Wafer-scale accelerators represent an emerging class of specialized AI architectures that construct an entire wafer as a single computing device, rather than dividing it into multiple individual chips. For example, the Wafer-Scale Engine (WSE)[17] distributes approximately 850K compute cores and 44 GB of local SRAM across a single wafer-scale chip. Motivation for GPU–WSE Integration: Modern AI workloads exhibit diverse computational and memory characteristics, making it difficult for a single accelerator architecture 1

Hu et al.

Second, even though existing heterogeneous systems account for hardware differences, the forms of heterogeneity they address differ from those in GPU–WSE systems. For example, heterogeneous GPU systems [8, 10, 15, 18, 19] primarily focus on heterogeneity across different generations of GPUs, while their underlying architectures remain largely the same. GPU–FPGA systems [1, 11] further introduce architectural heterogeneity, but FPGAs are general-purpose reconfigurable hardware whose design goals and execution characteristics differ substantially from those of specialized AI accelerators. Third, emerging heterogeneous platforms incorporating specialized AI architectures remain largely under development, and their performance characteristics remain largely unavailable to the research community. Consequently, existing heterogeneous system studies have not sufficiently explored heterogeneous cooperation between GPUs and specialized AI accelerators with substantially different architectures. This leaves three fundamental questions: 1) Do the architectural differences between GPUs and WSEs create exploitable performance complementarity? 2) Under what application-level execution constraints can such complementarity translate into end-to-end performance gains? 3) What system support is required to realize these gains? Our Approach: To answer these questions, we develop a hierarchical benchmarking methodology to systematically investigate the impact of architectural differences between GPUs and WSEs. At the application layer, we select three representative LLM execution paradigms—data parallelism, pipeline parallelism, and prefill–decode disaggregation—as case studies to analyze how architectural differences affect heterogeneous system performance. At the system layer, we investigate how compilers, communication mechanisms, and runtime systems should adapt to these architectural differences; at the hardware layer, we characterize the physical requirements for efficiently integrating these different architectures. Observation: Our results show that architectural difference is not inherently beneficial: its impact depends on whether the workload can be partitioned flexibly enough to accommodate architecture-specific asymmetry, and whether heterogeneous tasks can be mapped to the architectural str- engths of different devices. We further develop an analytical performance model that captures the key effects of architectural differences on heterogeneous execution. The main contributions are summarized as follows:

execution constraints can amplify, mitigate, or exploit GPU–WSE architectural differences. Based on these observations, we further identify the system and hardware support required to accommodate such architectural heterogeneity. • We develop a lightweight analytical model that formalizes these observations by capturing the interactions among workload decomposition, architectural asymmetry, resource utilization, and interconnect overhead. We validate the model against representative measured configurations and use it to characterize feasibility boundaries and reason about unmeasured GPU– WSE configurations.

2

Background

This section provides background on Wafer-Scale accelerators and commonly used parallelization strategies. 2.1

Architectural Differences between GPUs and Wafer-Scale Accelerators

GPUs and WSE exhibit significant architectural differences in the organization of compute, memory, and interconnection. We use the A100 and Cerebras WSE as representative architectures for comparison, as shown in Fig. 1. WSE

GPU

HBM

PE

PE

PE

SM SM

...

PE

PE

PE

PE

PE PE

...

Register (256KB)

SM

SM

PE

...

SM

Core

SM

...

Core

...

...

68

Core

SM

...

Core

SM

L2 Cache (40MB)

L1 Cache (192KB)

PE PE

...

Router

Core

PE

PE

PE

PE

SRAM (48KB)

Figure 1. Architecture comparison between A100 and WSE3, highlighting WSE’s on-chip memory with higher capacity and closer proximity to each compute core compared to the GPU. Compute. The A100 consists of multiple Streaming Multiprocessors (SMs) and employs the SIMT execution model to schedule a large number of threads on each SM, effectively utilizing highly parallel compute units to achieve high computational throughput. In contrast, the WSE leverages the massive chip area enabled by wafer-scale integration to distribute approximately 850K processing elements (PEs) across a single wafer. Each PE contains an independent compute core, with computation partitioned and mapped across a large number of PEs and driven by dataflow execution. Thus, GPUs achieve high arithmetic throughput through densely integrated compute units and massive thread-level parallelism, whereas the WSE exploits wafer-scale integration to deploy fine-grained compute units for distributed execution across the wafer. Memory. The A100 employs a hierarchical memory system consisting of on-chip registers, shared memory, and caches,

• We present a hierarchical, measurement-driven benchmarking methodology that examines how GPU–WSE architectural differences affect heterogeneous execution and its feasibility boundaries across the application, system, and hardware layers. • We apply this methodology to three representative LLM execution paradigms and reveal how different 2

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity

Table 1. Heterogeneous Workload Resource Requirements.

together with off-chip HBM, providing multiple levels with different capacities and bandwidths. In contrast, the WSE leverages its wafer-scale area to integrate a large amount of distributed SRAM. Each PE is tightly coupled with up to 48KB of local SRAM, providing approximately 44GB of onchip memory across the wafer. Thus, GPUs primarily rely on memory hierarchy and data reuse to reduce off-chip memory accesses, whereas the WSE allows larger working sets to be distributed and retained close to the compute cores, enabling more data accesses to remain on-chip. Scaling. GPU typically scale by adding discrete chips, with data exchange between GPUs relying on device-level interconnects such as NVLink. In contrast, the WSE can store model data in the external MemoryX system and stream it to the wafer on demand. For example, during training, the Weight Streaming mechanism sequentially transfers the weights of each layer from MemoryX to the wafer for execution. Thus, GPUs primarily scale computational capacity through multi-device parallelism, whereas a single WSE can execute the entire model computation. 2.2

Cross-Device Execution Strategies for LLM Workloads

Mem BW

Mem Cap

Net BW

Storage Cap

high low high low low

low high high medium high

medium high medium high high

medium low high high low

low low medium high medium

Table 2. Heterogeneous Resource Provisioning. Vendor

Chip

Cerebras NVIDIA

WSE3 Wafer-Scale A100 Von Neumann

3.2

Arch.

onchip

offchip

FLOPS Power

44 GB 0.19 GB

13 TB 40 GB

125P 312T

23 kW 400 W

Challenges in Exploiting GPU–WSE Architectural Complementarity

As discussed above, while the architectural differences between GPUs and WSEs create potential complementarity, they also introduce substantial architectural asymmetry, which raises three key challenges: Realizing architectural complementarity through heterogeneous execution. Although GPUs and WSEs provide complementary hardware capabilities, whether such complementarity can be effectively exploited under heterogeneous execution remains unclear. Existing parallelization strategies are primarily designed for homogeneous systems and typically rely on uniform workload partitioning across devices. In heterogeneous GPU–WSE systems, hardware asymmetry breaks this assumption, while the tolerance of these strategies to performance asymmetry remains unclear. Moreover, whether these strategies can effectively exploit the architectural complementarity between GPUs and WSEs is also unclear.

Motivation

This section presents the motivation for constructing heterogeneous systems that combine GPUs and wafer-scale architectures, as well as the key considerations involved. 3.1

Compute

Prefill Decode Training RAG Embedding

Limitations of Homogeneous AI Systems. Platforms with homogeneous accelerators provide fixed ratio of compute, bandwidth, and capacity per node, which is effective and scalable for workloads with consistent computation patterns. However, it become increasingly inefficient as workload heterogeneity grows, since different workloads are bottlenecked by different resource dimensions. As workload characteristics diverge, one resource configuration can no longer achieve efficient utilization, leading to a fundamental mismatch between workload requirements and system capabilities. Hardware Heterogeneity as an Opportunity. Different accelerator architectures exhibit distinct performance characteristics across resource dimensions (Table 2). This diversity enables complementary strengths across hardware platforms. By mapping workloads to the most suitable accelerators, heterogeneous systems can achieve better resource–demand alignment, improving both utilization and throughput.

Modern LLM systems rely on cross-device execution to distribute workloads and coordinate computation across multiple devices. During training, data parallelism (DP) [22] partitions the input data across devices, where each device maintains a model replica and synchronizes gradients through all-reduce. Pipeline parallelism (PP) [13] partitions the model into stages across devices and transfers intermediate activations between them. During inference, prefill–decode (PD) disaggregation [27] places the prefill and decode stages on different devices, allowing the two stages to be executed and managed separately. We use these three representative cross-device execution paradigms as case studies to study architectural heterogeneity in GPU–WSE systems.

3

Workload

Motivation for GPU–WSE Heterogeneous Integration

GPU–WSE heterogeneous systems provide an opportunity to address increasingly diverse AI workload demands through complementary hardware capabilities. AI Workload Heterogeneity. Modern AI workloads inherently consist of heterogeneous patterns. They present substantially different resource demands across compute, bandwidth, and capacity (Table 1). Such diversity results in varying performance bottlenecks across AI workloads. 3

Hu et al.

Adapting system support to architectural asymmetry. Realizing the potential benefits of GPU–WSE complementarity requires the system stack to coordinate devices with substantially different execution characteristics. Existing compiler, communication, and runtime mechanisms are primarily designed for homogeneous platforms and may not effectively accommodate such asymmetry. It therefore remains unclear what system support is required to efficiently coordinate heterogeneous execution. Meeting architectural requirements for heterogeneous platforms. Even with effective workload mapping and system support, the benefits of heterogeneous execution depend on whether the underlying hardware can efficiently support cross-device coordination. In particular, communication between architecturally distinct devices may offset their local performance advantages. It remains unclear what interconnect capabilities are required for GPU–WSE complementarity to translate into system-level performance gains.

4

conditions under which the potential benefits of heterogeneous execution can translate into end-to-end performance gains. We describe each layer in detail as follows: Application Layer. At this layer, we investigate what execution characteristics enable parallelization strategies to tolerate GPU–WSE performance asymmetry and exploit the complementarity between the two architectures. For heterogeneous execution, a key distinction lies in the constraints imposed by different execution paradigms. Based on existing execution models, we focus on two representative coordination paradigms: BSP-style execution follows a structured computation–communication–synchronization pattern with explicit global coordination, whereas task-based execution organizes computation as dependency-driven tasks, enabling more flexible task scheduling and resource mapping. These two paradigms represent different degrees and forms of execution constraints, providing representative contrasts for analyzing heterogeneous execution. System Layer. At this layer, we investigate how the existing system stack should adapt to GPU–WSE architectural asymmetry to support efficient heterogeneous execution. Such architectural asymmetry introduces three key requirements for the system stack: generating asymmetric workload mappings that account for distinct hardware capabilities, supporting asymmetric communication across architectures with different communication characteristics, and dynamically coordinating execution under performance asymmetry across devices. These requirements correspond to the core functionalities of the compiler, communication mechanisms, and runtime, respectively. We therefore analyze the system support required for GPU–WSE heterogeneous execution from these three aspects. Hardware Layer. At this layer, we examine whether the potential benefits of heterogeneous execution can translate into end-to-end performance gains under interconnect constraints. Heterogeneous execution inevitably introduces cross-architecture communication, and insufficient interconnect capability may amplify the impact of architectural asymmetry or offset architectural complementarity, potentially making heterogeneous execution less efficient than singlearchitecture execution. We therefore characterize the minimum interconnect bandwidth required for effective GPU– WSE heterogeneous execution.

Hierarchical Benchmarking Design

In this section, we present our hierarchical benchmarking study design and describe the benchmarking procedure for evaluating the GPU-WSE heterogeneous system across the three considerations. APPLICATION LAYER DATA PARALLELISM

PIPELINE PARALLELISM

Model

Stage1

Model

Model

sync

Stage2 p2p

Stage3 p2p

P/D DISAGGREGATION Prefill

Decode KV cache

HETEROGENEITY-AWARE LOAD BALANCING COMPILER

COMMUNICATION

RUNTIME

Performance-aware Partition Multi-configuration Generation

Cross-domain P2P Aggregation & Redistribution

Performance-aware Scheduling Configuration Selection

SYSTEM LAYER GPU CLUSTER Compute Efficiency

cross-domain Interconnect

ACCELERATORS Memory Efficiency

HARDWARE LAYER

Figure 2. Overview of hierarchical benchmarking design for GPU-WSE heterogeneous systems. The benchmarking methodology across three layers: application, system, and hardware, corresponding to the three considerations. 4.1

Benchmarking Framework

To address the three challenges discussed above, we develop a three-tier benchmarking framework spanning the application, system, and hardware layers, as illustrated in Fig. 2. The framework systematically explores how GPU–WSE architectural differences affect heterogeneous execution: the application layer investigates what characteristics of parallelization strategies determine their tolerance to architectural asymmetry and ability to exploit architectural complementarity; the system layer investigates how compiler, communication, and runtime mechanisms should adapt to architectural asymmetry; and the hardware layer examines the interconnect

4.2

Benchmarking Procedure

BSP-Style Execution. As illustrated in Fig. 3, BSP-style execution organizes computation into a sequence of supersteps, with a global synchronization barrier at the end of each superstep. When the GPU and WSE operate at different execution rates, the faster domain must wait for the slower one. Workload partitioning can mitigate this imbalance by adjusting the amount of work assigned to each domain, but its effectiveness is constrained by the partitioning dimensions and execution constraints of each parallelization strategy. 4

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity Superstep3

W1

W1’

5.1

W2’

We evaluate GPU–WSE heterogeneous execution using computing resources at the Argonne Leadership Computing Facility (ALCF), including measurements from a Cerebras WSE-3 cluster and an NVIDIA DGX A100 GPU cluster. We consider models with 7B, 13B, 35B, and 70B parameters, and select hardware configurations for each experiment according to the deployment and parallelization requirements of the corresponding workload. DP and PP are evaluated under training workloads, while PD disaggregation is evaluated under inference workloads. We independently measure the execution characteristics of the GPU and WSE on real hardware and combine them using the calibrated discrete-event simulator SimAI [24]. SimAI is designed for large-scale GPU-cluster simulation and supports user-defined workload and measured computation-time inputs, allowing us to incorporate the measured WSE behavior into heterogeneous execution simulation. By replaying measured configurations, we verify that the simulated results closely match the corresponding hardware measurements.

W2 W3 W4

W2 W3

Runtime

Superstep2

W1 Communicate

Compiler

Workload

Superstep1

W4

W3’ W4’

Figure 3. BSP workflow across heterogeneous hardware. Different colors represent different devices, and the length of each block denotes the execution time within a superstep.

Hardware2

Task 3

Runtime

Compiler

Workload

Task 2

Task 3

Communicate Hardware1

Task 2 Task 1

Task 1

Figure 4. Task-style workflow across heterogeneous hardware. The workload is decomposed into a task DAG and scheduled across heterogeneous devices.

We select DP and PP as two representative cases of BSPstyle execution to study how different coordination constraints shape the impact of architectural differences. DP is primarily constrained by BSP-style global synchronization, while batch-dimension partitioning provides relatively high flexibility and loose coordination between the GPU and WSE. In contrast, PP introduces additional pipeline structure, stage partitioning, and finer-grained cross-hardware coordination constraints. By comparing these cases from coarse- to finegrained coordination, we examine how increasing execution constraints change the impact of GPU–WSE architectural differences on heterogeneous execution. Task-Based Execution. As illustrated in Fig. 4, task-based execution decomposes the workload into dependency-driven tasks and allows them to be independently mapped onto heterogeneous hardware. Unlike BSP-style execution, it does not require the two hardware domains to maintain matched execution progress at global synchronization boundaries, but is primarily constrained by task dependencies and crossdomain data transfers. Unlike DP and PP, which mainly partition homogeneous computation in different ways, we select PD disaggregation as a representative case because prefill and decode naturally form two heterogeneous execution stages with distinct compute and memory characteristics, allowing us to study the impact of heterogeneity within the workload on heterogeneous execution.

5

5.2

Experimental Setup

BSP-Style Heterogeneous Data Parallelism

5.2.1 Overall Performance Results. Figure 5 shows the step latency and throughput of the GPU, WSE, and heterogeneous configurations across different global batch sizes. The latency results additionally show the GPU–WSE batch partition that minimizes the heterogeneous step latency, while the throughput results include the theoretical aggregate upper bound obtained by summing the standalone throughput of the two hardware domains. The results reveal three distinct execution regimes separated by two workload thresholds. At small workloads, architectural differences primarily manifest as latency asymmetry. Neither architecture is sufficiently utilized, while their different architectures exhibit different baseline execution latencies. Under BSP synchronization, assigning part of the workload to the higher-latency domain increases the synchronized step time, making single-domain execution preferable. The first threshold therefore marks the point beyond which heterogeneous workload partitioning begins to compensate for this latency asymmetry. Beyond the first threshold, the system enters a scalingasymmetry regime. As shown in Fig. 6, the two architectures exhibit substantially different utilization-scaling behaviors as the batch size increases. This difference stems from their distinct execution organizations and compute scales: the GPU domain can achieve high compute-resource utilization under relatively small workloads, whereas the wafer-scale architecture requires larger workloads to expose sufficient parallelism across its large PE array. In our measurements, the WSE does not achieve high utilization until the batch size approaches 32. These distinct utilization-scaling behaviors

Performance Evaluation and Analysis

In this section, we present experimental results and analysis for three parallelization strategies in heterogeneous systems. We further summarize the key insights of different execution paradigms in heterogeneous settings. 5

Hu et al.

13B

35B

1.0

70B

Ratio

Latency

102 7B

0.5

101

Throughput

100

1

8

256 1024

8

256 1024 1

Batch Size

1

16

512

1

16

512

1

Batch Size

32

1024

32

1024 1

Batch Size

1

32

1024 0.0

32

1024

Batch Size

40 20 0

1

Batch Size

WSE

GPU

Heter

Batch Size

Max

Threshold 1

Batch Size

Threshold 2

Batch Size

GPU Batchsize

WSE Batchsize

Figure 5. Latency (s) of different models under varying global batch sizes (left to right: 7B, 13B, 35B, and 70B). WSE, GPU, Heter, and Max denote the throughput of WSE-only, GPU-only, WSE–GPU hybrid data parallelism (hybrid DP), and its theoretical upper bound, respectively. Threshold 1 and Threshold 2 mark the points where hybrid DP starts to yield performance gains and where the gains are fully saturated, respectively. The bars indicate the batch size allocated to each platform.

Norm. Fraction

further result in different performance-scaling curves, causing the relative processing capabilities of the GPU and WSE to vary with workload size. Consequently, the optimal GPU– WSE batch partition also changes substantially throughout this regime. As the workload further increases, both architectures gradually enter stable utilization regime, forming the second threshold. Beyond this point, their relative processing capabilities stabilize, the optimal workload partition converges, and heterogeneous throughput approaches the theoretical aggregate upper bound. At this stage, architectural asymmetry is largely absorbed through workload partitioning. 100 7B 80 60 40 20 0

1 16 32

the GPU and WSE, indicating that workload partitioning mitigates architectural asymmetry but does not convert architectural differences into additional complementarity. System Layer Analysis. Although DP cannot exploit architectural complementarity to provide additional performance gains, proper workload partitioning can still enable heterogeneous execution to approach the aggregate capacity of the GPU and WSE, thereby utilizing otherwise idle heterogeneous hardware resources. To achieve this, the system stack must effectively accommodate architectural asymmetry. First, the compiler needs to generate asymmetric workload partitions according to the different execution capabilities of the two architectures. Second, in the scalingasymmetry regime, the relative performance of the GPU and WSE varies with workload scale, making the optimal batch partition unstable. The runtime therefore needs to dynamically adjust workload allocation based on observed execution behavior to maintain balanced execution across the two hardware domains. Finally, because large models are typically distributed across multiple devices in the GPU domain but organized within a single wafer-scale execution domain on the WSE, the two domains use different distributed representations. Cross-domain synchronization therefore requires the communication layer to perform the necessary gradient aggregation and redistribution to ensure correct global synchronization.

70B WSE Util GPU Util 64

128

Batch Size

192

WSE Util GPU Util 256

1 16 32

64

128

Batch Size

192

256

Figure 6. Utilization scaling asymmetry. GPU utilization is measured using MFU following [6]. WSE utilization is computed as the product of PE allocation ratio and the utilization of the allocated PEs following [12]. We compare utilization trends within each platform, not absolute values. 5.2.2

Layer-Wise Performance Analysis.

Application Layer Analysis. The three execution regimes show that, architectural differences primarily manifest as architectural asymmetry rather than exploitable architectural complementarity for DP. BSP synchronization exposes this asymmetry, while flexible partitioning along the batch dimension provides a degree of freedom to absorb it as workload increases. However, heterogeneous throughput remains bounded by the aggregate standalone capacity of

Hardware Layer Analysis. To enable heterogeneous DP to provide performance benefits, we further characterize the lower bound on the interconnect bandwidth required to preserve the gains from heterogeneous collaboration, as shown in Fig. 7. Below this lower bound, the benefits of heterogeneous execution are entirely offset by cross-domain 6

Effective Bandwidth

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity

160

of its cores due to its wafer-scale architecture. Under PP, however, all hardware domains must use a unified micro-batch size, preventing the WSE from operating in its preferred large-batch regime and thereby limiting its active-core fraction. Consequently, when one layer from each GPU stage is offloaded to the WSE, the WSE processes these layers under low utilization and eventually becomes the bottleneck of the entire pipeline.

7B 13B 35B 70B

120 80 40 0

8

16

32

64

128

256

Batch Size

512

1024

Figure 7. Effective bandwidth (GB/s) lower bound for heterogeneous execution to achieve throughput gains under varying global batch sizes.

Fraction (%)

100

communication overhead. As workload size increases, the required bandwidth lower bound gradually decreases.

5.3.1

Throughput

Standard PP. Figure 8 shows the results of heterogeneous pipeline parallelism (PP) across four model sizes with a microbatch size of 1, including GPU-only, Cerebras-only, and a range of heterogeneous configurations, where a fixed number of layers are offloaded from each GPU stage to the WSE while the remaining layers are retained on the GPU. In terms of throughput, none of the heterogeneous PP configurations outperform the single-platform setups.

20

3 4 GPU-only

35B

70B

10 0

7B

13B

Model

40

WSE GPU

20 1

8

16

32

Batch Size

48

WSE GPU 64

1

8

16

32

Batch Size

48

64

To determine whether any heterogeneous PP configuration can yield performance gains, we analyze the per-layer latency of the GPU and WSE under different micro-batch sizes. As shown in Fig. 10, among all heterogeneous PP configurations, offloading one layer from each GPU stage to the WSE achieves the highest throughput. We therefore use this configuration as a representative case and define a latency threshold for the WSE, as shown in Fig. 10. This threshold represents the critical execution latency that the WSE must achieve when processing the offloaded layers in order to avoid becoming the slowest stage in the pipeline. As the micro-batch size increases, the per-layer latency of the WSE gradually approaches this threshold. This indicates that, as WSE utilization improves, the performance gap between the WSE and GPU continues to narrow, making heterogeneous execution increasingly likely to provide performance benefits. Although the two curves do not intersect under our current hardware configuration, such an intersection may occur under a different GPU/WSE performance balance. However, due to the memory asymmetry, GPUs cannot in practice support such large micro-batch sizes, preventing the WSE from entering its high-utilization regime.

Overall Performance Results.

WSE-only 1 2

Core Utilization

Figure 9. Compute-core activation rate and utilization. For GPUs, compute cores refer to streaming processors, while for WSEs they refer to processing elements (PEs).

BSP-Style Heterogeneous Pipeline Parallelism

30

Active Cores

60

0

Takeaway 1: GPU-WSE architectural differences are primarily performance obstacles in DP. However, DP’s loose coupling allows flexible system-level workload partitioning to effectively absorb this asymmetry at sufficiently large workloads. 5.3

80

Figure 8. Throughput at global batch size 1024 and microbatch size 1 under GPU, WSE, and hybrid PP placements. Labels denote the number of layers offloaded from each GPU stage to WSE, relative to GPU baseline. Offloading is uniform across stages because PP is constrained by the slowest stage, and partial offloading does not alter bottleneck.

Optimized PP. The previous results show that architectural differences between GPUs and WSEs lead to different preferred workload scales. Forcing different architectures to process workloads of the same scale can prevent each platform from fully exploiting its architectural advantages. Based on this observation, we further explore an architecture-aware heterogeneous PP scheme that allows GPUs and the WSE to use different micro-batch sizes. Multiple GPU pipelines are paired with a single WSE pipeline, enabling the WSE to aggregate smaller GPU microbatches into a larger workload. As shown in Fig. 11, feasible

As shown in Fig. 9, under LLM workloads, the utilization of active cores remains relatively stable on both platforms, indicating that overall utilization is primarily determined by the fraction of activated cores. GPUs can activate a large fraction of their SMs even at very small batch sizes, whereas the WSE requires a batch size of at least 32 to activate nearly all 7

Latency

Hu et al.

102 7B

10

101

2

13B 10

101 1 2 4 8 16 32 64 Micro Batch Size

2

35B

101

70B 102

1 2 4 8 16 32 64 1 2 4 8 16 32 64 Micro Batch Size Micro Batch Size WSE Threshold For WSE GPU

101

1 2 4 8 16 32 64 Micro Batch Size

Figure 10. Per-layer execution latency(ms) of GPU and WSE across different micro-batch sizes, and the thresholds (vs. GPU-only) at which offloading one layer per GPU stage to WSE yields performance benefits. GPU (70B)

However, these configurations still do not reach the theoretical performance upper bound. This indicates that although workload-scale decoupling can alleviate the utilization mismatch caused by architectural differences, the structural constraints of PP and memory asymmetry still limit the system’s ability to absorb such heterogeneity, preventing the potential complementarity between the two architectures from being fully exploited.

WSE (70B)

GPU (35B)

GPU (7B)

WSE (13B)

GPU (13B)

WSE (7B)

WSE (35B)

Figure 11. Feasible placements under varying micro-batch sizes. Infeasible denotes placements that violate pipeline structural constraints; violation denotes placements exceed memory constraints; feasible denotes valid placements.

5.3.2

Application Layer Analysis. The two PP experiments show that, under BSP-style PP execution, architectural differences between GPUs and WSEs likewise primarily manifest as architectural asymmetry rather than exploitable architectural complementarity. Unlike DP, which can flexibly adjust the workload assigned to different hardware domains along the data dimension and use uneven data partitioning to absorb performance asymmetry as much as possible, PP partitions the workload along the layer dimension and inherently provides less workload-partitioning freedom. This freedom is further restricted by constraints such as a unified microbatch size, GPU memory capacity, and pipeline structure. As a result, PP has limited ability to flexibly match workloads to the performance characteristics of different architectures, making architectural asymmetry difficult to fully absorb and more likely to translate directly into a pipeline bottleneck.

configurations exist for all models, demonstrating that workload-scale decoupling can partially accommodate the different preferred workload scales of the two architectures. However, because PP requires fine-grained coordination, memory asymmetry and pipeline structural constraints still limit the freedom of the two architectures to independently select their workload scales, thereby substantially reducing placement flexibility. PP

Cerebras-only

Upper Bound

Communication

4.5

3.0

30 20

1.5

Communication

Throughput

50 40

System Layer Analysis. PP is not an ideal parallelization strategy for GPU–WSE heterogeneous systems, but it can still provide performance benefits under certain configurations by utilizing otherwise idle heterogeneous hardware resources. Therefore, if heterogeneous PP is considered, the most important first step is to determine whether it is actually beneficial. At the compiler level, architectural asymmetry may cause heterogeneous execution to perform worse than execution on a single hardware domain. The compiler therefore needs to perform a feasibility check to determine whether heterogeneous PP can provide performance gains under a given configuration or whether execution should fall back to a single hardware domain. At the same time, PP has a

10 0

7B

13B

35B

70B

Layer-Wise Performance Analysis.

0.0

Figure 12. Throughput (sample/s) of optimized PP vs. WSEonly, and network bandwidth requirements. Finally, for each model, we select a feasible configuration that maintains relatively high WSE utilization and compare its throughput at a global batch size of 1024 against the bestperforming WSE-only baseline in Fig. 12. From the figure, the 7B, 13B, and 35B models all achieve higher throughput. 8

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity Llama-7B Prefill

Llama-70B Prefill

Llama-7B Decode

Llama-70B Decode

Normalized

1.00 0.75 0.50 0.25 0.00

128

256

512 1k 2k 4k Sequence Length

6k

8k

128

256

512 1k 2k 4k Sequence Length

6k

GPU Lat

8k

128

WSE Lat

256

512 1k 2k 4k Sequence Length

GPU TP

6k

8k

128

256

512 1k 2k 4k Sequence Length

6k

8k

WSE TP

Figure 13. Prefill and Decode performance across sequence length. Lines denote latency, and bars represent throughput. Left: 7B; Right: 70B. large configuration search space, making static compiler decisions prone to misprediction. The runtime therefore needs to correct suboptimal compiler decisions based on observed execution behavior and trigger fallback or reconfiguration when necessary. At the communication layer, in addition to providing cross-domain P2P transfers, the system must, as in DP, handle inconsistencies in distributed activation representations across different hardware domains.

300,000

Number of PEs

250,000

150,000 100,000 50,000 0

0–9%

10–19%

20–29%

30–39%

40–49%

50–59%

60–69%

70–79%

80–89%

90–100%

Compute Utilization

Figure 14. WSE PE Utilization Snapshot. Only a small fraction of compute cores achieve utilization above 90%.

Hardware Layer Analysis. For heterogeneous PP, the interconnect still needs to provide sufficient bandwidth for basic cross-domain communication. Using the heterogeneous PP configurations in Fig. 12 as representative cases, we report the effective communication bandwidth required to preserve their performance benefits. Depending on the model size and configuration, the required bandwidth ranges from several hundred MB/s to a few GB/s, providing a practical reference range for GPU–WSE interconnect design.

Task-Based Heterogeneous PD Disaggregation

Memory Used (GiB)

5.4.1 Overall Performance Results. Figure 13 shows the performance of the two hardware platforms under different request shapes. In the prefill stage, GPU throughput increases rapidly with sequence length and consistently remains higher than that of the WSE. In contrast, during the decode stage, the WSE achieves significantly higher throughput and better latency scalability: as the context length increases, GPU decode latency grows from tens to hundreds of milliseconds, whereas the WSE maintains relatively stable microsecond-level latency across different settings. GPUs achieve better performance during prefill for two main reasons. First, the GPU’s SIMT execution model allows a large number of cores to execute the same instructions

20 18 16 WSE SRAM GPU HBM

14 128

256

512

1k

2k

4k

Context Length

6k

8k

Effective BW (TB/s)

on different data elements, making it well suited to regular, dense matrix computations. In contrast, the WSE’s dataflow execution model spatially maps different kernels onto different PE regions, which can produce load imbalance and cause overall performance to be limited by the most heavily loaded region. Second, GPUs operate at a smaller compute scale. Large matrix operations are typically divided into multiple tiles and processed in successive waves, allowing even relatively small workloads to maintain high core utilization. The WSE, by comparison, operates at wafer scale and requires a larger workload to fully utilize its on-chip PEs. Moreover, the shape of a matrix operation may not map efficiently onto its large-scale PE array, leaving some compute resources underutilized. To validate this analysis, we examined PE utilization during a representative interval of WSE execution. As shown in the Figure 14, a large fraction of the PEs exhibit low compute utilization, while only a small fraction achieve utilization above 90%. These results show that the WSE’s execution model and wafer-scale architecture make it more challenging to fully utilize its on-chip compute resources.

Takeaway 2: Under PP, GPU–WSE architectural differences primarily manifest as performance obstacles. PP’s fine-grained coordination and multiple execution constraints amplify the negative effects of architectural asymmetry. As a result, even with substantial system-level effort to accommodate these differences, only limited performance gains can be achieved. 5.4

200,000

10.0 7.5 5.0 2.5 0.0

WSE GPU

128

256

512

1k

2k

4k

Context Length

6k

8k

Figure 15. Memory Footprint and Bandwidth of the WSE and GPU During Decoding. 9

Hu et al. (a) Llama-7B

The WSE can complete a single decode step within tens of microseconds, primarily because of its large-capacity on-chip SRAM and dataflow execution model. As shown in Figure 15, GPUs store the KV cache in off-chip HBM, whereas the WSE can place the relevant working set in distributed on-chip SRAM. Because decoding has low arithmetic intensity, its performance is approximately bounded by 𝐴𝐼 × 𝐵𝑊 . The high bandwidth of on-chip SRAM therefore raises the WSE’s effective performance ceiling while reducing data movement overhead. Meanwhile, the dataflow model can trigger downstream computation as soon as data available, making it particularly suitable for decoding, which involves limited computation and strong data dependencies. As the context length and KV-cache traffic increase, GPU HBM approaches its bandwidth limit more quickly, causing decode latency to increase significantly. In contrast, the WSE maintains relatively stable latency for each decode step. 50:1

6K

20:1 10:1 4:1 2:1

2K

1:1

1K

1:2 1:4

512

1:10

256

Input Length

2K 1K 512

128 128

256

512

1K

2K

4K

Output Length

6K

8K

Hybrid > Aggregate

128

256

512

1K

2K

4K

6K

8K

Output Length

Hybrid < Aggregate

Figure 17. Hybrid Execution Throughput vs. Aggregate Standalone Throughput.

resident on chip, while its high on-chip memory bandwidth provides a substantially higher memory roof than GPU HBM. As a result, the WSE can outperform the GPU in the lowarithmetic-intensity decode regime. In contrast, the higherarithmetic-intensity prefill stage is better able to exploit the GPU’s compute capability. P/D disaggregation therefore exposes different arithmetic-intensity regimes to the architectures best suited to them, creating a workload–hardware matching space in which architectural differences can be exploited as complementarity.

1:20

128 128

256

512

1K

2K

4K

Output Length

6K

8K

128

256

512

1K

2K

4K

6K

System Layer Analysis. At the system layer, the compiler must first identify and separate workloads that are suitable for execution on GPUs and WSEs, while also considering working-set size to ensure, whenever possible, that workloads mapped to the WSE and their associated data can remain resident in on-chip SRAM and thereby exploit the high bandwidth of the wafer-scale architecture. Second, Figure 15 shows that different request shapes require different optimal GPU–WSE resource ratios. The system therefore faces a scheduling space jointly determined by hardware quantity and ratio, request characteristics, and cross-hardware task mapping. As the request distribution and system load change dynamically, the same GPU–WSE configuration may correspond to different optimal workload allocations. The runtime scheduler must therefore dynamically select task routing and determine how much workload should be assigned to GPUs and WSEs. Furthermore, when the dynamic workload does not match the available hardware ratio and leaves one hardware domain underutilized, the system should support work stealing to redirect part of the workload to the nonpreferred hardware and mitigate load imbalance. Finally, the communication layer must support efficient cross-domain point-to-point data transfer, together with conversion and redistribution of distributed KV-cache representations between GPUs and WSEs, to enable coordinated execution across the two hardware domains.

8K

Output Length

Figure 16. Optimal WSE-GPU Ratios for Different Request Shapes. After observing the architectural complementarity between GPUs and WSEs, we further investigate how this complementarity can be translated into actual performance gains. As shown in Figure 16, the optimal GPU–WSE hardware ratio varies across request shapes as the relative prefill and decode workloads change. We then fix the hardware ratio to the optimal configuration for the 1K/1K request shape and apply it to other request shapes, as shown in Figure 17. The results show that a single fixed hardware ratio does not provide performance gains across all request shapes. Compared with the 7B model, GPUs and WSEs exhibit more pronounced performance specialization across the prefill and decode stages for the 70B model, resulting in stronger architectural complementarity and a larger opportunity for performance gains. Therefore, for dynamically changing workloads, architectural complementarity alone does not guarantee performance improvement, and dynamic load balancing remains a key issue. 5.4.2

4K

256

(b) Llama-70B

4K

Input Length

6K

Optimal GPU:WSE Ratio

(a) Llama-7B 8K

(b) Llama-70B

8K

Layer-Wise Performance Analysis.

Application Layer Analysis. Unlike DP and PP, P/D disaggregation decomposes the workload into sub-tasks with substantially different arithmetic intensities. For decoding, the large on-chip SRAM capacity of the wafer-scale architecture allows model weights and the KV cache to remain

Hardware Layer Analysis. Figures 15 and 16 show that the GPU–WSE hardware ratio is closely related to workload demand. Therefore, when designing such heterogeneous systems, the hardware configuration should be selected to 10

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity

Table 3. Lower bound of average KV-cache bandwidth under steady-state prefill throughput. Model Size

Device

Node

7B 70B

6.55 GB/s 26.2 GB/s 209.6 GB/s 4.6 GB/s 18.8 GB/s 150.4 GB/s

Takeaway 4: Architectural heterogeneity pays off only when workload structure provides sufficient flexibility to either absorb architectural asymmetry or align heterogeneous tasks with architecture-specific strengths. The system stack determines whether this potential complementarity can be translated into end-to-end performance gains.

Cluster

7

Building on the three case studies, we develop a lightweight performance model that captures how workload partitioning, architectural differences, and arithmetic intensity jointly shape GPU–WSE heterogeneous execution.

cover as broad a range of workloads as possible, allowing more workloads to exploit the architectural complementarity between the two platforms. In addition, we model the bandwidth demand based on prefill and use the bandwidth required at peak prefill throughput as the minimum communication bandwidth needed to sustain steady-state execution. Table 3 reports the bandwidth requirements at the device, node, and cluster levels under our GPU configuration. Overall, compared with DP and PP, P/D disaggregation imposes significantly higher network bandwidth requirements.

7.1

Performance Model

System abstraction. Since all experiments are conducted on combinations of two homogeneous systems, we do not extend our model to a more general multi-subsystem setting. Thus, we consider a heterogeneous system composed of two homogeneous subsystems, denoted as 𝐻 𝐸 = {𝐻 𝐸 1, 𝐻 𝐸 2 }, where the 𝑖-th subsystem is represented as 𝐻 𝐸𝑖 = (𝑃𝑖 , 𝐵𝑖 , 𝐶𝑖 , 𝑀𝑖 ).

Takeaway 3: Under P/D disaggregation, GPU–WSE architectural differences become exploitable complementarity when heterogeneous tasks exhibit distinct arithmetic intensities that align with architecture-specific strengths. Realizing this complementarity requires architecture-aware computing resource provisioning and runtime scheduling.

6

Performance Modeling

Here, 𝑃𝑖 , 𝐵𝑖 , 𝐶𝑖 , and 𝑀𝑖 denote the peak compute capability, memory bandwidth, intra-system communication bandwidth, and memory capacity of subsystem 𝑖, respectively. These parameters collectively characterize the fundamental hardware constraints governing execution within each subsystem, including computational capacity, data access capability, and resource scale. In addition, we use 𝑆 to denote the effective communication bandwidth between the two subsystems, which captures the system-level constraint on cross-subsystem data exchange during coordinated execution.

Discussion

Using the aggregate throughput of independently running GPUs and WSEs as an additive baseline, we observe that DP can readily approach this baseline, PP struggles to reach it, and PD disaggregation can even exceed it. This demonstrates that workload characteristics are the key factor determining whether architectural differences can be effectively exploited. DP provides sufficient flexibility along the data dimension and can therefore absorb performance losses caused by architectural differences by redistributing the workload quantity between the two hardware domains. PP, however, imposes numerous constraints, making it difficult to partition the workload in a way that accommodates architectural differences. PD disaggregation exposes two heterogeneous tasks with different arithmetic intensities, creating opportunities to match workload characteristics with the strengths of each hardware architecture and thereby exploit architectural complementarity. Therefore, future GPU–WSE heterogeneous systems should be designed with the characteristics of their target applications in mind to avoid resource underutilization and performance loss.

Model overview. Given the system hardware parameters, we model the performance of the heterogeneous system as 𝑦 = 𝑓 {𝐻 𝐸,𝑆 } (𝑤).

(1)

Here, 𝐻 𝐸 and 𝑆 denote fixed system parameters. The workload 𝑤 is treated as an independent variable and is partitioned into two components, 𝑤 1 and 𝑤 2 , mapped to different subsystems. Model Details. The model details are given in Eq. (2)– Eq. (5). The formulation consists of three components: the end-to-end performance function, the subsystem-level performance model, and the utilization degradation model. ! 2 ∑︁ 𝑓 = min 𝑔𝑖 , 𝑣 𝐼 (𝑤) 𝑆 , (2) 𝑖=1

𝑔𝑖 = min 𝑢𝑃𝑖 𝐻 𝐸𝑃𝑖 , 𝑢𝐵𝑖 𝑤 𝐴𝐼𝑖 𝐻 𝐸𝑖𝐵 , 𝑢𝐿𝑖 𝑤𝐶𝐼𝑖 𝐻 𝐸𝐿𝑖 , 11

(3)

Hu et al.

𝑢 HO,𝑖 (𝑤𝑖 ) 𝑢𝑖 = , 1 + 𝜂𝑖 (𝑤 1 : 𝑤 2 )

𝑖

𝜂 (𝑤 1 : 𝑤 2 ) =

4 ∑︁

𝑢 HO,𝑖 (𝑤𝑖 ) ∝ 𝑤𝑖 ,

(4)

𝐻 𝐸 1𝑗 − 𝐻 𝐸 2𝑗  , max 𝐻 𝐸 1𝑗 , 𝐻 𝐸 2𝑗

(5)

𝛼 𝑖𝑗 (𝑤 1 : 𝑤 2 )

𝑗=1

From Eq. (2), we have 𝑓 ≈ 𝑔1max + 𝑔2max = 𝑓 max . At this point, the system throughput reaches a constant value. Beyond validation, we further evaluate the model’s predictive capability under unseen settings by changing both the dataset and the numbers of GPUs and WSEs. These configurations are not used during model calibration, and we apply the model directly without re-calibration. Despite these changes, the model consistently captures the performance trends of heterogeneous DP, achieving a median prediction error of 6.8% and a 90th-percentile error of 13.4%.

𝑖 ∈ {1, 2}. Eq. (2) models the overall performance of heterogeneous execution as the minimum between aggregated subsystem computation and inter-subsystem communication. The term 𝑔𝑖 denotes the effective performance of subsystem 𝑖, while 𝑣𝐼 (𝑤)𝑆 captures the communication limitation. Eq. (3) specifies 𝑔𝑖 using a Roofline-style model, where performance is bounded by the dominant bottleneck among compute, memory bandwidth, and communication bandwidth. The workload characteristics are described by the memory access intensity 𝑤 𝐴𝐼𝑖 and the communication intensity 𝑤𝐶𝐼𝑖 . Building on this, Eq. (4) introduces the utilization vector 𝑢 = (𝑢𝑃 , 𝑢𝐵 , 𝑢𝐿 ), which characterizes the utilization of system resources. In heterogeneous settings, resource asymmetry leads to reduced achievable utilization compared to the homogeneous case 𝑢 𝐻𝑂 . This degradation is quantified by Eq. (5), where 𝜂 (𝑤) captures the degree of asymmetry across resource dimensions and its impact on performance, with 𝛼 𝑗 (𝑤) representing the penalty coefficient to imbalance in the 𝑗-th resource dimension. 7.2

PD Disaggregation. As shown in Fig.13, assigning prefill and decode to different subsystems improves the aggregate throughput. We now use our model to explain this behavior and examine whether it is consistent with the model prediction. Prefill and decode have different arithmetic intensities 𝑤 𝐴𝐼 : prefill has a higher arithmetic intensity, whereas decode has a lower arithmetic intensity, causing different limiting terms in Eq. (3) to become active. On the GPU, decode is bandwidth-bound because the bandwidth term 𝑢𝐵𝑖 𝑤 𝐴𝐼𝑖 𝐻 𝐸𝑖𝐵 in Eq. (3) is smaller than the compute term. Mapping decode to the WSE increases this bandwidth term, thereby raising 𝑔𝑖 toward a higher bound in Eq. (3). Conversely, prefill is compute-bound on the WSE, and mapping it to the GPU increases the compute term 𝑢𝑃𝑖 𝐻 𝐸𝑃𝑖 in Eq. (3), thereby also increasing 𝑔𝑖 . Consequently, the performance 𝑔𝑖 of each Í subsystem increases, leading to a larger 𝑖 𝑔𝑖 and, in turn, higher overall performance 𝑓 in Eq. (2).This model-based analysis explains the observed throughput improvement: better workload–hardware alignment increases the effective performance 𝑔𝑖 of each subsystem, thereby improving the aggregate system performance 𝑓 . Beyond validation, we further evaluate the model’s predictive capability using request shapes that are not included in Fig. 17. We use the calibrated model to predict heterogeneous P/D performance under these unseen request shapes and compare the predictions with the corresponding measurements. The model achieves a median prediction error of 8.2% and a 90th-percentile error of 16.4%, indicating that it can reasonably capture performance variation across previously unseen request compositions.

Model Validation

Due to space constraints, we select two representative experiments for model validation. Data Parallel. As shown in Fig. 5, as the workload increases, all results exhibit a three-stage, two-threshold performance behavior. We now use our model to explain and characterize this behavior. Ignoring interconnect effects, Eq. (2) shows that the system performance satisfies 𝑓 ≈ 𝑔1 + 𝑔2 . When the workload partition is optimal, Eq. (5) gives 𝛼 𝑗 (𝑤) = 0, which leads to 𝜂 (𝑤) = 0. Substituting into Eq. (4), the utii lization satisfies 𝑢 𝑖 = 𝑢 HO . For small workloads, 𝑤 = (0, 𝑤), resulting in one subsystem being effectively inactive (𝑔 𝑗 ≈ 0). The system is thus dominated by a single subsystem, i.e., 𝑓 ≈ 𝑔𝑖 . As the workload exceeds the first threshold, the performance can be approximated as 𝑓 ≈ 𝑔1 +𝑔2 , where both subsystems become active (𝑔1, 𝑔2 > 0) but remain under-utilized. In this regime, 𝑖 according to Eq. (4), as 𝑤 1 and 𝑤 2 increase, 𝑢 HO increases, which in turn increases 𝑢 𝑖 . This raises 𝑔𝑖 in Eq. (3) and consequently increases 𝑓 in Eq. (2). As the workload increases further, the utilization of each subsystem approaches its maximum, i.e., 𝑢 𝑖 approaches 𝑢 𝑖max in Eq. (4), and the performance becomes bounded by the aggregate subsystem throughput.

8

Related Work

Heterogeneous GPU Systems. Existing studies mainly focus on heterogeneous clusters composed of different generations of GPUs within a single vendor ecosystem. In training, systems such as [21], [9], and [26] incorporate GPUs with diverse capabilities into a unified training pipeline through nonuniform partitioning of computation and model structure, to mitigate imbalances in compute and memory resources. In inference, systems such as [20] model GPU and network resources, and optimize model placement and request routing to improve serving performance on heterogeneous clusters. In addition, a small number of recent efforts begin to explore 12

When Does Heterogeneity Pay Off? Understanding GPU–WSE System Complementarity

multi-vendor GPU heterogeneity. For example, [14] enables efficient communication across GPUs from different vendors, while [5] explores LLM serving across multi-vendor GPU environments. Roofline-based Performance Models for Distributed Systems. Many works extend the Roofline model to modern multidevice settings. [7] generalizes Roofline from single-program analysis to end-to-end workflow analysis, identifying nodelevel, system-level, preprocessing-level, and control-flow bottlenecks in scientific workflows on HPC systems. [2], [4] further extend Roofline to distributed-memory HPC and distributed systems by incorporating communication into the model, thereby jointly characterizing performance bounds imposed by multi-resources. [23] extends roofline-style modeling to CPU+GPU heterogeneous systems, focusing on the performance upper bound under a given workload partition.

9

Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res. 24 (2023), 240:1–240:113. https://jmlr.org/papers/v24/22-1144.html [7] Nan Ding, Brian Austin, Yang Liu, Neil Mehta, Steven Farrell, Johannes P. Blaschke, Leonid Oliker, Hai Ah Nam, Nicholas J. Wright, and Samuel Williams. 2024. A Workflow Roofline Model for End-toEnd Workflow Performance Analysis. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2024, Atlanta, GA, USA, November 17-22, 2024. IEEE, Article 65. doi:10.1109/SC41406.2024.00071 [8] Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. CoRR abs/2404.14527 (2024). arXiv:2404.14527 doi:10.48550/ARXIV. 2404.14527 [9] Runsheng Benson Guo, Utkarsh Anand, Arthur Chen, and Khuzaima Daudjee. 2025. Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models. In Proceedings of the 39th ACM International Conference on Supercomputing, ICS 2025, Salt Lake City, UT, USA, June 8-11, 2025. ACM, 368–383. doi:10.1145/3721145.3730418 [10] Runsheng Benson Guo, Utkarsh Anand, Khuzaima Daudjee, and Rathijit Sen. 2025. Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters. CoRR abs/2507.10392 (2025). arXiv:2507.10392 doi:10.48550/ARXIV.2507.10392 [11] Xin He, Jiawen Liu, Zhen Xie, Hao Chen, Guoyang Chen, Weifeng Zhang, and Dong Li. 2021. Enabling energy-efficient DNN training on hybrid GPU-FPGA accelerators. In ICS ’21: 2021 International Conference on Supercomputing, Virtual Event, USA, June 14-17, 2021, Huiyang Zhou, Jose Moreira, Frank Mueller, and Yoav Etsion (Eds.). ACM, 227– 241. doi:10.1145/3447818.3460371 [12] Ziyu Hu, Zhiqing Zhong, Weijian Zheng, Zhijing Ye, Xuwei Tan, Xueru Zhang, Zheng Xie, Rajkumar Kettimuthu, and Xiaodong Yu. 2025. DABench-LLM: Standardized and In-Depth Benchmarking of PostMoore Dataflow AI Accelerators for LLMs. In IEEE International Symposium on Workload Characterization, IISWC 2025, Irvine, CA, USA, October 12-14, 2025. IEEE, 127–141. doi:10.1109/IISWC66894.2025.00020 [13] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. 2018. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. CoRR abs/1811.06965 (2018). arXiv:1811.06965 http://arxiv.org/abs/1811. 06965 [14] Heehoon Kim, Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, and Jaejin Lee. 2026. HetCCL: Accelerating LLM Training with Heterogeneous GPUs. CoRR abs/2601.22585 (2026). arXiv:2601.22585 doi:10.48550/ARXIV. 2601.22585 [15] Z. Jonny Kong, Qiang Xu, and Y. Charlie Hu. 2025. PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via Pool-Based Pipeline Parallelism. In Proceedings of the 2025 USENIX Annual Technical Conference, USENIX ATC 2025, Boston, MA, USA, July 7-9, 2025, Deniz Altinbüken and Ryan Stutsman (Eds.). USENIX Association, 679–698. https://www.usenix.org/conference/atc25/ presentation/kong [16] Yunjae Lee, Juntaek Lim, Jehyeon Bang, Eunyeong Cho, Huijong Jeong, Taesu Kim, Hyungjun Kim, Joonhyung Lee, Jinseop Im, Ranggi Hwang, Se Jung Kwon, Dongsoo Lee, and Minsoo Rhu. 2025. Debunking

Conclusion

This work studies heterogeneous GPU–WSE execution across the application, system, and architecture layers. Through DP, PP, and P/D disaggregation, we show when architectural differences become performance obstacles or exploitable complementarity, governed primarily by workload partitionability and task–hardware affinity. We further develop a unified performance model to capture these effects and guide heterogeneous system design and scheduling.

References [1] Walther Carballo-Hernández, Maxime Pelcat, and François Berry. 2021. Why is FPGA-GPU Heterogeneity the Best Option for Embedded Deep Neural Networks? CoRR abs/2102.01343 (2021). arXiv:2102.01343 https://arxiv.org/abs/2102.01343 [2] David Cardwell and Fengguang Song. 2019. An Extended Roofline Model with Communication-Awareness for Distributed-Memory HPC Systems. In Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region, HPC Asia 2019, Guangzhou, China, January 14-16, 2019. ACM, 26–35. doi:10.1145/3293320.3293321 [3] Cerebras Systems. 2026. AWS and Cerebras Collaboration Aims to Set a New Standard for AI Inference Speed and Performance in the Cloud. https://www.cerebras.ai/press-release/awscollaboration. Press release. Accessed: 2026-07-11. [4] Fabio Checconi, Jesmin Jahan Tithi, and Fabrizio Petrini. 2022. Ridgeline: A 2D Roofline Model for Distributed Systems. CoRR abs/2209.01368 (2022). arXiv:2209.01368 doi:10.48550/ARXIV.2209. 01368 [5] Xing Chen, Rong Shi, Lu Zhao, Lingbin Wang, Xiao Jin, Yueqiang Chen, and Hongfeng Sun. 2025. Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs. CoRR abs/2509.17542 (2025). arXiv:2509.17542 doi:10.48550/ ARXIV.2509.17542 [6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, 13

Hu et al.

[26] Yongji Wu, Xueshen Liu, Shuowei Jin, Ceyu Xu, Feng Qian, Z. Morley Mao, Matthew Lentz, Danyang Zhuo, and Ion Stoica. 2025. HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs. CoRR abs/2504.03871 (2025). arXiv:2504.03871 doi:10.48550/ ARXIV.2504.03871 [27] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210. https://www.usenix.org/conference/osdi24/presentation/zhongyinmin

the CUDA Myth Towards GPU-based AI Systems: Evaluation of the Performance and Programmability of Intel’s Gaudi NPU for AI Model Serving. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA 2025, Tokyo, Japan, June 21-25, 2025. ACM, 1760–1776. doi:10.1145/3695053.3731050 [17] Sean Lie. 2023. Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning. Ieee Micro 43, 3 (2023), 18–30. available: https://doi.org/10.1109/MM.2023.3256384. [18] Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, and K. V. Rashmi. 2026. Coral: Cost-Efficient MultiLLM Serving over Heterogeneous Cloud GPUs. CoRR abs/2605.04357 (2026). arXiv:2605.04357 doi:10.48550/ARXIV.2605.04357 [19] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2024. Helix: Distributed Serving of Large Language Models via Max-Flow on Heterogeneous GPUs. CoRR abs/2406.01566 (2024). arXiv:2406.01566 doi:10.48550/ARXIV.2406. 01566 [20] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, Lieven Eeckhout, Georgios Smaragdakis, Kaitai Liang, Adrian Sampson, Martha A. Kim, and Christopher J. Rossbach (Eds.). ACM, 586–602. doi:10.1145/3669940.3707215 [21] Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, and Young-ri Choi. 2020. HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism. In Proceedings of the 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, Ada Gavrilovska and Erez Zadok (Eds.). USENIX Association, 307–321. https://www.usenix.org/conference/atc20/presentation/park [22] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, Christine Cuicchi, Irene Qualters, and William T. Kramer (Eds.). IEEE/ACM, 20. doi:10.1109/SC41405.2020.00024 [23] Li Tang, Xiaobo Sharon Hu, and Richard F. Barrett. 2015. PerDome: a performance model for heterogeneous computing systems. In Proceedings of the Symposium on High Performance Computing, HPC 2015, part of the 2015 Spring Simulation Multiconference, SpringSim ’15, Alexandria, VA, USA, April 12-15, 2015, Layne T. Watson, Josef Weinbub, Masha Sosonkina, and William I. Thacker (Eds.). SCS/ACM, 225–232. http://dl.acm.org/citation.cfm?id=2872627 [24] Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, Yang Liu, Pengcheng Zhang, Kun Qian, Kunling He, Jiaqi Gao, Ennan Zhai, Dennis Cai, and Binzhang Fu. 2025. SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. In 22nd USENIX Symposium on Networked Systems Design and Implementation, NSDI 2025, Theophilus A. Benson and Radhika Niranjan Mysore (Eds.). USENIX Association, 541– 558. https://www.usenix.org/conference/nsdi25/presentation/wangxizheng-simai [25] Elliott Wen, Sean Ma, Ewan D. Tempero, Jens Dietrich, Daniel Xiapu Luo, Jiaxing Shen, Kaiqi Zhao, Bruce Sham, Yousong Song, Jiayi Hua, and Jia Hong. 2025. Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators. CoRR abs/2511.11601 (2025). arXiv:2511.11601 doi:10.48550/ARXIV.2511.11601

14

Related documents

• When Does Heterogeneity Pay Off? Understanding GPU-WSE System Complementarity#1490617
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1412633
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1394205
• Operating System Support for Accelerators as First-Class Compute Elements in Heterogeneous Systems#1455861
• Operating System Support for Accelerators as First-Class Compute Elements in Heterogeneous Systems#1450764