arXiv CSopen access

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators Zhiwen Mo1 , Guoyu Li1 , Hao (Mark) Chen1 , Yu Cheng2 , Zhengju Tang2 , Qianzhou Wang1 , Lei Wang2 , Shuang Liang1 , Lingxiao Ma3 , Xianqi Zhou3 , Yuxiao Guo3 , Wayne Luk1 , Jilong Xue3 , Hongxiang Fan1 1 Imperial College London, 2 Peking University, 3 Tile-AI

Recent breakthroughs in advanced packaging, such as highdensity hybrid bonding [33, 122], backside power delivery [28, 42, 88], and localized thermal management [5, 89], have driven active research towards 3D DRAM-stacked accelerator designs. By vertically integrating DRAM with compute dies using hybrid bonding and fine-pitch through-silicon vias (TSVs) [91], these architectures provide significantly higher bandwidth and near-processor memory capacity, fundamentally mitigating memory bottlenecks versus traditional 2.5D designs. This has also drawn increasing industry attention for large-scale distributed 3D- stacked AI systems [62, 115]. However, the compounded complexity at both the system and hardware levels (up to 2.5 × 1014 configurations) has hindered the development of accurate and efficient DSE frameworks for distributed 3D-stacked AI systems, despite their importance in guiding earlystage design decisions. Unlike modeling a single 3D chip, building an accurate and efficient DSE framework for multi-chip distributed 3D systems introduces several unique challenges:

arXiv:2604.04750v2 [cs.AR] 9 Apr 2026

Abstract Advances in hybrid bonding and packaging have driven growing interest in 3D DRAM-stacked accelerators with higher memory bandwidth and capacity. As LLMs scale to hundreds of billions or trillions of parameters, distributed inference across multiple 3D chips has become essential for future AI serving. With cross-stack co-design being increasingly critical in pushing AI efficiency, we propose DeepStack, an accurate and efficient performance model and tool to enable early-stage system–hardware co-design space exploration (DSE) for distributed 3D-stacked AI systems. At the hardware level, DeepStack captures fine-grained 3D memory semantics, such as transaction-aware bandwidth, bank activation constraints, buffering limitations, and thermal–power modeling. At the system level, DeepStack incorporates comprehensive parallelization strategies and execution scheduling for distributed LLM inference. With novel modeling techniques such as dual-stage network abstraction and tile-level compute–communication overlap, we achieve up to 100,000× faster runtime over state-of-the-art simulators at comparable accuracy, cross-validated the modeling accuracy against our in-house 3D designs, NS-3 backend (2.12%), and vLLM serving on 8×B200 GPUs (12.18%). Together with hierarchical design space search, DeepStack enables efficient exploration over ∼2.5 × 1014 design points spanning 3D-stacked DRAM layers, DRAM vertical connectivity, interconnect, compute-memory allocation, and distributed scheduling. Compared with baseline design spaces, DeepStack achieves up to 9.5× higher throughput through co-optimized parallelism and 3D architecture search. Beyond modeling and DSE, we demonstrate that DeepStack can be leveraged to derive design implications for distributed 3D AI systems, guiding performance optimization across the stack. We intend to open source DeepStack to support future research.

• Chip-level accurate modeling with unique characteristics of 3D-stacked hardware. 3D-stacked DRAM provides significantly higher bandwidth, but fully utilizing it demands proportionally larger on-chip buffering (Little’s Law [21]) and careful handling of bank-level access semantics such as large transaction sizes and high concurrency requirements (Sec. 2.3.1). Vertical stacking also exacerbates thermal dissipation challenges, introducing powerconstrained regimes that must be co-modeled with performance. • System-level complicated multi-dimensional parallelism and communication modeling. Distributed LLM inference integrates multiple parallelism strategies, including tensor (TP), pipeline (PP), data (DP), expert (EP), sequence (SP), context (CP), and fully sharded data parallelism (FSDP). Each strategy imposes different compute, memory, and communication demands (Sec. 2.3.2). Existing frameworks rely on simplified linear models [44, 117] or restrict to TP/PP/DP [83, 105], failing to explore the comprehensive search space. • Vast system–hardware co-design space. The interaction between 3D hardware characteristics, thermal constraints, and distributed execution strategies creates a large co-design space spanning per-chip architecture, interconnect topology, parallelism strategy, and execution scheduling (Sec. 2.3.3). Optimizing across this space—rather than treating chip- and system-level decisions in isolation—is essential for informed architectural choices.

Introduction

The scaling laws of LLMs [31, 43] have pushed model sizes beyond a trillion parameters [3, 99]. This growth, combined with agentic tasks and increasingly long context windows [59], drives memory demand and bandwidth requirements far beyond traditional single 2D accelerators, especially under batch-serving workloads in cloud deployments. As a result, distributed LLM inference across multiple accelerators and nodes has become a fundamental requirement [47, 77, 85, 118]. Emerging trends such as test-time compute scaling [10, 11, 60, 95] and agentic memory design [114] further intensify pressure on memory subsystems.

To address these challenges, we propose DeepStack, a full-stack modeling and DSE framework for distributed 3D-stacked accelerators. The key distinctions of DeepStack over prior work are summarized in Table 1. At the chip level, DeepStack adopts fine-grained

Emails: {zhiwen.mo25, hongxiang.fan}@imperial.ac.uk 1

Zhiwen Mo et al.

Table 1: Comparison and novelty over prior work.

bank-level memory modeling that accounts for data layout, bank activation, traffic distribution, bank conflicts, and transaction-aware bandwidth efficiency. At the system level, DeepStack introduces a dual-stage network abstraction combining traffic matrix construction with physical topology mapping and routing, enabling accurate modeling of both on-chip and off-chip interconnects. This is enhanced by tile-level compute–communication overlap modeling and thermal–power co-modeling that prunes infeasible configurations early. Built upon these capabilities, DeepStack supports comprehensive parallelism search across all seven strategies and enables architects to systematically navigate the co-design space to derive quantitative design guidance (Sec. 5.5) for diverse deployment scenarios. Cross-validated against Cadence Palladium cycle-accurate emulation of our in-house 3D chip and production vLLM serving on 8×B200 GPUs (12.18% Mean Absolute Percentage Error, or MAPE), DeepStack achieves consistently low error rates across a wide range of workloads. Our contributions are summarized as follows: • A full-stack modeling and DSE framework that captures chiplevel 3D-memory semantics, including transaction-size-dependent bandwidth, bank-activation constraints, and buffering limits, which integrate with system-level parallelization and execution scheduling. Furthermore, it incorporates thermal–power modeling for early-stage design feasibility assessment. • Novel dual-stage network abstraction and tile-level compute– communication overlap that achieve up to 100,000× speedup over state-of-the-art simulators. Cross-validation against Cadence Palladium cycle-accurate emulation and production vLLM serving on 8×B200 (12.18% MAPE, §5.2.1) demonstrates modeling fidelity across the full stack. • Comprehensive DSE across ∼2.5 × 1014 configurations identifies designs that achieve up to 9.5× higher throughput over baseline 3D design spaces via comprehensive system and hardware codesign. • Beyond modeling and DSE, DeepStack can be leveraged to provide design guidance (§5), such as that batch size in distributed settings is a more fundamental architectural driver than the prefill/decode distinction, energy-efficient and throughput-optimized designs require fundamentally different 3D architectures, and incomplete parallelism search misleads distributed 3D design.

ASTRA- LLMH2- StraOurs STCO Helios sim v2 Compass LLM tum Deep[91] [52] [105] [113] [51] [74] Stack

Feature Fine-Grained 3D Modeling 1

✗

Collectives Auto Search2

✗

✓

Thermal & Power Distributed Inference: Comprehensive and Flexible4 Parallelism

✓: Full support. #: Partial. ✗: Not supported.

✗

✓

✗

✓

✗

✓

✗

✓e

✗

1 Transaction-aware BW, bank activation, Little’s Law, bank conflicts. Stratum models tier-dependent

activation latency across monolithic 3D layers; Helios models NUMA-aware PE-bank mapping with per-bank traffic effects. Neither models transaction-size-dependent BW or Little’s Law buffering. 2 Auto-tuned collective algorithm per topology and message size. 3 ASTRA-sim supports multiple algorithms but user-specified, not auto-searched.

4 Flexible per-module parallelism, e.g., TP in Attention, EP in MoE. b TP/PP. c TP/EP. d TP/PP/EP. e TP/EP/SP/CP/DP/PP/FSDP.

Parallelism: a TP/PP/DP/FSDP.

DRAM Die

TSV Cu-Cu Hybrid Bonding

Compute Logic Die Base Die PHY

PHY

Ctrl.

DRAM Die Ctrl.

Ctrl.

PHY

Ctrl.

µbumps

(a) 2.5D Architecture

Compute Die Interposer

C4 bumps

Package balls

Logic Base Die

Package Substrate (b) 3D Architecture

Figure 1: 2.5D hardware and 3D-stacked architecture.

2.2

3D-Stacked DRAM Architectures

In conventional 2.5D designs (Fig. 1 (a)), compute dies connect to stacked DRAM through an interposer, but bandwidth is constrained by off-stack I/O, and PHY macros occupy tens of 𝑚𝑚 2 [74]. As shown in Fig. 1 (b), recent packaging advances enable 3D-stacked accelerators that place DRAM dies directly atop logic dies via TSVs and hybrid bonding, providing high-bandwidth, high-capacity memory close to compute—crucial for LLM decoding. Wafer-scale 3.5D systems [61] further connect multiple stacks via an interposer for scalable AI.

2 Background and Motivation 2.1 Distributed LLM Inference

2.3

Motivation

2.3.1

Fine-Grained 3D Hardware Modeling.

As LLMs scale to hundreds of billions or trillions of parameters [1, 23, 25, 109], their footprints exceed single-chip capacity even with stacked DRAM, making large-scale multi-chip distributed inference not merely beneficial but essential for serving today’s LLMs. A core challenge lies in optimizing parallelism strategies and collective communication patterns, which directly affect hardware utilization. LLM inference consists of two stages: prefill, which processes the prompt and builds the key–value (KV) cache, and decoding, which generates tokens sequentially using the KV cache. Three metrics capture serving performance: TTFT (Time to First Token, response latency), UTPS (User Tokens Per Second, user-perceived throughput), and STPS (System Tokens Per Second, raw serving capacity including speculative or padding tokens).

Little’s Law Consideration for 3D-Stacked Accelerators. The traditional allocation of area to compute units, on-chip buffers, and network on chip (NoC) resources must be re-evaluated for 3D DRAM-stacked systems. One of the critical constraints is cache and scratchpad sizing: matching the substantially higher DRAM bandwidth with on-chip storage would require prohibitively large area. This is closely tied to Little’s Law [21], where the effective bandwidth achievable is bounded by buffer capacity: Effective Bandwidth ≤ Buffer Size / Latency

(1)

For instance, the NVIDIA B200 GPU [20] already requires over 40 KiB of shared memory per Thread Block to reach 90% of DDR bandwidth utilization. However, most prior work [52, 74, 91, 113] 2

150 100

Bank Charging

Latency 0

1000 2000 3000 4000 5000 6000 Transaction Size (Bytes)

60 40

Bank Charging

Bandwidth Utilization 0

P99

200 100

Utilization Upper Bound

P95

200

100

# of Parallel Strategies

Latency(Cycles)

250

Bandwidth Utilization (%)

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

20 0

4 6 STPS (K token/s)

1000 2000 3000 4000 5000 6000 Transaction Size (Bytes)

Figure 3: The number of distinct parallelisms achieved at a certain STPS, showing a wide performance distribution.

Figure 2: Latency and Bandwidth Utilization of 3D-Stacked DRAM (TSV, No Interleaving).

5× (§5.6). An incomplete search space can mislead the DSE into incorrect chip designs irrecoverable after fabrication.

failed to consider this issue, making their modeling questionable for extremely high-bandwidth 3D memory modeling.

2.3.3

Unique Bank-Level Access Semantics. Accurate memory traffic modeling and bandwidth utilization, especially in a distributed 3D system, are dominated by two factors. First, the most efficient pattern is to stream a full bank row (e.g., ∼2 KiB). As shown in Fig. 2, partial accesses fail to amortize activation and precharge overheads, leading to lower effective bandwidth. Although techniques such as customized DRAM arrays or sub-row activation can mitigate this [72], they introduce additional design complexity and area/power overhead. Second, stacked DRAM exposes a large number of independently connected banks (tens to hundreds). Achieving peak bandwidth requires concurrent streaming across many banks, so memory traffic smaller than a few hundred kilobytes generally cannot saturate TSV bandwidth. Consequently, when per-bank traffic is small, e.g., under finegrained tiling or aggressive parallel sharding, achieved bandwidth drops sharply. However, most previous modeling works [51, 91, 113] assume idealized bank interleaving or uniform streaming and thus overestimate achievable bandwidth in practical distributed LLM serving.

Vast Co-Design Space and Thermal Constraints.

System–Hardware Co-Design. A comprehensive system and hardware co-design requires exploring broad design choices: • Per-chip architectural configuration: compute units, on-chip buffering, and memory hierarchy. • On-chip and inter-chip network topology: bandwidth provisioning and latency/area trade-offs. • Distributed parallelism planning: selection and mapping of TP/PP/DP/EP/SP/CP to physical devices. • Single-chip scheduling: tile size, kernel fusion, memory placement, and scheduling pipelining. • Collective communication strategies: choice of algorithms (e.g., all-reduce versus reduce-scatter+all-gather) and fine-grained compute–communication overlap. More critically, these dimensions are tightly coupled: parallelism strategies determine per-chip memory traffic, which in turn dictates buffer sizing (via Little’s Law) and optimal DRAM stacking depth. Yet prior approaches typically optimize them independently or explore only a subset. As shown in §5.6, our joint DRAM-layer and NoC co-optimization contributes 33–39% throughput improvement over a fixed configuration.

Modeling of 3D-Stacked Connectivity. To maximize DRAM bandwidth, 3D-stacked accelerators can connect each bank directly to the logic die via TSVs, bypassing traditional multi-bank interleaving used to hide activation and turnaround latency [27, 38, 87]. However, direct bank access is only one point in the design space: practical systems may also instead employ multi-bank interleaving or connect only a subset of the stacked layers. To facilitate the exploration of this design choice, DeepStack supports both directaccess and interleaved designs, where we parameterize both modes and investigate their impact in § 5.5.3.

Thermal Consideration. The thermal issue further compounds the co-design space: additional DRAM layers increase thermal resistance, and as we demonstrate in §5.5, many high-bandwidth configurations exceed the 85◦ C thermal limit under sustained decode workloads. Without thermal-aware modeling, architects risk committing to configurations that appear optimal in performance models but are thermally infeasible in practice.

3 DeepStack Framework 3.1 DeepStack Overview

2.3.2 Compound Parallelism Strategies. Distributed LLM serving combines multiple parallelism strategies, including TP, PP, DP, EP, SP, CP, and FSDP [47, 98, 118]. Each strategy leads to different per-device compute, memory, and communication characteristics. As a result, the optimal parallelism combination usually varies from model to model. As shown in Fig. 3, performance varies significantly across different parallelism configurations for a given model, and high-performance strategies are rare. However, existing frameworks typically use simplified cost models [44, 105, 117] or restrict parallelism to TP/PP/DP [83, 105], without accurate modeling of compute–communication overlap at tile granularity. For instance, limited parallelism strategies force mixture of experts (MoE) models to shard experts via large TP, reducing throughput by up to

Fig. 4 illustrates an overview of DeepStack. It takes a model computation graph supporting diverse LLM building blocks, along with batch size, sequence lengths, and decoding modes. On the hardware side, it consumes candidate configurations of compute-die parameters (buffer sizes, compute units) and NoC settings (hierarchical topologies with bandwidth and latency). All hardware configurations are first validated through an area cost model (Sec. 4.1.1) given a per-die area budget. For feasible hardware, a parallel-strategy composer enumerates configurations across TP/EP/SP/CP/DP/FSDP/PP. An empirical filter removes invalid or non-beneficial options (e.g., SP for single-step 3

Zhiwen Mo et al.

Input

Modeling

Scheduling & Search Parallel Composer

LLM Graphs Chip Arch 3D DRAM Spec.

Memory Footprint Checker

Area Model

Inter Connect Topologies

Auto-Tuner (Collectives Algo.)

Stage 1: Traffic Matrix

Deduced Communications

Tile-Based Overlap Modeling

Report

Stage 2: Routing to Physical

Statistics Traffic, Utilization

Compute

Deduced Op. Shapes

L1 Memory

Auto-Tuner (Tiling, Fusion, Layout, ...)

3D DRAM

Perfomance TTFT, STPS, UTPS

L2 Memory

Power Engergy, Heating

Comm.

M Y

HB Y

PH PH

Y PE P

PE PE

PE X BA

(a)Processing Engine View

Compute Layer

Chips Ethernet

Shared Memory / L1 Cache

Die UCIe Interposer Substrate

Clusters

Logic Base Layer

PE0 Vector Matrix ... Vector Matrix SFU Unit SFU Unit Unit Unit

Inter-Cluster Connection

Scheduler

Figure 4: Overview of DeepStack DSE Framework.

(b) DRAM Stack Cluster View

(d) Chip View

(e) System Overview

Figure 5: Example of Cross Sectional and Top View of a 3D-Stacked DRAM Architecture. decoding, FSDP when DP= 1). A memory-footprint checker then prunes configurations whose model size, KV-cache, and peak activations exceed memory capacity. For each remaining configuration, DeepStack determines perchip operator shapes and collective communication primitives. Two auto-tuners operate at this stage: (i) an operator-level tuner searching tiling, fusion, and layout strategies for wave-level execution, and (ii) a collective tuner selecting the most efficient algorithm for the given topology and data size. Finally, a tile-based compute–communication performance model (Sec. 4.1.5) simulates the full execution timeline while maximizing overlap. It produces a report with TTFT, UTPS, STPS, and utilization statistics across compute, cache, stacked DRAM, and the hierarchical NoC, guiding further architectural refinement.

3.2

layer count (e.g., 4-layer stack), and bank size (typically 2 KiB). DeepStack distinguishes stacked layers (contributing capacity) from connected layers (additionally contributing bandwidth). An example of architectural options is shown in Table 2. (c) Die Level. A die (Fig. 5(c)) contains multiple clusters connected via a configurable Level-1 (L1) network with selectable topology, link bandwidth, and per-hop latency. (d) Chip Level. A chip (Fig. 5(d)) integrates multiple dies via universal chiplet interconnect express (UCIe) links, modeled with a Level-2 (L2) network with configurable topology and link parameters. (e) System Level. Multiple chips are linked via Ethernet (Fig. 5(e)), forming the Level-3 (L3) network with configurable topology and bandwidth. For (c)–(e), DeepStack supports six fundamental topologies: switch, 1D chain/ring, 2D mesh/torus, and fully connected, with hierarchical compositions across L1/L2/L3. Users can specify routing rules and port mappings for hybrid topologies. Example network configurations are in Table 3.

Hardware Model

To ensure the generality of DeepStack, we follow a hierarchical hardware model by generalizing our in-house 3D designs. As shown in Fig. 5, it progresses from a fine-grained processing engine to the full multi-chip system, which consists of: (a) Processing Engine (PE). The PE is the fundamental compute unit, analogous to a Streaming Multiprocessor of GPUs. Each PE (Fig. 5(a)) contains a configurable mix of Special Function Units (SFUs), Vector units, and Matrix units, along with an L0 register file and L1 shared memory with configurable capacity and bandwidth. Users can also specify the minimum matrix tile shape. (b) 3D-Stacked DRAM Cluster. As shown in Fig. 5(b), a cluster consists of multiple PEs, an L2 cache, local interconnect, and vertically stacked DRAM layers bonded over a logic base die. Configurable parameters include PE count, L2 size/bandwidth, DRAM

3.3

System Scheduling Configuration

System performance is highly sensitive to scheduling, especially in distributed settings. Therefore, DeepStack incorporates comprehensive distributed parallelisms and collective communication choices into the design space. 3.3.1 Parallel Strategy. DeepStack supports a wide range of parallelism combinations, including TP [93], EP [24, 50], SP [55], CP [70], DP [45], FSDP [116], and PP [35, 66]. Following the common practice from Megatron [93] and DeepSpeed [84]: TP splits attention heads and FFN/MoE intermediate dimensions, EP partitions routed experts (defaulting to DP for non-MoE layers), SP partitions the sequence dimension, mainly in prefill, CP splits KV-cache across 4

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

devices for both prefill and decode, PP partitions layers with microbatching for pipeline efficiency, DP splits the batch, and FSDP further shards weights on top of DP. Given a target parallelism degree 𝑁 , the composer enumerates all valid, non-duplicate factorizations: 𝑇 𝑃 × 𝐸𝑃 × 𝑆𝑃 × 𝐶𝑃 × 𝐷𝑃 × 𝑃𝑃 = 𝑁 .

Power Model. DeepStack collects per-operator memory traffic (register, shared memory, L2, DRAM), NoC traffic, and compute operations (vector, SFU, matrix), then multiplies by technologycalibrated per-bit energy coefficients for memory [32] and NoC [53, 65, 76], as well as per-operation energy for compute [63, 72, 107]. Static power is 10% of Thermal Design Power (TDP) [72].

(2)

Thermal Model. Junction temperature follows 𝑇 = 𝑇amb +𝑅(𝑚) · 𝑃. We anchor at a 4-layer, 100 W baseline with 6000 W/m2 K convection, and sweep 𝑚 ∈ [1, 12] in ANSYS Fluent [49] to fit a linear model: 𝑅(𝑚) = 0.56 + 0.01 · 𝑚 (°C/W).

Candidates are enumerated by prime-factorizing 𝑁 and applying stars-and-bars allocation across six dimensions. Since FSDP is optional with DP, each candidate can be set with FSDP=True or False. 3.3.2 Single Chip Scheduling. Single-chip scheduling has been extensively explored in prior compilers and frameworks [13, 14, 69, 73, 92, 101, 102, 123]. DeepStack incorporates scheduling spaces, including tiling, fusion, and software pipelining to hide memory latency. For instance, in matrix multiplication, it searches over tile sizes across memory hierarchies, the number of software-pipeline stages to overlap computation with memory access, and fusing lightweight epilogue operations such as bias addition. Beyond classical optimizations, DeepStack adds constraints from 3D-stacked DRAM: directly connected banks without interleaving can cause wasted accesses or poor row locality. We incorporate penalty terms for strategies that lower effective bank utilization or cause bank conflicts due to layout misalignment.

Dynamic Voltage and Frequency Scaling (DVFS).. More DRAM layers raise 𝑅(𝑚), reducing sustainable power. Since 𝑃dyn ∝ 𝑉 2 𝑓 and 𝑉 ∝ 𝑓 [32], we have 𝑃dyn ∝ 𝑓 3 . The thermal-limited frequency scales as 𝑓th = (𝑃 dyn (𝑚)/𝑃dyn (4)) 1/3 , where 𝑃dyn (𝑚) = TDP × 𝑅(4)/𝑅(𝑚) − 𝑃static . A second power-wall check further reduces frequency if actual per-device power exceeds TDP. 4.1.3 Single-Die Performance Modeling. Our single-die model follows a tile-level abstraction inspired by prior frameworks [34, 46, 48, 64, 75, 120], which have shown tile-level simulation is sufficiently accurate for non-3D architectures. To incorporate the unique behaviors of 3D-stacked DRAM, we identify four key features: (i) Transaction-size–dependent bandwidth. Without multibank interleaving, peak bandwidth requires reading a whole bank row per access. Achievable bandwidth decreases with smaller transactions. (ii) Constrained buffering imposed by Little’s Law. Saturating TSV bandwidth requires sufficient buffering (buffer ≥ bandwidth × latency). Insufficient buffering directly reduces throughput. (iii) Bank parallelism limited by operator size. With directly attached banks, small operators may not issue enough requests to activate all banks, reducing effective bandwidth. (iv) Bank conflicts from layout and access patterns. Without interleaving, certain tensor layouts may map many tiles to the same bank, introducing conflict-induced serialization.

3.3.3 Collectives Implementation. Once a parallel strategy is fixed, the required communication collectives for tensor reconstruction are determined. However, each collective operation admits multiple algorithmic implementations, whose performance varies widely depending on the underlying network topology, link bandwidth, and hop latency. For instance, all-reduce can be implemented using ring, Rabenseifner, double-tree, and double-recursive halvingand-doubling algorithms. Our DeepStack evaluates mainstream candidate algorithms for each collective, and the modeling stage selects the highest-performing option based on topology-aware communication costs and potential computation overlap.

4 Modeling and Design Space Exploration 4.1 Modeling Methodology

To model these efficiently, DeepStack applies the following rules. For (i), we obtain achievable bandwidth from the bandwidth–transactionsize curve in Fig. 2, derived from the 3D-DRAM vendor’s C-model. For each tile access, our method computes the effective transaction size and queries this curve. For (ii), bandwidth is further bounded by Little’s Law:

4.1.1 Area Modeling. We synthesized our baseline design ("Standard" in Table 2, "TMS1" in Table 3) using the ASAP7 7 nm predictive PDK [17] and extracted post-synthesis area figures as reference anchors. During exploration, compute-unit areas are linearly scaled by throughput [51, 113]. On-chip SRAM areas are decomposed into capacity-proportional and bandwidth-proportional parts, calibrated from Memory Compiler-generated macros [74]. Memory controller area scales with peak DRAM bandwidth based on [108], with additional DRAM peripheral overhead derived from published logic-die breakdowns [51, 74]. On-chip interconnect area is scaled from the baseline with sub-linear bandwidth elasticity. Total die area includes a 15% overhead for control and routing, observed from our baseline synthesis. Designs exceeding the area budget are pruned before performance evaluation.

BWeff = min(BW, buffer / latency)

(3)

For (iii) and (iv), DeepStack explicitly maps each tensor tile to a DRAM bank according to the chosen layout and swizzling policy. For every memory wave, we build a per-bank access histogram. The most heavily loaded bank determines the wave’s service time. This unified mechanism simultaneously models (a) limited bank-level parallelism for small operators and (b) bank conflicts caused by less efficient data layouts. Based on these, DeepStack preserves the efficiency of tile-level modeling while capturing the core performanceshaping behaviors unique to 3D-stacked DRAM architectures.

4.1.2 Power and Thermal Modeling. In 3D stacking, overheating increases leakage and can render the device non-functional. Typically, 95 °C is the hard limit [91] and 85 °C is a safer target [52, 111, 112].

4.1.4 On-Chip/Off-Chip Network Modeling. The primary goal of our network modeling is to efficiently estimate latency and link 5

Zhiwen Mo et al.

𝑉𝑙 𝑇𝑛𝑒𝑡 = max (Hops𝑝 × 𝛿ℎ𝑜𝑝 ) + max , 𝑝∈P 𝑙 ∈ L 𝐵𝑊𝑙 | {z } | {z } 𝐿𝑛𝑒𝑡

40 30 20 10 0

20 40 Destination

Link Bandwidth 70 60 50 40 30 20 10 0

40 60 Destination

Link Utilization

7 6 5 4 3 2 1 0

Source

70 60 50 40 30 20 10 0

7 6 5 4 3 2 1 0

(b) Topology bandwidth.

Traffic (bytes)

Source

(a) Traffic Matrix. Link Traffic (bytes) 1e7

1e11 8 Bandwidth (bytes/s)

Traffic (bytes)

Source

1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0

70 60 50 40 30 20 10 0

0.8 0.6

Utilization

Traffic Matrix Heatmap (N=64) 1e6

Source

utilization across diverse topologies. Existing tools such as ASTRAsim-v2 [105] face significant trade-offs: analytical backends can incur up to 58% error, while NS-3 [86] backends require over an hour per GiB-scale collective, making it infeasible for billion-point DSE. To address this challenge, we propose a novel dual-stage network abstraction, which is described as follows: Stage-1: Traffic Matrix Construction. We represent each collective communication step using a Traffic Matrix (TM), a logical num_nodes × num_nodes matrix where entry (𝑠, 𝑑) denotes the data volume sent from source 𝑠 to destination 𝑑. All traffic within a TM starts simultaneously, while complex collectives decompose into TM sequences. Crucially, the TM captures logical patterns independent of physical topology. During construction, DeepStack aggregates traffic from all parallelism dimensions. To reflect realistic optimization, DeepStack prioritizes mapping logically intensive communication pairs (e.g., TP) to physically adjacent nodes. This ensures that high-volume logical edges benefit from higher bandwidth and lower latency in the physical layer. Stage-2: Physical Mapping and Routing. Given a TM, DeepStack maps logical (𝑠, 𝑑) flows to physical routes defined by the target topology. After routing, it accumulates the traffic volume 𝑉𝑙 on each physical link 𝑙 with bandwidth 𝐵𝑊𝑙 . The total network time 𝑇𝑛𝑒𝑡 is modeled as the sum of the structural hop latency and the bottleneck serialization delay:

0.4 0.2 0

0.0

40 60 Destination

(d) Final utilization.

Figure 6: Example of mapping 64-node logical EP all-to-all traffic onto a three-layer torus–mesh–mesh topology. (a) Logical TM, (b) topology bandwidth, (c) routed physical traffic, and (d) resulting link utilization.

(4)

latency 𝐿𝑛𝑒𝑡 derived in § 4.1.4: 𝑇𝑒2𝑒 = 𝜏𝑐𝑜𝑚𝑝 + (𝜏𝑐𝑜𝑚𝑚 + 𝐿𝑛𝑒𝑡 ) {z } |{z} |

𝑇𝑐𝑜𝑛𝑔

Prologue

+ max (𝑊 −1)𝜏𝑐𝑜𝑚𝑝 , (𝑊 −1)𝜏𝑐𝑜𝑚𝑚 + 𝐿𝑛𝑒𝑡

where P is the set of active paths and 𝛿ℎ𝑜𝑝 is the per-hop delay. The first term 𝐿𝑛𝑒𝑡 is the zero-load hop latency, used as a key parameter in overlap modeling (§ 4.1.5). The second term 𝑇𝑐𝑜𝑛𝑔 captures congestion from the most utilized link. The contention is implicitly modeled here, because links shared by multiple logical flows accumulate a higher 𝑉𝑙 , increasing serialization time. Fig. 6 visualizes this process of mapping the traffic matrix to the final utilization based on physical-topology bandwidth. This two-stage abstraction avoids event-level modeling while still capturing volume-based contention. It reduces evaluation time to under 0.1 seconds for a 256-node topology while maintaining less than 5% error compared to an NS-3 backend (Fig. 10), making billion-scale DSE feasible.

(5)

Epilogue

The term in the second line represents the steady-state pipelined phase. Note that 𝐿𝑛𝑒𝑡 is explicitly included in the communication terms to account for the signal propagation delay that cannot be hidden by bandwidth-bound pipelining. If 𝑊 ≤ 1, the model reduces to a serialized execution: 𝑇𝑐𝑜𝑚𝑝 + 𝑊 · 𝜏𝑐𝑜𝑚𝑚 + 𝐿𝑛𝑒𝑡 . This modeling formulation highlights a critical architectural trade-off between compute efficiency and overlap efficiency. Larger tile sizes generally improve data reuse (higher 𝜏𝑐𝑜𝑚𝑝 efficiency) but result in fewer total waves (𝑊 ). A smaller 𝑊 reduces the depth of the pipeline, making the non-overlappable prologue and epilogue a larger fraction of the total execution time. Conversely, smaller tiles increase 𝑊 , maximizing overlap opportunities, but may degrade arithmetic intensity. As shown in Fig. 7, with low NoC bandwidth (panels a, b), small tiling (a) is preferable to maximize waves and hide communication. However, with high NoC bandwidth (panels c, d), the communication cost keeps low, making large tiling more favorable since it prioritizes compute efficiency over overlap. DeepStack automatically navigates this complex design space to find the optimal configuration.

4.1.5 Compute-Communication Overlap Modeling. To facilitate accurate modeling of distributed LLM inference, DeepStack models compute–communication overlap at the granularity of tile-based wavefronts. As illustrated in Fig. 7, once the computation for a wave of tiles is completed, DeepStack initiates the corresponding data transfer. This allows the communication of the 𝑖-th wave to proceed in parallel with the computation of the (𝑖 + 1)-th wave. Let 𝑊 = ⌈num_tiles/num_SMs⌉ denote the number of waves. We define the per-wave compute time as 𝜏𝑐𝑜𝑚𝑝 and the per-wave transmission time as 𝜏𝑐𝑜𝑚𝑚 . For 𝑊 > 1, the end-to-end latency 𝑇𝑒2𝑒 is modeled as a three-stage pipeline, incorporating the network hop

4.2

DSE Framework

4.2.1 Workloads. DeepStack supports a broad range of LLM architectures by operating directly on their computation graphs. For 6

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

(a) HW. Config. 1, Small Tiling

Weight

Comm.

Weight

T=5t Act

(b) HW. Config. 1, Large Tiling

Wave1

Wave2

Wave3

Wave4

Weight

Compute

T=4.5t Act

(d) HW. Config. 2, Large Tiling

Comm.

Wave1

Wave2

Wave3

Wave4

Weight

Wave1

Wave2

Compute

FP16 FP32 SFU L2BW L1BW WG TFLOPS TFLOPS TFLOPS (TB/s) (TB/s) MMA H100* 504.6 31.5 2.0 9.8 15.8 ✔ H200* 504.6 31.5 2.0 11.8 15.8 ✔ Standard 367.0 11.5 1.4 13.1 22.9 ✘ WGMMA 367.0 11.5 1.4 13.1 22.9 ✔ Weak NoC 412.9 12.9 1.6 13.1 25.8 ✘ Strong NoC 321.1 10.0 1.3 13.1 20.0 ✘ Strong L2 275.3 8.6 1.1 32.0 17.2 ✘ Strong L1 321.1 10.0 1.3 13.1 40.1 ✘ Less SM 321.1 10.0 1.3 13.1 20.1 ✘ Large Matrix 458.8 14.3 1.8 13.1 28.7 ✘ Large Vector 321.1 20.1 2.5 13.1 20.1 ✘

Compute

Comm.

T=5.5t Act

Table 2: Architecture Specifications per 800 mm2

Comm.

T=4t Act

Wave1

Wave2

Compute

Figure 7: Tile-level compute–communication overlap modeling for GEMM. Panels compare Low (a,b) vs. High (c,d) NoC bandwidths using Small (a,c) vs. Large (b,d) tiling strategies. The visualization demonstrates how optimal tiling shifts from overlap-oriented (small tiles) to compute-oriented (large tiles) as network bandwidth increases.

H100* and H200* denote scaling them from 4 nm to 7 nm to decrease SM counts while keeping other aspects unchanged. DRAM configuration: H100*: 3.3 TB/s, 80 GiB HBM3; H200*: 4.8 TB/s, 141 GiB HBM3e; Others (4-layer 3D-stacked): 13.1 TB/s, 64 GiB. WGMMA denotes support for four matrix units sharing input operands. All configurations satisfy the 800 mm2 area constraint.

attention mechanisms, it includes standard Multi-Head Attention (MHA) [100], Grouped-Query Attention (GQA) [4], Multi-Query Attention (MQA) [15], and DeepSeek’s Multi-Head Latent Attention (MLA) with automatic absorbed/non-absorbed form selection. For feed-forward networks (FFNs), both dense SwiGLU layers (e.g., LLaMA family) and Mixture of Experts (MoE) blocks (e.g., DeepSeek, Qwen3) are supported. To ensure accurate MoE communication and load-balance modeling, we extract realistic routing patterns from actual inference traces.

Table 3: NoC Configurations Level 3 Level3 BW Level 2 Level2 BW Level 1 Level1 BW Topo (GB/s) Topo (GB/s) Topo (GB/s) H100* SWITCH SWITCH 200 450 32x 4 8 TORUS MESH SWITCH TMS1 75 384 800 4x4 2x2 4 TORUS MESH SWITCH TMS2 75 768 800 4x4 2x2 4 TORUS MESH SWITCH TMS3 112 576 1200 2x2 4x4 4 TORUS MESH SWITCH 640 TMS4 60 307 2x2 4x4 4 TORUS MESH SWITCH TMS5 75 384 600 2x2 2x2 16 TORUS MESH SWITCH TMS6 100 384 800 4x4 2x2 4 TORUS MESH SWITCH TMS7 75 384 960 4x4 2x2 4 TORUS MESH MESH TMM8 75 384 800 2x2 4x4 2x2

4.2.2 Design Space Size. As summarized in the workflow (§ 3.1), the full design space spans both hardware and software dimensions. At the hardware level, DeepStack explores single-chip architectural configurations, 3D DRAM stacking depth (𝑚=1–16 stacked layers with 𝑛≤𝑚 connected layers), on-chip SRAM capacity/bandwidth settings, on-chip/off-chip network topologies, NoC hop latency, and per-layer NoC bandwidth—all subject to area and thermal constraints. Given a feasible hardware configuration, the system level introduces three additional degrees of freedom: distributed parallel strategies, single-chip operator scheduling, and collectivecommunication algorithm selection. Together, these form a vast joint hardware–system search space. For any given configuration, DeepStack generates an end-to-end performance report including TTFT, UTPS, STPS, as well as detailed utilization statistics for memory, compute, and each NoC layer, along with power and thermal estimates (§ 4.1.2). However, this raw design space is extremely large. Consider a scenario with batch sizes {1, 16, 128, 1024}, six input/output sequence lengths, and 4 model architectures. On the hardware side, we explore 30 base architecture/topology combinations, 136 stacking configurations (𝑚=1–16, 𝑛≤𝑚), 3 SRAM capacity/bandwidth settings, 5 NoC hop-latency multipliers, and per-layer NoC bandwidth scaling across the three-level hierarchy (43 =64 combinations). With 256 devices, the software space comprises 2,574 parallel strategies, 4 collective algorithms, and ∼64 single-chip schedules (6.6 × 105 per setting). The total theoretical design space reaches ∼2.5 × 1014 configurations.

*In this paper, we use T for TORUS, M for MESH, and S for SWITCH, and arrange them according to an L3–L2–L1 hierarchy. * The H100*32× topology is expressed using an 800mm2 die as the minimum unit, whereas all other topologies use a 100mm2 3D-stack cluster as the basic building block.

𝑚>10 are eliminated, as Little’s law and L1 bandwidth constraints cause effective DRAM bandwidth to decrease beyond this point despite increasing theoretical bandwidth (§5.5), making them strictly dominated on both bandwidth and compute FLOPS. (2) Parallelstrategy pruning: Invalid or dominated strategies are removed (e.g., 𝑆𝑃 >1 in single-step decoding, 𝐹𝑆𝐷𝑃=True with 𝐷𝑃=1, infeasible 𝐷𝑃/𝑃𝑃 given batch size), eliminating ∼80% of candidates. (3) Memoryfootprint check: Model weights, KV cache, and peak activations must fit in DRAM (with 10% reserved for CUDA graphs and fragmentation), pruning ∼50% of remaining configurations. (4) Hierarchical NoC search: Rather than exhaustively enumerating the 5×64=320-point NoC latency/bandwidth grid, we adopt a hierarchical approach. DeepStack first searches over base architecture and stacking configurations, then advances only the top 5% into the NoC latency sweep, and finally promotes the top 5% of that tier into per-layer bandwidth tuning. Together, these strategies reduce the effective search cost to ≈2 days on a 512-core CPU.

4.2.3 Design Space Pruning and Hierarchical Search. To enable efficient search on the vast design space (up to ∼2.5 × 1014 ), we apply multi-stage pruning to make this space tractable. (1) Paretodominated hardware pruning: Configurations where stacked layers 7

Zhiwen Mo et al.

Torus

(e) Reduce-Scatter GEMM

Switch

2 1

Decode TPS

(a) Qwen3 235B

(b) Llama3 70B

10000

Batch Size

Decode TPS

Batch Size

5.2

(d) DeepSeek V3

105

107

103 103

105

107

107 105

105

103

103 105

107

108

104

106

108

104

106

108

106 104 103 105 107 ASTRA-sim NS3 time (ns)

Modeling Accuracy

5.2.1 End-to-End LLM Serving Modeling Accuracy. We crossvalidate our model against in-house 3D designs using Cadence Palladium [8] cycle-accurate emulation (e.g., M3584×N3584×K4096), which achieves less than 5% error relative to the reference. Because commercial 3D-stacked DRAM systems remain limited, we additionally validate against 8×H100 and 8×B200 NVLink systems to further demonstrate the accuracy of our compute and NoC modeling. On H100 (Fig. 8) using Triton-Distributed kernels from LLM Attention and MoE blocks, DeepStack achieves 3.97% average error for All-Gather GEMM and 8.4% weighted error across all kernels. On B200 (Fig. 9), end-to-end validation with vLLM [47] (CUDA Graphs, TP8/EP8) yields a MAPE of 12.18% across models, batch sizes, and parallelism strategies; residual gaps stem from implementation details (e.g., FlashMLA dynamic KV-splitting) outside our analytical scope.

5000

Batch Size

103

105

8 16 32 64 128 256 512

10000

0 1 2 4 8 16 32 64 128 256 512

103

H100*/H200* scaled to 7 nm using 800 mm2 units, with a 1.5× downscaling factor applied to the 4 nm H100 (reducing throughput to 504.6 TFLOPS) based on transistor density differences [18, 19]. NoC Configuration. To evaluate the impact of interconnect design, we explore a range of NoC configurations that vary in bandwidth and topology, as summarized in Table 3.

Ground Truth (vLLM on 8xB200)

20000

0 1 2 4 8 16 32 64 128 256 512

105

103

Figure 10: DeepStack modeling accuracy vs. ASTRA-sim (NS-3 and analytical backends) across Switch and Torus topologies. DeepStack matches NS-3 fidelity with up to ∼ 105 × speedup.

Figure 8: DeepStack modeling accuracy compared to 8×H100 GPUs. The x-axis enumerates Triton-distributed kernels with different shapes. DeepStack (Modeling for 8xB200)

ASTRA-sim analytical All-Reduce 107

107

103

DeepStack All-To-All

105

107 Time (ns)

107

(d) All-Gather GEMM Time (ms)

Time (ms)

2 Time (ms)

Time (ms)

ASTRA-sim NS3 All-Gather

Ground Truth (8×H100) Time (ns)

DeepStack (Modeling)

(a) All-Reduce GEMM (b) MoE EP All-To-All

8 16 32 64 128 256 512

Batch Size

Figure 9: DeepStack modeling accuracy on vLLM TP8 and EP8 for MoE on an 8×B200 GPU system.

5 Evaluation and Design Implications 5.1 Experimental Setup Workloads & Metrics. We evaluate DeepStack using Llama-3.370B/405B [23] (A16W16, A/W denote activation/weight), DeepSeekR1/V3 [25, 57] (A16W8), and Qwen3-235B [109] (A16W16), with batch sizes ranging from 1 to 1024. Sequence lengths follow [90], with both input and output lengths set to 1024. Following [22], we report TTFT for prefill, and UTPS/STPS for both prefill and decoding. Platforms. Real-hardware experiments are conducted on an H100-SXM cluster [19] and a B200 cluster [20], both equipped with AMD EPYC or Intel Xeon hosts. We use Triton-Distributed [119] for multi-GPU kernel implementation, vLLM v0.15.1 [47] with CUDA 13.0 for multi-GPU LLM inference, and ASTRA-Sim v2.0 [105] for baseline NoC simulation. 3D Memory Configuration. By default, the system adopts four 3D-stacked DRAM layers with direct connectivity to the compute die. § 5.5 further evaluates the impact of varying the number of DRAM layers and memory connectivity. Compute Configuration. All designs are normalized to 7 nm under a fixed 25,600 mm2 total area budget. Baselines (Table 2) are

5.2.2 Comparison with State-of-the-Art Simulators. To compare with prior work, we validate DeepStack against ASTRA-sim-v2 (NS-3 backend [86]) across bandwidths (32–512 GB/s), latencies (32– 512 ns), topologies (Mesh, Torus, Switch), and collective algorithms. As shown in Fig. 10, DeepStack achieves weighted errors of only 2.12% (Switch) and 1.62% (Torus) versus the NS-3 backend, while their analytical baseline deviates by up to 58%. Crucially, DeepStack delivers up to 100,000× speedup (0.1 s vs. 3 h for GiB-scale collectives), enabling billion-scale DSE at discrete-event-level accuracy.

5.3

System-Hardware Co-Design Exploration

The fast and accurate modeling (§5.2) enables efficient design space exploration across both hardware and system. Fig. 11 presents the performance (STPS & UTPS) under different typical settings (Tables 2 and 3) and workloads (different models & batch sizes), showing only the best-performing designs under each setting. 8

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

DeepSeek V3

Qwen3 235B A22B

Llama3 405B

Llama3 70B

UTPS (token/s)

STPS

BS=1024

BS=256 BS=128

BS= 64/16 /4/1

UTPS (token/s)

System TPS (K tokens/s)

Figure 11: Pareto frontiers from DeepStack’s DSE across design points. A representative subset is shown for clarity.

DeepSeek-V3 Decode DeepStack DSE H100(Scaled) H200(Scaled)

20 0

60 80 User TPS (tokens/s)

Qwen3-235B-A22B Decode

20 0

Llama3 405B Decode

DeepStack DSE H100(Scaled) H200(Scaled)

60 80 100 User TPS (tokens/s)

5 0

120

Llama3 70B Decode

DeepStack DSE H100(Scaled) H200(Scaled)

40 60 User TPS (tokens/s)

DeepStack DSE H100(Scaled) H200(Scaled)

40 20 0

75 100 125 150 175 User TPS (tokens/s)

Figure 12: UTPS/STPS decoding performance comparison. DeepStack DSE denotes the best configuration searched by our framework. 3D-stacked architectures deliver substantial gains in the memory-bound decode phase. Key Takeaway 1: For compute-bound prefill, Large-Matrix configurations dominate the Pareto frontier with overlapping NoC variants. In contrast, small-batch decode is NoC-sensitive, with bandwidth and hierarchy differences causing large STPS variations. Model-specific NoC preferences emerge: Llama favors TMS5 (stronger L1 switching) due to its reliance on TP, while DeepSeek benefits from TMS6 (stronger L3/off-chip bandwidth) driven by largescale EP. Key Takeaway 2: The optimal parallelism strategy shifts dramatically with batch size: TP dominates at small BS, but drops monotonically as BS grows and communication overhead becomes the bottleneck. PP shows the opposite trend, as larger batches amortize the pipeline bubble. For MoE models, EP is increasingly favored at large BS.

5.4

(141 GiB HBM3e vs. 64 GiB 3D-DRAM) offset the bandwidth advantage of 3D designs. Key Takeaway 3: The 3D-stacked design serves as an effective accelerator for the decode phase, achieving up to 2.79× STPS improvement. During compute-bound prefill, its performance remains comparable to that of 2.5D baselines.

5.5

Case Study: 3D DRAM Layer Design Space

Central design questions in 3D-stacked architectures are: i) how many DRAM layers should be stacked, and ii) how many should be actively connected via hybrid bonding? Stacked layers contribute capacity and increase thermal resistance; connected layers contribute bandwidth but consume additional power and PHY/controller area. We systematically explore this design space and derive quantitative guidance.

Quantitative Analysis of 3D vs. 2.5D Systems

5.5.1 Layer–Bandwidth Trade-off. We sweep L1 capacity (128/ 256/512 KiB) and bandwidth (128/256/512/1024 B/cycle). Fig. 13 shows a representative subset. Counter-intuitively, effective bandwidth peaks at fewer than 10 layers under 7 nm. Beyond this point, Little’s law and L1 bandwidth constraints cause effective bandwidth to decline even as theoretical bandwidth rises. We therefore prune configurations exceeding 9 layers from the subsequent design space. Key Takeaway 4: DRAM layer stacking exhibits an inverted-U curve: beyond ∼9 layers, effective bandwidth decreases despite increasing theoretical bandwidth, mainly due to buffering limitation.

To quantitatively analyze the benefits of 3D-stacked DRAM for distributed LLM inference, we compare 3D designs against H100* / H200* 2.5D baselines (Table 2) scaled to 7 nm, modeling a 32-GPU cluster with NVSwitch intra-node and InfiniBand inter-node [71]. For DeepStack, we perform a comprehensive hardware–software cosearch across compute configurations and NoC topologies (Table 3). All candidates, including all 2.5D and 3D designs, are constrained to an identical total area budget of 25,600 mm2 . Fig. 12 shows the Pareto frontiers derived from over one million design points. In the memory-bound decoding, 3D-stacked designs significantly outperform 2.5D baselines, with speedups of 1.30–1.48× at BS = 1024 and up to 2.79× at BS = 1, where per-token memory traffic dominates. These results use a fixed 4-layer stacking. DRAM layer and NoC co-optimization (§5.5) further pushes the DeepSeek-V3 advantage to 1.73× at BS = 1024 (Table 4). In the compute-bound prefill phase, 3D designs perform comparably to H200*, as the latter’s higher peak TFLOPs and larger capacity

5.5.2 End-to-End Performance vs. Stacked Layers. Fig. 14 shows the end-to-end throughput and area breakdown, sweeping stacked layers (all connected) under a fixed area budget where DRAM layers compete directly with compute SMs for die area. The optimal stacking depth varies significantly with workload: 9

Zhiwen Mo et al.

3D DRAM Peak BW After Charging Limits

Little's Law Limit L1 Limit (Effective BW)

H200 (4.8 TB/s) A100 (2.0 TB/s)

Best Eff. BW SM Count

size defines a more fundamental architectural divide than the prefill/decode distinction. The optimal hardware naturally clusters into three groups: (1) large-batch prefill (shallow stacks, maximum compute), (2) small-batch prefill together with large-batch decode (moderate stacks, balanced), and (3) small-batch decode (deep stacks, maximum bandwidth). Batch-size-aware hardware disaggregation may therefore be more effective than PD disaggregation alone.

SM Count

BW (TB/s)

L1 Cap. 128KiB, BW 128B/c L1 Cap. 128KiB, BW 256B/c L1 Cap. 128KiB, BW 1024B/c 50

SM Count

BW (TB/s)

L1 Cap. 512KiB, BW 128B/c L1 Cap. 512KiB, BW 256B/c L1 Cap. 512KiB, BW 1024B/c 40

5.5.3 Connected vs. Stacked Layers: Decoupled Design. We differentiate stacked layers (𝑚, capacity) from connected layers (𝑛 ≤ 𝑚, bandwidth). Unconnected layers add capacity without the power and PHY area overhead of active connections but increase thermal resistance. As shown in Fig. 15, the throughput-optimal and energy-optimal designs diverge significantly: Throughput-optimal configurations maximize connected layers to saturate bandwidth, with optimal (𝑚, 𝑛) shifting from (6, 6) at BS=4 to (2, 2) at BS=1024 for prefill. This Δ𝑚=4 gap is driven by batch size, whereas the prefill-vs-decode gap at the same BS can be smaller or even zero, further confirming that batch size drives architectural divergence more than inference phase. Energy-efficient configurations consistently favor more stacked but fewer connected (idle) layers, operating at 0.5–0.7 W/mm2 (10– 48% below throughput-optimal) with 3–24% tokens/J gains. These designs compensate for reduced bandwidth through larger on-chip buffers and improved data reuse, accompanied by a shift toward EPconcentrated parallelism (e.g., tp1/ep32 vs. tp4/ep4/dp4/pp4) that partitions MoE weights across devices to reduce memory access energy. Key Takeaway 6: These results show that energy-efficient and throughput-optimal designs require fundamentally different architectures. 3D integration should be co-designed with the optimization objective, not just the workload.

2 4 6 8 10 12 14 16

Stacked 3D DRAM Layers

2 4 6 8 10 12 14 16

Stacked 3D DRAM Layers

Figure 13: Theoretical and effective DRAM bandwidth breakdown & SM count as a function of stacked 3D DRAM layers (stack = connected, 800 mm2 , 7 nm process ). A representative subset is shown for clarity.

Throughput (K tok/s)

BS=4

BS=16

(a) Decode Throughput

BS=64

BS=256

BS=1024

(b) Prefill Throughput

40 20 0

OOM

Budget (800 mm²)

Mem Ctrl

DRAM PHY+L2 Cache

L1 NoC

D2D (L2 NoC)

11 C2C (L3 NoC)

Die Area (mm²)

800 600 400 200

54%

49%

47%

43%

41%

37%

34%

29%

25%

24%

20%

0 1

5 7 9 Stacked DRAM Layers

5.5.4 Thermal Feasibility. We conduct thermal analysis for different workloads. As shown in Fig. 16, large-batch decode is the most thermally challenging regime, with many configurations exceeding 85◦ C, while prefill and small-batch decode remain safe. High-throughput designs within the thermal envelope do exist: larger on-chip buffers and coarser tiling reduce DRAM access frequency, lowering power without sacrificing throughput. Notably, all thermally feasible designs identified by our DSE operate below ∼0.8 W/mm2 , well under the ≥1.34 W/mm2 already sustained by the NVIDIA Vera Rubin GPU [16] with production liquid-cooling solutions. Key Takeaway 7: Thermal limits do not rule out high-throughput designs, but they constrain the viable design space, especially for large-batch decode. Thermal-aware DSE is therefore necessary to identify designs that are both high-performance and feasible.

14%

Figure 14: End-to-end TPS for DeepSeek-V3 and area breakdown across different DRAM stacking layers. The optimal layer depends on batch size and inference phase.

• Small-batch decode (BS=4): favors deep stacks (∼9 layers) to maximize bandwidth for memory-bound single-token generation. • Large-batch decode (BS=1024): the optimum drops to 6–7 layers, as the workload becomes partially compute-bound and the area consumed by additional DRAM layers is better allocated to SMs. • Small-batch prefill: prefers ∼7 layers, balancing bandwidth with compute capacity. • Large-batch prefill: only 2 layers are optimal, as the workload is fully compute-bound and every additional DRAM layer displaces productive compute area.

5.5.5 NoC Sensitivity Analysis. As shown in Fig. 17, we perform a sensitivity study on NoC bandwidth and hop latency, scaling all three hierarchical layers defined in Table 3 simultaneously. Importantly, we co-optimize NoC bandwidth with compute area: reducing bandwidth frees die area for additional SMs (e.g., 0.75× BW yields 9 SMs vs. 8 at 1× BW), while increasing to 2× BW leaves only 5 SMs. At BS=1024, latency has minimal impact (3.6% STPS variation over a 0.25×-to-4× sweep), as abundant parallelism hides communication latency. Bandwidth matters only through its area trade-off:

Key Takeaway 5: This analysis reveals a design insight that goes beyond conventional prefill–decode (PD) disaggregation: batch 10

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

Prefill

BS = 4

Stacked DRAM Layers

System Throughput 14 12 10 8 6 4 2

47K

52K

54K

39K

50K

56K

49K

34K

55K

65K

66K

57K

55K

73K

76K

68K

52K

65K

77K

47K

59K

BS = 1024

Stacked DRAM Layers

2 14 105K 12 122K 10 147K 8 166K 6 178K 4 184K 2 194K 2

60000

40000

Best Throughput

102K

78K

54K

102K

76K

54K

146K

123K

102K

166K

150K

123K

179K

155K

Best Throughput

4.21

4.17

3.86

4.21

4.35

4.17

3.86

4.36

4.45

4.39

4.35

4.36

4.54

4.55

4.39

4.22

4.29

4.24

3.64

3.76

2.70

180000 14 160000 12 140000 10 120000 8 100000 6 80000 4 60000 2

174K

14 12 10 8 6 4 2

70000

50000

36K

Decode

Tokens/J

Connected DRAM Layers

Best Power Efficiency

8.73

8.69

8.07

7.15

8.84

8.57

8.56

7.95

7.11

9.16

9.15

8.86

8.57

9.28

9.17

8.84

9.15

9.16

9.03

9.19

9.18

9.00 8.50 8.00 7.50

9.21

4.50 4.25 4.00 3.75 3.50 3.25 3.00 2.75

Best Power Efficiency

Connected DRAM Layers

Decode

System Throughput 14 12 10 8 6 4 2

257

393

454

323

257

385

454

455

323

239

385

458

492

493

239

372

442

482

239

362

429

226

333

114

14 12 10 8 6 4 2

27K

31K

32K

25K

28K

38K

36K

32K

25K

28K

39K

41K

42K

28K

41K

47K

46K

27K

41K

50K

25K

40K

40000 30000 20000 10000

Best Throughput

14 12 10 8 6 4 2

450 400 350 300 250 200 150

Best Throughput

2.6K

Tokens/J

Connected DRAM Layers

0.028

0.031

0.032

0.029

0.028

0.030

0.032

0.029

0.028

0.030

0.031

0.032

0.028

0.031

0.032 0.031

0.028

0.031

0.032

0.028

0.030

0.013

14 12 10 8 6 4 2

Best Power Efficiency

2.31

2.34

2.20

2.32

2.42

2.34

2.20

2.33

2.47

2.48

2.43

2.33

2.51

2.53

2.49

2.33

2.51

2.53

2.14

2.34

2.00 1.50

Best Power Efficiency

2.50

1.00

0.241

0.030 0.028 0.025 0.022 0.020 0.018 0.015

Connected DRAM Layers

0.500

Figure 15: DSE heatmaps for throughput-optimal and energy-optimal designs across stacked (𝑚) and connected (𝑛) DRAM layer configurations. Energy-efficient designs consistently favor more stacked but fewer connected layers.

80 60

40 100

STP200 S (to 300 k/s) 400

500

1112 910

40 10k

STP20k S (to 30k k/s) 40k

rs aye

A DR

50k

aye

A DR

Figure 16: Temperature distribution across the 3D DRAM design space. Large-batch decode is the most thermally challenging regime. High-throughput configurations within the safe 85◦ C envelope exist.

1.0 0.9

0.25 0.50 1.00 2.00 4.00

BW Multiplier

2×

1.5×

2×

1× 0.25×

4× 0.5× 1×

2×

1.5×

2×

1× 0.25×

y nc

4× 1×

2×

1.5×

2×

1× 0.25×

(K tokens/s)

Decode STPS

0.9

1.0

BW Multiplier

BS=16

BS=64

Scale L2

0.25 0.50 1.00 2.00 4.00

BW Multiplier

BS=4096

Scale L3

1.0 0.8

0.25 0.50 1.00 2.00 4.00

BS=256

0.5

+ More Parallel Dims

BS=1024 1.0

BW Multiplier

Scale L3

0.8 0.6

0.25 0.50 1.00 2.00 4.00

BW Multiplier

y nc

Figure 17: NoC bandwidth and hop latency sensitivity. Reducing bandwidth frees die area for additional SMs.

Qwen3-235B-A22B

20 10 0

150 100 50 0

+ Flexible Module Parallelism (DeepStack) 30

30 20 10 0

0.8

(K tokens/s)

4× 0.5×

0.8 0.7

1.0

Prefill STPS

0.8 0.7

1.1

1.1 1.0 0.9

0.8

1×

Normalized STPS

1.1 1.0 0.9

0.7

0.5×

Normalized STPS

1.1 1.0 0.9

BS=4

DeepSeek-V3

BS = 1024

Normalized STPS

BS = 64

BS=1024

Figure 18: Per-layer NoC bandwidth sensitivity. Each hierarchical layer is scaled independently while others remain at baseline. Upper: decode phase; lower: prefill phase. ASTRA-sim*

BS = 4

BS=256

0.25 0.50 1.00 2.00 4.00

BS=1

Scale L1

BS=64

Scale L2

Normalized STPS

BW Multiplier

BS=16

1.00 0.75 0.50

0.25 0.50 1.00 2.00 4.00

Prefill

1112 910

Normalized STPS

BS=4

Normalized STPS

100

BS=1

Scale L1

1.00 0.95 0.90

Normalized STPS

Temp (°C)

80 60

100

120

85 °C

100

Temperature (°C)

120

85 °C

120

Normalized STPS

Decode

Decode, BS = 1024

Temp (°C)

Decode, BS = 4

Batch Size

128 1024

200 150 100 50 0

Batch Size

128 1024

Figure 19: Effectiveness of expanded parallelism in DeepStack. ASTRA-sim* supports only TP/PP/DP.

0.75× BW gains 2.4% from one extra SM, whereas 2× BW loses 27.2% from fewer SMs. At BS=4, the trend reverses: latency becomes dominant (up to 31.3% variation, 1.36× speedup at 0.25×), and reducing BW to 0.75× still improves STPS by 8.8% by freeing area for compute. The overall optimal is 0.75× BW with 0.25× latency, achieving up to 17.3% improvement for BS=4. Fig. 18 decomposes the contribution of each NoC layer by independently scaling its bandwidth. The sensitivity is non-uniform: L2 (inter-tile) bandwidth can be modestly reduced without performance loss, while L3 (off-chip) benefits from a slight increase,

suggesting asymmetric interconnect area allocation across the hierarchy.

5.6

Ablation Study

5.6.1 Expanded Parallelism Search Space. Fig. 19 isolates the parallelism impact across models at decode BS = 1024. Replacing ASTRAsim’s TP/PP/DP-only space with full parallelism (including EP) 11

Zhiwen Mo et al.

Table 4: Ablation study on DeepStack techniques. All configurations satisfy the area (32 × 800 mm2 ) and thermal constraints. # Ablation Study 1 2

Baseline (ASTRA-sim:DP/TP/PP/FSDP) + Full Parallel(EP/SP/CP/FSDP) Flex. Parallel Across Modules 3 + (E.g., EP in MoE and TP in Atten.) 4 + Search On-chip Arch. 5 + Comm.-Comp. Overlap 6 + Stacking DRAM Layer DSE 7 + NoC DSE Total Speedup

STPS𝑏𝑠=4

STPS𝑏𝑠=1024

177.1 256.4 (+45%)

5,729 21,252 (+271%)

256.4 (—)

24,488 (+15%)

314.2 (+23%) 340.5 (+8%) 493.3 (+45%) 494.1 (+0.2%)

31,350 (+28%) 38,061 (+21%) 51,095 (+34%) 54,280 (+6.2%)

2.8×

9.5×

works each address important aspects of 3D design; our work is complementary in that it provides an end-to-end DSE framework that jointly explores parallelism scheduling, hierarchical NoC design, and thermal constraints in a distributed setting (Table 1).

6.2

Modeling and DSE Tools

Single-chip modeling frameworks such as Timeloop [75], MAESTRO [46], TileFlow [120], and MIND [34] provide accurate dataflowlevel performance estimation but focus on individual accelerator chips. For distributed systems, ASTRA-sim [82, 105] models hierarchical network topologies and supports TP, PP, DP, and FSDP for training simulation, with its NS-3 backend offering high-fidelity network modeling. LLMCompass [113] employs high-level analytical models to rapidly evaluate LLM inference across hardware configurations with TP and PP support. Our work extends these efforts to the 3D-stacked domain by adding transaction-aware DRAM modeling, a broader parallelism search space (including EP, SP, CP, and per-module flexibility), compute–communication overlap, and thermal–power co-modeling within a unified DSE loop.

and per-module flexibility yields 5.03× for DeepSeek-V3 and 2.31× for Qwen3-235B, since DeepStack allows each module to adopt an independent parallelism strategy and automatically inserts the necessary collectives to reconstruct the required input tensors across module boundaries. 5.6.2 Cumulative Technique Contribution. Table 4 quantifies the cumulative gain of all techniques atop the ASTRA-sim baseline, reaching 2.8× speedup at BS=4 and 9.5× at BS=1024. EP (Step 2) and DRAM stacking DSE (Step 6) contribute the largest individual gains; Steps 6–7 (DRAM layer + NoC DSE) further push the decode advantage over H200* from 1.30× to 1.73× at BS=1024, a 33–39% gain missed by fixed stacking. Notably, omitting even a single parallelism dimension does not merely reduce throughput—it actively misleads the architecture search. At BS=1024, removing EP causes the DSE to converge on a pipeline-parallel-heavy schedule (tp=16, pp=8) and a different chip (8 stacked/connected layers, 5 SMs) that hits the power wall; with EP, the design shifts to an all-to-all regime (ep=32, tp=4, 7 stacked/connected layers, 6 SMs) within thermal limits. At BS=4, the divergence is even starker: stacked layers (7 vs. 10), shared memory (128K vs. 256K), and SM count (6 vs. 4) all differ, producing qualitatively different chips. Key Takeaway 8: Incomplete schedule search permanently distorts architecture design. The resulting silicon mismatch is irrecoverable by software tuning, underscoring the necessity of co-searching the full system–hardware space before committing to silicon.

6.3

Thermal Challenges in 3D Stacking

3D stacking increases power density without enlarging the dissipation area, making thermal a critical constraint. HotSpot [29], CACTI3DD [12], and COMET [94] provide compact thermal modeling for 3D-stacked DRAM and multi-core systems. On the cooling side, emerging solutions include embedded microfluidics [104], inter-tier liquid cooling [96], Through-Chip Microchannels (TCMCs) capable of 14.1 kW/cm2 dissipation [5], full immersion cooling [81], and Carbon-Nanotube-based TSVs for enhanced passive conduction [26]. The NVIDIA Vera Rubin GPU already sustains at least 1.34 W/mm2 in production with liquid metal TIM and micro-channel cold plates [16]. By integrating thermal modeling into the DSE loop to prune infeasible configurations early, our throughput-optimal 3D-stacked designs stay under 0.8 W/mm2 , comfortably within this envelope.

6.4

Distributed LLM Inference

Large-scale LLM training and inference are supported by mature frameworks [7, 50, 54, 68, 79, 84, 93]. A rich body of work explores parallelism and schedule search [36, 41, 44, 56, 67, 103, 117, 121, 124], extends sequence and expert parallelism for long-context and MoE workloads [37, 39, 58, 106], optimizes serving runtimes for iteration-level batching, KV-cache management, and prefill–decode disaggregation [2, 47, 80, 97, 110, 118], and fuses collectives with compute to hide network latency [9, 40, 78, 119]. These efforts target software mapping on fixed hardware, and the analytical models they rely on (e.g., Alpa [117], DFModel [44]) use simple linear models that serve as relative cost functions but fall short of accurate performance modeling for emerging 3D-stacked systems with fine-grained 3D characteristics. Our work complements them by co-searching the hardware architecture alongside the parallelism strategy, particularly for emerging 3D-stacked hardware.

6 Related Work 6.1 3D-Stacked DRAM Architecture Exploration Recent works explore 3D-stacked or hybrid-bonded DRAM for AI accelerators from different angles. H2-LLM [51] targets nearmemory compute on the logic die for low-batch inference. Stratum [74] exploits heterogeneous access latency in monolithic 3D DRAM for MoE models. TASA [30] proposes thermal-aware 3D architectures for LLM inference, and AccelStack [6] provides a cost modeling framework for hybrid-bonded manufacturing flows. STCO [91] evaluates 3D-stacked DRAM and V-Cache architectures with area, power, and thermal analysis. Helios [52] proposes NUMAcentric optimizations for KV-cache placement in 3D DRAM-based LLM serving with TP, PP, and EP. LIMINAL [22] compares SRAM, HBM, and stacked-DRAM architectures at the system level. These 12

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

Conclusion

Operating Systems Design and Implementation (OSDI 25). [15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113. [16] Wega Chu, Dylan Patel, Daniel Nishball, et al. 2026. Vera Rubin – Extreme Co-Design: An evolution from Grace Blackwell Oberon. SemiAnalysis Newsletter. https://newsletter.semianalysis.com/p/vera-rubin-extreme-co-design-anevolution Accessed: 2026-03-20. [17] Lawrence T Clark, Vinay Vashishtha, Lucian Shifren, Aditya Gujja, Saurabh Sinha, Brian Cline, Chandarasekaran Ramamurthy, and Greg Yeric. 2016. ASAP7: A 7-nm finFET predictive process design kit. Microelectronics Journal 53 (2016), 105–115. [18] NVIDIA Corporation. 2020. NVIDIA A100 Tensor Core GPU Architecture. Technical Report. NVIDIA Corporation. https://images.nvidia.com/aem-dam/enzz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf [19] NVIDIA Corporation. 2023. NVIDIA H100 Tensor Core GPU Architecture. Technical Report. NVIDIA Corporation. https://resources.nvidia.com/en-us-tensorcore [20] NVIDIA Corporation. 2024. NVIDIA Blackwell Architecture Technical Brief. Technical Report. NVIDIA Corporation. https://resources.nvidia.com/en-usblackwell-architecture [21] NVIDIA Corporation. 2025. CUDA Techniques to Maximize Memory Bandwidth and Hide Latency (Session S72683). https://www.nvidia.com/en-us/on-demand/ session/gtc25-s72683/. [22] Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Christos Kozyrakis. 2025. Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need. arXiv preprint arXiv:2507.14397 (2025). [23] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints (2024), arXiv–2407. [24] William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. [25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [26] Heebo Ha, Hongju Kim, Sumin Lee, Sooyong Choi, Chunghyeon Choi, Wan Yusmawati Wan Yusoff, Ali Shan, Sooman Lim, and Byungil Hwang. 2025. Overview of thermal management solution for 3D integrated circuits using Carbon-Nanotube-Based Silicon Through-Vias. Micromachines (2025). [27] Ramyad Hadidi, Bahar Asgari, Burhan Ahmad Mudassar, Saibal Mukhopadhyay, Sudhakar Yalamanchili, and Hyesoon Kim. 2017. Demystifying the characteristics of 3D-stacked memories: A case study for hybrid memory cube. In 2017 IEEE international symposium on Workload characterization (IISWC). IEEE, 66–75. [28] Walid Hafez, P Agnihotri, M Asoro, M Aykol, B Bains, R Bambery, M Bapna, A Barik, A Chatterjee, PC Chiu, et al. 2023. Intel PowerVia technology: Backside power delivery for high density and high-performance computing. In 2023 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits). IEEE, 1–2. [29] Jun-Han Han, Xinfei Guo, Kevin Skadron, and Mircea R. Stan. 2022. From 2.5D to 3D Chiplet Systems: Investigation of thermal implications with HotSpot 7.0. 2022 21st IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (iTherm) (2022), 1–6. https://api.semanticscholar.org/ CorpusID:252625064 [30] Siyuan He, Peiran Yan, Yandong He, Youwei Zhuo, and Tianyu Jia. 2025. Tasa: Thermal-aware 3D-stacked architecture design with bandwidth sharing for LLM inference. arXiv preprint arXiv:2508.07252 (2025). [31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022). [32] Mark Horowitz. 2014. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC). IEEE, 10–14. [33] Han-Wen Hu and Kuan-Neng Chen. 2021. Development of low temperature CuCu bonding and hybrid bonding for three-dimensional integrated circuits (3D IC). Microelectronics Reliability 127 (2021), 114412. [34] Qijing Huang, Po-An Tsai, Joel S Emer, and Angshuman Parashar. 2024. Mind the gap: Attainable data movement and operational intensity bounds for tensor algorithms. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 150–166. [35] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019).

We present DeepStack, a performance modeling and efficient DSE framework for distributed 3D-stacked accelerators. By capturing 3Dstacked memory semantics, such as transaction-aware bandwidth, bank activation constraints, buffering restrictions and thermal– power co-modeling, as well as comprehensive parallelism and scheduling search, DeepStack enables systematic exploration of the vast hardware–system co-design space at early design stages. Our extensive DSE reveals that optimal DRAM stacking depth is governed by a non-trivial bandwidth–area trade-off, and that batch size creates a more fundamental architectural divide than the prefill/decode distinction. Critically, our ablation shows that parallelism strategy and hardware architecture are tightly coupled: optimizing one without the other leads to suboptimal designs irrecoverable by software. We hope DeepStack and these findings will guide early-stage architectural decisions for next-generation scalable AI infrastructure.

References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In USENIX Symposium on Operating Systems Design and Implementation. https: //api.semanticscholar.org/CorpusID:268249103 [3] Moonshot AI. 2025. Kimi-K2 Thinking. Moonshot AI. https://moonshotai. github.io/Kimi-K2/thinking.html [4] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023). [5] Lihong Ao and Aymeric Ramiere. 2024. Through-chip microchannels for three-dimensional integrated circuits cooling. Thermal Science and Engineering Progress 47 (2024), 102333. [6] Chen Bai, Xin Fan, Zhenhua Zhu, Wei Zhang, and Yuan Xie. 2025. AccelStack: A Cost-Driven Analysis of 3D-Stacked LLM Accelerators. In 2025 ACM/IEEE International Conference on Computer-Aided Design (ICCAD). [7] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. http://github.com/jax-ml/jax [8] Cadence Design Systems. 2026. Palladium Emulation. https://www.cadence. com/en_US/home/tools/system-design-and-verification/emulation-andprototyping/palladium.html. Accessed: 2026-03-29. [9] Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuan Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. 2024. FLUX: Fast software-based communication overlap on GPUs through kernel fusion. ArXiv abs/2406.06858 (2024). https://api.semanticscholar.org/ CorpusID:270380238 [10] Hao Mark Chen, Guanxi Lu, Yasuyuki Okoshi, Zhiwen Mo, Masato Motomura, and Hongxiang Fan. 2025. Rethinking optimal verification granularity for compute-efficient test-time scaling. arXiv preprint arXiv:2505.11730 (2025). [11] Hao Mark Chen, Zhiwen Mo, Guanxi Lu, Shuang Liang, Lingxiao Ma, Wayne Luk, and Hongxiang Fan. 2026. FastTTS: Accelerating test-time scaling for edge LLM reasoning. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 732–748. [12] Ke Chen, Sheng Li, Naveen Muralimanohar, Jung Ho Ahn, Jay B Brockman, and Norman P Jouppi. 2012. CACTI-3DD: Architecture-level modeling for 3D die-stacked DRAM main memory. In 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 33–38. [13] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. { TVM } : An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 578– 594. [14] Yu Cheng, Lei Wang, Yining Shi, Yuqing Xia, Lingxiao Ma, Jilong Xue, Yang Wang, Zhiwen Mo, Feiyang Chen, Fan Yang, et al. 2025. PipeThreader: Softwaredefined pipelining for efficient DNN execution. In 19th USENIX Symposium on 13

Zhiwen Mo et al.

[56] Zhiqi Lin, Youshan Miao, Guanbin Xu, Cheng Li, Olli Saarikivi, Saeed Maleki, and Fan Yang. 2024. Tessel: Boosting distributed execution of large dnn models via flexible schedule search. In 2024 IEEE International Symposium on HighPerformance Computer Architecture (HPCA). IEEE, 803–816. [57] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [58] Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. ArXiv abs/2310.01889 (2023). https: //api.semanticscholar.org/CorpusID:263608461 [59] Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407 (2025). [60] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. 2025. Can 1b llm surpass 405b LLM? rethinking computeoptimal test-time scaling. arXiv preprint arXiv:2502.06703 (2025). [61] Shuqing Luo, Ye Han, Pingzhi Li, Jiayin Qin, Jie Peng, Yang Katie Zhao, Yu Cao, and Tianlong Chen. 2025. Mozart: Modularized and efficient MoE training on 3.5 D wafer-scale chiplet architectures. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [62] Xiaoyu Ma and David A. Patterson. 2026. Challenges and Research Directions for Large Language Model Inference Hardware. ArXiv abs/2601.05047 (2026). https://api.semanticscholar.org/CorpusID:284543821 [63] Stefan Mach, Fabian Schuiki, Florian Zaruba, and Luca Benini. 2020. FPnew: An open-source multiformat floating-point unit architecture for energyproportional transprecision computing. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 29, 4 (2020), 774–787. [64] Zhiwen Mo, Lei Wang, Jianyu Wei, Zhichen Zeng, Shijie Cao, Lingxiao Ma, Naifeng Jing, Ting Cao, Jilong Xue, Fan Yang, et al. 2025. LUT Tensor Core: A software-hardware co-design for LUT-based low-bit LLM inference. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 514–528. [65] Manuel Mota. 2022. Unpacking the Rise of Multi-Die SoCs with UCIe. Synopsys Technical Article. https://www.synopsys.com/articles/ucie-multi-die-socs. html Accessed: 2026-03-24. [66] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems principles. 1–15. [67] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei A. Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. Proceedings of the 27th ACM Symposium on Operating Systems Principles (2019). https: //api.semanticscholar.org/CorpusID:202488191 [68] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis. 1–15. [69] NVIDIA. 2026. CUTLASS. https://github.com/NVIDIA/cutlass. [70] NVIDIA. 2026. Megatron-LM. https://github.com/NVIDIA/Megatron-LM. [71] NVIDIA Corporation. 2023. NVIDIA DGX H100/H200 User Guide. https://docs. nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html Accessed: 2025-11-17. [72] Mike O’Connor, Niladrish Chatterjee, Donghyuk Lee, John Wilson, Aditya Agrawal, Stephen W Keckler, and William J Dally. 2017. Fine-grained DRAM: Energy-efficient DRAM for extreme bandwidth systems. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture. 41–54. [73] Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D Owens. 2023. Stream-k: Work-centric parallel decomposition for dense matrixmatrix multiplication on the GPU. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. 429–431. [74] Yue Pan, Zihan Xia, Po-Kai Hsu, Lanxiang Hu, Hyungyo Kim, Janak Sharda, Minxuan Zhou, Nam Sung Kim, Shimeng Yu, Tajana Rosing, et al. 2025. Stratum: System-hardware co-design with tiered monolithic 3D-stackable DRAM for efficient MoE serving. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture®. 1–17. [75] Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emer. 2019. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEEE international symposium on performance analysis of systems and software (ISPASS). IEEE, 304–315. [76] Sudeep Pasricha and Mahdi Nikdast. 2020. A survey of silicon photonics for energy-efficient manycore computing. IEEE Design & Test 37, 4 (2020), 60–81. [77] Dylan Patel, Kimbo Chen, Daniel Nishball, et al. 2025. InferenceMAX™: Open Source Inference Benchmarking. https://newsletter.semianalysis.com/

[36] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Z. Chen. 2018. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Neural Information Processing Systems. https: //api.semanticscholar.org/CorpusID:53670168 [37] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. 2022. Tutel: Adaptive mixture-of-experts at scale. ArXiv abs/2206.03382 (2022). https://api.semanticscholar.org/CorpusID: 249431713 [38] Bruce Jacob, David Wang, and Spencer Ng. 2010. Memory systems: cache, DRAM, disk. Morgan Kaufmann. [39] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509 (2023). [40] Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madan Musuvathi, Todd Mytkowicz, and Olli Saarikivi. 2021. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (2021). https://api.semanticscholar.org/CorpusID:237292610 [41] Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems 1 (2019), 1–13. [42] Anne Jourdain, Michele Stucchi, Geert Van der Plas, Gerald Beyer, and Eric Beyne. 2022. Buried power rails and nano-scale TSV: technology boosters for backside power delivery network and 3D heterogeneous integration. In 2022 IEEE 72nd Electronic Components and Technology Conference (ECTC). IEEE, 1531–1538. [43] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). [44] Sho Ko, Nathan Zhang, Olivia Hsu, Ardavan Pedram, and Kunle Olukotun. 2024. DFModel: Design space optimization of large-scale systems exploiting dataflow mappings. arXiv preprint arXiv:2412.16432 (2024). [45] Alex Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 (2014). [46] Hyoukjun Kwon, Prasanth Chatarasi, Vivek Sarkar, Tushar Krishna, Michael Pellauer, and Angshuman Parashar. 2020. Maestro: A data-centric approach to understand reuse, performance, and hardware cost of dnn mappings. IEEE micro 40, 3 (2020), 20–29. [47] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. [48] Seonho Lee, Amar Phanishayee, and Divya Mahajan. 2025. Forecasting GPU performance for deep learning training and inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 493–508. [49] Seung-Hoon Lee, Su-Jong Kim, Ji-Su Lee, and Seok-Ho Rhi. 2025. Thermal issues related to hybrid bonding of 3D-stacked high bandwidth memory: A comprehensive review. Electronics 14, 13 (2025), 2682. [50] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 (2020). [51] Cong Li, Yihan Yin, Xintong Wu, Jingchen Zhu, Zhutianya Gao, Dimin Niu, Qiang Wu, Xin Si, Yuan Xie, Chen Zhang, et al. 2025. H2-LLM: Hardwaredataflow co-exploration for heterogeneous hybrid-bonding-based low-batch LLM inference. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 194–210. [52] Cong Li, Yihan Yin, Chenhao Xue, Zhao Wang, Fujun Bai, Yixin Guo, Xiping Jiang, Qiang Wu, Yuan Xie, and Guangyu Sun. 2026. Hardware-software co-design for 3D-DRAM-based LLM serving accelerator. arXiv preprint arXiv:2603.04797 (2026). [53] Hao Li, Ganesh Balamurugan, James Jaussi, and Bryan Casper. 2018. A 112 Gb/s PAM4 linear TIA with 0.96 pJ/bit energy efficiency in 28 nm CMOS. In ESSCIRC 2018-IEEE 44th European Solid State Circuits Conference (ESSCIRC). IEEE, 238–241. [54] Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing. 766–775. [55] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2023. Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2391–2404. 14

DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators

[98] Jonas Svedas, Hannah Watson, Nathan Laubeuf, Diksha Moolchandani, Abubakr Nada, Arjun Singh, Dwaipayan Biswas, James Myers, and Debjyoti Bhattacharjee. 2025. A survey of end-to-end modeling for distributed DNN training: Workloads, simulators, and TCO. arXiv preprint arXiv:2506.09275 (2025). [99] Qwen Team. 2025. Qwen3-Max: Just Scale it. [100] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [101] Lei Wang, Yu Cheng, Yining Shi, Zhengju Tang, Zhiwen Mo, Wenhao Xie, Lingxiao Ma, Yuqing Xia, Jilong Xue, Fan Yang, and Zhi Yang. 2025. TileLang: A composable tiled programming model for AI systems. ArXiv abs/2504.17577 (2025). https://api.semanticscholar.org/CorpusID:278033547 [102] Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, et al. 2024. Ladder: Enabling efficient low-precision deep learning computing through hardware-aware tensor transformation. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 307–323. [103] Minjie Wang, Chien-chin Huang, and Jinyang Li. 2019. Supporting very large models using automatic dataflow graph partitioning. In Proceedings of the Fourteenth EuroSys Conference 2019. 1–17. [104] Shaoxi Wang, Yue Yin, Chenxia Hu, and Pouya Rezai. 2018. 3D integrated circuit cooling with microfluidics. Micromachines 9, 6 (2018), 287. [105] William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2023. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 283–294. https://doi.org/10.1109/ISPASS57527.2023.00035 [106] Bingya Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently serving long-context large language models with elastic sequence parallelism. Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (2024). https://api.semanticscholar.org/CorpusID: 269149599 [107] Di Wu, Tianen Chen, Chienfu Chen, Oghenefego Ahia, Joshua San Miguel, Mikko Lipasti, and Younghyun Kim. 2019. SECO: A scalable accuracy approximate exponential function via cross-layer optimization. In 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED). IEEE, 1–6. [108] Rui Xie, Yunhua Fang, Asad Ul Haq, Linsen Ma, Sanchari Sen, Swagath Venkataramani, Liu Liu, and Tong Zhang. 2025. Making strong error-correcting codes work effectively for HBM in AI inference. arXiv preprint arXiv:2512.18152 (2025). [109] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [110] Gyeong-In Yu and Joo Seong Jeong. 2022. Orca: A distributed serving system for Transformer-based generative models. In USENIX Symposium on Operating Systems Design and Implementation. https://api.semanticscholar.org/CorpusID: 251734964 [111] Zhiheng Yue, Huizheng Wang, Jiahao Fang, Jinyi Deng, Guangyang Lu, Fengbin Tu, Ruiqi Guo, Yuxuan Li, Yubin Qin, Yang Wang, et al. 2024. Exploiting similarity opportunities of emerging vision ai models on hybrid bonding architecture. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 396–409. [112] Zhiheng Yue, Yang Wang, Chao Li, Shaojun Wei, Yang Hu, and Shouyi Yin. 2025. 3D-PATH: A hierarchy LUT processing-in-memory accelerator with thermal-aware hybrid bonding integration. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 78–93. [113] Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, and David Wentzlaff. 2024. Llmcompass: Enabling efficient hardware design for large language model inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 1080–1096. [114] Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2025. A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems 43, 6 (2025), 1–47. [115] Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, et al. 2025. Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 1731–1745. [116] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 (2023). [117] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and { Intra-Operator } parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and

p/inferencemax-open-source-inference Accessed: 2025-11-18. [78] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. 2024. T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (2024). https://api.semanticscholar.org/CorpusID:267320307 [79] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently scaling Transformer inference. ArXiv abs/2211.05102 (2022). https://api.semanticscholar.org/CorpusID:253420623 [80] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation - a KVCache-centric architecture for serving LLM chatbot. In USENIX Conference on File and Storage Technologies. https: //api.semanticscholar.org/CorpusID:276840869 [81] Delong Qiu, Liqiang Cao, Qidong Wang, Fengze Hou, and Xugang Wang. 2017. Experimental and numerical study of 3D stacked dies under forced air cooling and water immersion cooling. Microelectronics Reliability 74 (2017), 34–43. [82] Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2020. ASTRA-SIM: Enabling SW/HW co-design exploration for distributed DL training platforms. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 81–92. https://doi.org/10.1109/ ISPASS48437.2020.00018 [83] Saeed Rashidi, William Won, Sudarshan Srinivasan, Puneet Gupta, and Tushar Krishna. 2025. FRED: A Wafer-scale fabric for 3D parallel DNN training. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 34–48. [84] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506. [85] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al. 2020. MLperf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 446–459. [86] George F Riley and Thomas R Henderson. 2010. The ns-3 network simulator. In Modeling and tools for network simulation. Springer, 15–34. [87] Scott Rixner, William J Dally, Ujval J Kapasi, Peter Mattson, and John D Owens. 2000. Memory access scheduling. ACM SIGARCH Computer Architecture News 28, 2 (2000), 128–138. [88] Julien Ryckaert, Anshul Gupta, Anne Jourdain, Bharani Chava, Geert Van der Plas, Diederik Verkest, and Eric Beyne. 2019. Extending the roadmap beyond 3nm through system scaling boosters: A case study on Buried Power Rail and Backside Power Delivery. In 2019 Electron Devices Technology and Manufacturing Conference (EDTM). IEEE, 50–52. [89] Swapnil S Salvi and Ankur Jain. 2021. A review of recent research on heat transfer in three-dimensional integrated circuits (3-D ICs). IEEE Transactions on Components, Packaging and Manufacturing Technology 11, 5 (2021), 802–821. [90] SemiAnalysis. 2025. InferenceMAX. https://inferencemax.semianalysis.com/. Accessed: 2025-11-13. [91] Janak Sharda and Shimeng Yu. 2025. System-technology co-optimization methodology for LLM accelerators with advanced packaging. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2025). [92] Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. 2023. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 701–718. [93] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). [94] Lokesh Siddhu, Rajesh Kedia, Shailja Pandey, Martin Rapp, Anuj Pathania, Jörg Henkel, and Preeti Ranjan Panda. 2022. CoMeT: An integrated interval thermal simulation toolchain for 2D, 2.5 D, and 3D processor-memory systems. ACM Transactions on Architecture and Code Optimization (TACO) 19, 3 (2022), 1–25. [95] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm testtime compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024). [96] Arvind Sridhar, Alessandro Vincenzi, Martino Ruggiero, Thomas Brunschwiler, and David Atienza. 2010. 3D-ICE: Fast compact transient thermal modeling for 3D ICs with inter-tier liquid cooling. In 2010 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 463–470. [97] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. ArXiv abs/2406.03243 (2024). https://api.semanticscholar.org/CorpusID: 270258417 15

Zhiwen Mo et al.

[121] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. { DistServe } : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. [122] Anqi Zhou, Yu Zhang, Fei Ding, Ziqi Lian, Renxi Jin, Yudong Yang, Qidong Wang, and Liqiang Cao. 2024. Research progress of hybrid bonding technology for three-dimensional integration. Microelectronics Reliability 155 (2024), 115372. [123] Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, et al. 2022. { ROLLER } : Fast and efficient tensor compilation for deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 233–248. [124] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. 2025. { NanoFlow } : Towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 749–765.

Implementation (OSDI 22). 559–578. [118] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583. [119] Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang, Chenhui Huang, Tianqi Li, Haojie Duanmu, Renze Chen, Ruifan Xu, Yifan Guo, Ningxin Zheng, Ziheng Jiang, Xinyi Di, Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang, Liqiang Lu, Yun Liang, Jidong Zhai, and Xin Liu. 2025. Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler. arXiv:2504.19442 [cs.DC] https://arxiv.org/abs/2504.19442 [120] Size Zheng, Siyuan Chen, Siyuan Gao, Liancheng Jia, Guangyu Sun, Runsheng Wang, and Yun Liang. 2023. TileFlow: A framework for modeling fusion dataflow via tree-based analysis. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture. 1271–1288.

Record · ID 2572 · SHA-256 13c091dffd2d5a8b
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.