When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference

When Does Disaggregation Pay? Simulating Prefill–Decode–Attention–FFN Specialization for Agentic LLM Inference Przemyslaw Forys∗ , Haoran Wu† , Can Xiao∗ , Jiayi Nie† , Tony Liu∗ , Rika Antonova† , Timothy Jones† , Robert Mullins† , Wayne Luk∗ , Aaron Zhao∗ , George A. Constantinides∗ † University of Cambridge, UK

Abstract—Agentic inference now dominates the LLM inference landscape, requiring LLMs to actively engage in multi-turn interactions with tool-calling capabilities. This introduces a more complex workload for the underlying inference system: serving stages such as prefill and decode exhibit substantially different behaviors and demand distinct compute and memory-bandwidth capabilities. As a result, a single homogeneous GPU system now struggles to support agentic inference, motivating an industry shift toward heterogeneous systems with disaggregated serving capabilities, such as the emerging Vera-Rubin platform with GPUs and Groq LPUs. However, the question of what the optimal hardware should look like for each component in a heterogeneous system remains underexplored. To this end, we propose a novel simulation framework for disaggregated serving, termed HeteroPanacea, that enables system-level simulation across three dimensions: 1) disaggregated quantization, 2) automated intraand inter-device parallelization scheduling, and 3) PDAF (prefilldecode-attention-FFN) NPU architectural heterogeneity. By combining these three axes, we provide a cross-stack simulation framework for future heterogeneous agentic serving systems. We confirm the benefit of Prefill Decode disaggregation, simulating increased serving throughput by up to 75% compared to traditional serving with current GPUs and demonstrate 4 way Prefill Decode Attention FFN disaggregation is the most consistent for increasing throughput across different models, assuming custom NPUs. We also investigate the relationship between model architecture and gain from disaggregation by running a set of ablation studies.

38k

Tokens

40k 30k 20k 10k 0

1.2k Standard Agentic Chatbot CUA

Throughput (TPS)

arXiv:2608.03741v1 [cs.DC] 4 Aug 2026

∗ Imperial College London, UK

4.0k

Std · Prefill Std · Decode

CUA · Prefill CUA · Decode

2.0k 0

Standard Serving

PD Disaggregated HeteroPenacea Serving Serving

(a) Agentic workloads (b) System performance comparison of difhave larger token us- ferent serving systems, split into prefill and age. decode throughput.

(c) Feature comparison of standard serving, homogeneous PDdisaggregated serving, and HeteroPanacea NPU serving with stageoptimized configurations, indicated by different colors.

I. I NTRODUCTION

Fig. 1: Motivation of HeteroPanacea, a heterogeneous NPU serving system for agentic LLM workloads.

Large language models (LLMs) are increasingly moving beyond single-turn chatbot interactions toward agentic applications, where models reason, act, and interact with external environments over multiple turns. Representative workloads include computer-use agents (CUAs) [25], autonomous coding agents [11], [19], and web-use agents [2], [8], [25]. During these interactions, screenshots, web content, code context, intermediate reasoning, tool calls, and user feedback are repeatedly accumulated into the prompt, placing substantially higher memory and compute demands on the serving system than traditional chatbot inference [27]. Context length consequently grows rapidly during inference: on the OSWorld [25] benchmark it averages 38K tokens and can reach 100K, more than an order of magnitude beyond a standard chatbot session, as shown in Figure 1a.

As context length grows, the architectural mismatch between the distinct stages of the inference pipeline becomes increasingly pronounced on current AI inference devices, and a single homogeneous device can no longer efficiently sustain the end-to-end inference process. Prefill-Decode (PD) disaggregation has consequently emerged as a popular architectural paradigm in modern AI accelerator design [9], [31]. As illustrated in Figure 1b, where all systems are simulated under the same power budget and request rate, PD disaggregation delivers up to a 1.82× throughput improvement over a unified execution model on agentic workloads, against 1.29× on a standard chatbot workload. Building on this paradigm, recent

1

systems push disaggregation further by tailoring the hardware to each stage. NVIDIA’s upcoming Vera Rubin platform, for example, pairs Rubin GPUs – which handle the prefill stage and attention computation during decode – with Groq LPUs dedicated to FFN computation during decode [1]. However, despite the rapid industry shift toward heterogeneous, disaggregated systems, the question of what the optimal hardware should look like for each disaggregation stage remains largely underexplored, precisely because the NPU architecture best suited to one disaggregation stage can look nothing like the one best suited to the next. Two questions follow: to what extent does PD disaggregation remain beneficial under different workload profiles, and how do software-level optimizations such as quantization interact with, and shift, the optimal hardware configuration for each stage? Answering them – that is, systematically exploring this hardware design space – is non-trivial. It requires jointly reasoning across multiple, tightly coupled dimensions: the quantization chosen for each disaggregated stage; the parallelization strategy (e.g., tensor, pipeline, or expert parallelism) applied independently to prefill versus decode; the underlying hardware architecture selected for prefill versus decode NPUs; and even finer-grained heterogeneity within a single stage, such as assigning different hardware or precision to Attention versus FFN layers (also known as AF disaggregation). Existing simulation tools, summarized in Table I, treat these dimensions in isolation, if at all. To address these issues, we propose HeteroPanacea, a simulator for disaggregated, heterogeneous inference systems serving agentic workloads. HeteroPanacea enables systemlevel design-space exploration along five axes: (i) data, tensor, and pipeline parallelism; (ii) Prefill-Decode disaggregation; (iii) Attention-FFN disaggregation; (iv) fine-grained heterogeneous NPU configurations (e.g., matrix-engine size, SRAM capacity); and (v) mixed-precision quantization. Beyond throughput and latency, HeteroPanacea also models the accuracy impact of stage-wise quantization: attention and linear layers in the prefill and decode stages can each be assigned independent MXINT configurations, enabling finegrained accuracy-efficiency trade-off analysis. We use HeteroPanacea to study the design principles underlying nextgeneration AI infrastructure for agentic workloads, and will open-source the framework in full – simulator, configurations, and evaluation scripts – upon acceptance. The main contributions are as follows: • We propose HeteroPanacea, a heterogeneous system-level simulation framework for agentic LLM inference. HeteroPanacea supports configurable exploration of parallelization strategies, heterogeneous NPU architectures, mixedprecision configurations, and disaggregated serving designs. • We demonstrate the flexibility of the simulation framework by systematically evaluating disaggregated serving across diverse model architectures and workload profiles, and by benchmarking conventional Prefill-Decode and AttentionFFN disaggregation against a novel four-stage PDAF architecture (Prefill/Decode × Attention/FFN) that we in-

troduce, which jointly disaggregates both dimensions. In HeteroPanacea, the NPU design at each disaggregation stage can be independently customized: Prefill-Attention, PrefillFFN, Decode-Attention, and Decode-FFN can each be assigned a distinct NPU hardware architecture, as illustrated in Figure 1c. • We characterize the limitations of homogeneous hardware for disaggregated serving and investigate fully heterogeneous systems in which each inference stage is mapped to a stage-specialized NPU configuration. In our experiments, PD and PDAF disaggregation both benefit from specialized hardware for each stage, achieving up to a 2.06× throughput gain on agentic workloads compared to traditional serving. Crucially, this gain is conditional: disaggregation only clears parity once the workload is prefill-heavy, and the four-way PDAF split only outperforms plain PD when the hardware design space is rich enough to give attention and FFN genuinely different devices. II. BACKGROUND AND R ELATED W ORK A. Disaggregated Inference Serving Modern LLMs, built on the Transformer architecture [22], generate text autoregressively: given an input prompt, the model first processes all input tokens in a parallelizable prefill phase, then generates output tokens one at a time in an iterative decode phase. These two phases exhibit fundamentally different computational characteristics – prefill is computebound and amenable to high GPU utilization, while decode is memory-bandwidth-bound due to the sequential nature of token generation and the growing key-value (KV) cache [31]. As serving systems scale to handle thousands of concurrent requests, the heterogeneity between these computational phases creates resource contention and latency inefficiencies. A growing body of work has therefore explored disaggregated LLM serving – architectures that decouple different phases or components of inference onto separate hardware pools – in order to independently optimize throughput, latency, and resource utilization for each stage [14], [32]. Prefill–decode (PD) disaggregation places the prefill and decode phases on separate groups of serving devices. In conventional co-located serving, prefill requests interfere with decode iterations, increasing both TTFT and TPOT. Isolating and pipelining the two phases lets the prefill pool be optimised for TTFT and compute throughput while the decode pool is optimised for TPOT and memory efficiency. Representative systems such as DistServe [32] and Splitwise [14] show that this separation improves service-level objective (SLO) attainment and achieves better goodput. Two system-level problems arise from this separation. First, the prompt’s KV must be transferred from the prefill instance to the decode instance over the network, so the handoff cost is set by the topology and link bandwidth rather than by the model. Second, the two phases do not proceed at the same rate, so the pools must be provisioned and load-balanced against each other — prefill running ahead backs KV up behind decode, prefill lagging leaves the decode devices idle — and the ratio that balances them follows a

2

TABLE I: Feature comparison of representative LLM inference simulation frameworks. HeteroPanacea uniquely combines heterogeneous hardware modeling, PD and AF disaggregation, disaggregated quantization, multiple parallelization strategies, and design-space search. Framework LLMCompass [29] MemExplorer [23] DistServe [31] LLMServingSim2.0 [3] HeteroPanacea

Heterogeneity

PD Disagg.

AF Disagg.

Disagg. Quant.

Parallelism

Search

✗ ✓ ✗ ✓ ✓

✗ ✓ ✓ ✓ ✓

✗ ✓ ✗ ✓ ✓

✗ ✗ ✗ ✗ ✓

DP TP/DP/PP/EP TP/DP/PP TP/DP/PP/EP TP/DP/PP/EP

✓ ✗ ✓ ✗ ✓

prompt-to-output length mix that shifts at runtime. Simulating PD is therefore no longer an intra-replica scheduling problem but an inter-replica routing and bandwidth one. Attention–FFN (AF) disaggregation goes one level finer and splits the decode phase itself. It places the attention and FFN modules on separate GPU groups. Batching moves the two in different directions. Attention reads a distinct KV cache per request, so a larger batch adds memory traffic without adding reuse and the operator stays memory-bandwidth-bound. The FFN instead applies the same weights to every token it serves, so a larger batch enables more reuse. AF introduces two costs of its own. First, hidden states cross between the groups at every layer, which introduces smaller and more frequent network traffic. Second, the groups run as a pipeline over several microbatches, and their per-stage latencies must be matched almost exactly, since any imbalance stalls the pipeline and leaves one side idle [21]. Simulating AF therefore requires accurate modelling of per-layer inter-group traffic, together with the pipeline balance it depends on. In this paper, we focus on a four-way PDAF disaggregation, a finer-grained scheme that decomposes inference into four distinct stages, prefill-attention, prefill-FFN, decode-attention, and decode-FFN, which we refer to as disaggregation stages throughout the paper. It is worth noting that PDAF subsumes the two coarser-grained schemes as special cases: merging attention and FFN within each of the prefill and decode stages recovers conventional PD disaggregation, while merging prefill and decode within each of the attention and FFN stages recovers conventional AF disaggregation.

architecture because it provides a highly parameterizable and representative NPU substrate rather than a fixed accelerator instance. III. M OTIVATION Prefill and decode place fundamentally different demands on hardware: prefill is compute-bound at long sequence lengths, while decode is bound by the bandwidth needed to read KV cache and model weights for a single new token per user request. This mismatch is well understood and has already driven industry adoption of prefill/decode (PD) disaggregation, where the two phases run in separate hardware pools that are sized independently [14], [16], [32]. PD is, however, only the coarsest cut through the inference pipeline: it treats each phase as a monolithic unit. In fact, prefill and decode each contain two compute-heavy sub-layers, attention and feedforward (FFN), whose resource profiles diverge from each other just as sharply as prefill diverges from decode, though the character of that divergence differs by phase. In decode, attention’s cost is dominated by reading a KV cache that grows with context length, making it a bandwidthbound, capacity-hungry sub-stage. Because MoE sparsity lives entirely in the FFN, this is independent of sparsity; the KV footprint does depend heavily on the attention variant (MHA, GQA, or MLA [4]), but that axis is orthogonal to sparsity. Decode-FFN’s cost, by contrast, is dominated by streaming weight matrices, and in mixture-of-experts (MoE) models only a small fraction of experts activate per token, so reaching high FFN utilization requires aggregating a batch large enough that each active expert sees enough tokens, and the sparser the routing, the larger that batch must be [21]. At any fixed serving batch, then, FFN’s effective arithmetic intensity falls as sparsity increases, pushing it toward a different point on the compute/bandwidth roofline than attention. In prefill both sub-layers are compute-bound rather than bandwidth-bound, but their intensities still diverge: attention compute grows quadratically with sequence length (the O(n2 ) QK ⊤ and softmax(·)V terms), while FFN is a dense GEMM whose cost scales with token count and, for MoE, with routing. The attention/FFN mismatch is thus real within each phase, not only across the prefill/decode boundary. Under PD, the “decode” node still forces decode-attention (DA) and decode-FFN (DF), two sub-stages with different ideal compute-to-bandwidth ratios, onto the same physical device. Whatever hardware is chosen sizes correctly for one

B. Heterogeneous Accelerators and Memory Hierarchies Different inference stages have distinct compute, bandwidth, and capacity requirements, motivating heterogeneous accelerator and memory designs. Prior simulation frameworks such as LLMServingSim 2.0 [3] and MemExplorer [23] explore heterogeneous serving systems. However, LLMServingSim 2.0 focuses on fixed hardware and relies on profiling, while MemExplorer does not model distributed parallelism. HeteroPanacea instead enables exploration of custom NPUs, parallelization strategies, disaggregation, and mixed-precision quantization. Several accelerator architectures have been proposed for efficient neural-network and LLM inference, including PLENA [24], FlightLLM [26], MicroscalingQ [18], and the Coral NPU [6]. We adopt PLENA as the baseline compute

3

TABLE II: Device design space explored in the hardware sweep. Each memory technology contributes 2–5 representative (capacity, bandwidth) operating points.

and wastes capacity on the other. This residual, within-phase mismatch is exactly what motivates us to go one step further than PD to full four-way disaggregation (PDAF): separating prefill-attention, prefill-FFN, decode-attention, and decodeFFN onto independently specialized hardware pools. Disaggregating attention from FFN has been shown to enable independent scaling and heterogeneous deployment of the two sublayers [21]; PDAF extends that split across the prefill/decode boundary as well. This residual mismatch is not static: it is widening. Growing context lengths, driven by long-document processing, longchain-of-thought reasoning, and especially the rapid rise of agentic workloads (repeated tool calls and retrievals that reprefill an ever-growing context on nearly every turn), inflate KV-cache volume and therefore decode-attention’s bandwidth demand, while decode-FFN’s cost is governed instead by expert count and routing sparsity. As these two pressures increasingly diverge, the case for separating DA from DF only strengthens, and the same argument applies on the prefill side as prompts lengthen – there through attention’s quadratic compute rather than KV bandwidth. PD’s coarse split cannot capture this; only stage-level disaggregation can. Realizing PDAF, however, multiplies the design space: each of the four sub-stages can now be given its own hardware (compute throughput, memory capacity and bandwidth, interconnect), parallelism strategy, and replica count, under a shared power or cost budget, and the benefit of this extra granularity is not guaranteed to be worth its complexity: it depends on model architecture and, as we show, on the workload’s prefill/output ratio. Evaluating this joint space empirically, across enough hardware and workload points to know when PDAF’s extra specialization pays for itself over plain PD, is infeasible on real clusters. This motivates a fast, analytically grounded simulator that can sweep this space cheaply and identify the conditions under which the fourthway split, attention from FFN rather than just prefill from decode, is actually worth deploying.

TFLOPS: 25

100

250

2,500

10,000

20,000

Memory Type Capacity (GB) Bandwidth (GB/s) SRAM

0.1 0.1

1,000 10,000

HBM

16 40 80 140 190

1,000 2,000 3,500 4,800 8,000

DDR

4 16

20 60

LPDDR

8 32

100 200

GDDR

4 16

300 600

capturing the two resources that determine inference latency: compute throughput and memory bandwidth. a) Device parameterization.: A device is described by a peak compute rate F (TFLOPS, precision-agnostic peak), an exclusive memory technology class drawn from seven candidates, and that class’s paired per-device capacity C (GB) and bandwidth B (GB/s). The memory subsystem is parameterized by representative (C, B) points drawn from each technology’s empirical design space rather than a freeform continuous sweep. Table II lists the full device parameter set; the parallelism degrees each pool may be configured with are described in Section IV-D. b) Roofline execution time.: Every stage of inference (prefill-attention, prefill-FFN, decode-attention, decode-FFN) is decomposed into a FLOP count and a byte count from the model’s architecture (accounting for GQA/MHA vs. multihead latent attention (MLA) KV-cache layout, and dense vs. mixture-of-experts (MoE) FFN routing), and executed under the standard roofline bound:   β Φ t = max , , (1) Feff Beff

IV. I NFERENCE S IMULATION This section describes the models used to simulate inference, and the experiments run to validate them. A. Validation Platform

where Φ and β are the operation’s FLOP and byte counts and Feff , Beff are the effective compute rate and bandwidth after accounting for tensor- and pipeline-parallel scaling (Section IV-D). No queuing, kernel-launch, or occupancy effects are modeled below this per-stage granularity. The achieved TFLOPS predicted by the roofline model are compared against measured data in Figure 2. c) Power.: For GPU-based systems we use the power reported in the datasheet. NPU device power is modeled using data from existing GPU specifications, as described below. Device power is the sum of two independently modeled terms, P = Pcompute + Pmem , evaluated at each device’s peak specification. Pcompute follows a sub-linear power law, Pcompute = Pref (F/Fref )α , where the reference

Measurements in this section were collected on a single server: 8× NVIDIA B200, Intel Xeon 6960P, CUDA 12.8, PyTorch 2.10.0+cu128, NCCL 2.27.5, Python 3.12.13. Compute kernels use cuBLAS/cuBLASLt via torch.matmul (BF16) and torch._scaled_mm (FP8 E4M3); collectives use NCCL. Timings are CUDA-event based with warm-up and an adaptive iteration count sized to a fixed wall-clock budget. B. Compute Device (NPU) Model Rather than a cycle-accurate microarchitectural model, each accelerator in the design space is characterized by a small set of peak specifications and a roofline execution model built on top of them. This keeps the design space tractable while still

4

Fig. 2: Comparison of achieved TFLOPS of the roofline model compared with experimental data TABLE III: Measured versus simulated latency for intra-op (tensor) and inter-op (pipeline) parallelism, L = 8 layers, dmodel = 8192, M = 2048 tokens. Simulated values use each mode’s own calibrated bandwidth: a measured all_reduce rate for TP and a point-to-point rate for PP. Mode

#GPUs

Intra-op (TP)

Inter-op (PP)

Fig. 3: Comparison of simulated power of GPUs with datasheet TDP

Measured (ms)

Simulated (ms)

Sim/Real

1 2 4 8

1.23 34.47 32.62 38.16

0.98 34.33 34.44 39.05

0.79× 1.00× 1.06× 1.02×

1 2 4 8

1.45 2.42 4.32 7.90

0.98 1.85 3.61 7.12

0.67× 0.77× 0.84× 0.90×

TABLE IV: Measured versus simulated latency for two communication primitives

point (Pref , Fref ) is anchored to the H100 datasheet and the exponent α is fit to published TDP minus memorysubsystem power across three real accelerators spanning three device generations. The sub-linear exponent reflects diminishing marginal power cost per FLOP as process node and architecture improve. Pmem is a physics-based dynamic-plus-leakage estimate per memory technology, adapted from MemExplorer [23]: onchip technologies (SRAM, 3D-SRAM) use an on-chip power estimator; off-chip technologies (HBM, HBF, DDR, LPDDR, GDDR) use per-bit read/write energy scaled by bandwidth for the dynamic term, plus an idle-mode power density scaled by capacity for the leakage term. We compare simulated power with actual TDP of GPUs in Figure 3.

Primitive

Tokens

Measured (µs)

Simulated (µs)

Sim/Real

PP point-to-point

128 1024 8192

67 449 3,507

55 437 3,495

0.82× 0.97× 1.00×

EP all-to-all

128 1024 8192

63 383 2,891

55 437 3,495

0.86× 1.14× 1.21×

D2D intra-node, device-to-device bandwidth. This is the bandwidth used for tensor- and expert-parallel collectives and for pipeline-parallel activation hand-off between stages co-located on the same node. • N2N inter-node bandwidth, used specifically for the KVcache and activation transfers that cross a disaggregation boundary (Section IV-D) between physically separate stage pools. •

C. Interconnect Model: Device-to-Device (D2D) and Nodeto-Node (N2N) Communication is modeled with the same roofline philosophy as compute: a byte count derived analytically from the operation being performed, divided by a bandwidth appropriate to where that transfer physically occurs. Two distinct bandwidth domains are exposed per device:

5

Quantization Search

Hardware Search PLENA Compute and Memory Search

W/A/KV Quantization

Parallelism Search AttnPrefill AttnDecode

Prefill Server

TP

FFNPrefill FFNDecode

PP DP

MASE

Hardware Model

System Model

Accuracy Evaluator

Single device PLENA performance

Models for each device and parallelism scheduling

PLENA (Prefill Attn) D2D

PLENA (Prefill FFN)

Decode Server PLENA Cluster

PLENA (Decode Attn) D2D

KV Cache N2N

PLENA (Decode FFN)

Fig. 4: Overview of HeteroPanacea. TP - Tensor Parallelism, PP - Pipeline Parallelism, DP - Data parallelism, D2D - Device to device, N2N - Network to network

D. Parallelism Model

E. Simulator

Each stage pool (Section IV-B) is configured with four independent parallelism degrees: tensor (TP, t), pipeline (PP, p), data (DP, d), and expert (EP, e) subject to t · p · d ≤ the number of physical devices assigned to that pool. Parallelism is orthogonal to stage disaggregation: each pool independently chooses its own (t, p, d, e) and device allocation. a) Tensor parallelism (TP).: Each of the t devices in a TP group holds 1/t of every weight matrix (row- or columnsharded). Aggregate compute and bandwidth for the group therefore scale linearly with t, at the cost of a ring all-reduce after every attention output projection and every FFN downprojection (Section IV-C). b) Pipeline parallelism (PP).: The model’s layers are split into p sequential stages, each holding L/p layers on separate devices. Every inter-stage boundary additionally requires one point-to-point activation transfer (Section IV-C). Unlike TP, PP does not reduce per-request compute; its throughput benefit comes entirely from overlapping micro-batches across stages, not from per-request speedup. c) Expert parallelism (EP).: For MoE FFN layers, the E experts are sharded E/e per device. A token activated for k experts (top-k routing) must be dispatched to whichever device(s) own its routed experts and its output combined afterward, contributing two all-to-all collectives per MoE layer whose volume scales with the fraction of experts not colocated with the token, (e − 1)/e. d) Data parallelism (DP).: DP is handled outside the roofline/communication model entirely: d independent replicas, each a full copy of that stage’s model and hardware allocation, are instantiated as separate workers with disjoint request queues at the scheduler level. DP therefore has no communication term and no effect on per-request roofline time; it affects only aggregate throughput (via the number of independent servers) and memory footprint (weights are replicated d times). We validate the latency of communication primitives in Table IV, and of the parallelism modes built on them in Table III.

We combine the models above into an end-to-end inference simulator. Disaggregation modes. The simulator supports four deployment topologies of increasing stage separation: no disaggregation (prefill and decode share hardware), prefill/decode disaggregation, attention/FFN disaggregation, and a fully disaggregated mode separating all four sub-stages (prefillattention, prefill-FFN, decode-attention, decode-FFN) onto independent hardware pools. Each mode defines a fixed routing graph between worker pools, with explicit KV-cache and activation transfers modeled wherever stages are physically separated. Scheduling. The simulator advances through discrete events (arrivals, batch formation, stage completions, transfers, decode steps). Memory-aware continuous-batching schedulers govern prefill (greedy accumulation with timeout-based flushing) and decode (FIFO admission bounded by available KV-cache capacity). V. H ETERO PANACEA S EARCH A. Quantization search As a first method of optimizing the serving system, we explore the feasibility of reducing the precision of certain compute stages in order to reduce the compute and bandwidth required. A layer is only quantized if doing so results in no accuracy loss on the benchmark. Unlike the hardware search, precision cannot be ranked analytically, since accuracy loss is only observable by running the quantized model; this search is therefore empirical rather than roofline-driven. Search space. We assign an independent element bit-width qs ∈ Q to each of the four stages s ∈ {PA, PF, DA, DF}, giving a precision assignment q = (qPA , qPF , qDA , qDF ) ∈ Q4 (e.g. Q = {4, 8, 16}). Both weights and activations are quantized to the MXint format: a microscaling integer format sharing one exponent across a block of B elements, with ws set independently per tensor role (weights, activations, KV cache) and, within attention, per sub-operation (QK ⊤ , AV ) [20]. A single stage’s precision is therefore itself a small

6

vector of block/width settings rather than one scalar, which we summarize as ws for the search. Enforcing the assignment. Applying q requires distinguishing quantization along two axes: by layer type (attention vs. FFN) and by inference phase (prefill vs. decode). The former is handled natively by the MASE [28] quantization pass, which applies a distinct config per matched layer pattern. The latter is not: MASE has no notion of prefill vs. decode, so we attach a PhaseAutoSwitch hook to each quantized layer that inspects the sequence length L of the tensor passing through it (L > 1 ⇒ prefill config, L = 1 ⇒ decode config) and switches to the corresponding per-phase width at runtime. Search procedure. For each candidate q ∈ Q4 we serve the quantized model and evaluate task accuracy A(q) against a held-out benchmark suite. We evaluate uniform fp16 and fp8 assignments as baselines, and compare them against settings in which one stage is reduced to 4 bits with the remaining three fixed at 8.

modes over the custom NPU design space (Section VI-B) and then repeat the sweep over commercially available GPUs (Section VI-C), which lets us separate what disaggregation buys from what stage-specialized hardware buys. Section VI-D inspects the per-stage hardware the search selects in each case to explain the difference. Finally, we isolate the contribution of individual model-architecture choices through a controlled ablation (Section VI-E) and characterize the accuracy cost of per-stage quantization (Section VI-F). A. Experimental setup Models. The experiments span eight models across dense and mixture-of-experts architectures at a range of scales, from Llama-3.1-405B [7] and DeepSeek-V4 [5] to GPTOSS [13], Llama4-Scout, Llama4-Maverick [10], Qwen3235B-A22B [17], and GLM-4.6 [30]. Workload. The workload is parameterized by the prefill/output token ratio (I/O) with output length held fixed at 1000 tokens and input length scaled accordingly; perrequest input and output lengths are then drawn from a normal distribution centered on these targets to introduce realistic variability. The length of the output sequences has been determined by running BFCL [15] and GSM8K [27] on real models. For each config we simulate 500 requests at a fixed rate of 125 requests/second. The setup was determined experimentally to provide high utilization of the hardware without overloading. Disaggregation modes. We compare four modes: ND (non-disaggregated) co-locates all four stages on a single homogeneous pool; PD disaggregates prefill from decode; AF disaggregates attention from FFN; PDAF applies both splits, yielding four independently provisioned stages (prefillattention, prefill-FFN, decode-attention, decode-FFN). The sweeps of Section VI-B and Section VI-C have been run with full precision models; the compute cost of quantizing MoE models was too high to run the search.

B. Hardware and parallelism search Given a workload specification, the search finds the optimal hardware and parallelism setup. It proceeds in two phases. Phase 1 ranks hardware candidates per stage analytically, without simulation, using the roofline throughput model to compute a capacity score σ(hw) = d ·

tput(B) , Λs

(2)

the provisioned throughput of d DP replicas divided by the stage’s demand rate Λs (λ for prefill stages, λO for decode stages). Candidates exceeding Pbudget or unable to hold model weights are discarded. Phase 2 allocates the power budget across the mode’s k stages. For each slice, the cheapest per-stage candidate reaching σ ≥ 1 is chosen, unspent budget is redistributed to the weakest stage, and the joint quality of the resulting configuration is its bottleneck,

B. Comparison of disaggregation modes for NPUs σjoint = min σi . i=1,...,k

(3)

The NPU search explores a synthetic device design space (Table II): compute and memory technology are varied independently and combined freely up to a fixed installed power budget. We assume each disaggregated cluster is placed on NVSwitch [12] – allowing up to 72 devices of 3600 GB/s bidirectional bandwidth – and that the NVSwitch instances are connected with Infiniband of 50 GB/s. Devices in different disaggregation stages may differ: the Vera-Rubin platform (Groq LPU for prefill and Rubin GPU for decode) falls within our search space. Figure 6 shows the ranking of serving strategies at each workload ratio. At low and balanced prefill/output ratios (I/O = 0.01 and I/O = 1), ND is the strongest configuration for every model tested: PD, AF, and PDAF all remain below 1.00× ND across the board, with PD the closest (up to 0.89× for GPT-OSS at I/O = 1) but never crossing parity. Disaggregation overhead simply outweighs any benefit when prefill work is small relative to decode.

Unique configurations are ranked by σjoint and only the top-K are actually simulated end-to-end. The winning configuration for each mode is the simulated candidate with the highest achieved throughput. cfg⋆M = arg max tok/s(cfg),

(4)

cfg ∈ top-K

The two searches compose into one optimal-system-design pipeline: the quantization search fixes q⋆ , the accuracyvalidated per-stage precision; the hardware/parallelism search then takes q⋆ as dtype bytes per stage and finds the throughput-optimal hardware and parallelism configuration. VI. E VALUATION We evaluate PDAF disaggregation across a range of workload and hardware characteristics to understand where and why its benefits hold. We first sweep the disaggregation

7

2.5

Figure 7 describes results for the GPU experiment. The crossover into a disaggregation-favoring regime happens earlier than on the NPU design space: already at I/O = 0.01, PD exceeds ND for all eight models (1.18–2.50×) and wins outright in every case, where the equivalent NPU sweep places ND ahead of every disaggregated mode at the same ratio. The picture is also markedly less uniform. PD is above parity for 8/8 models at I/O = 0.01 and 7/8 at I/O = 100, but only 4/8 at I/O = 1, where it ranges from 0.01× to 2.64×; PDAF is above parity for 6/8, 2/8, and 4/8 models at the three ratios respectively. AF never exceeds ND at any ratio or for any model (best case 0.96×). Notably, the finer four-way split is not rewarded here: PD matches or outperforms PDAF for six of eight models at I/O = 100, the reverse of the NPU ranking at the same ratio.

crossover

PDAF (×ND)

2.0 1.5 1.0

ND parity

0.5 0

0.01

0.1

1

10

100

1000

Prefill/output ratio I/O

Fig. 5: PDAF throughput relative to ND, averaged across all models in the NPU sweep, over nine prefill/output ratios (O=1000 tokens fixed). ND is 1.00× by definition. The shaded decade brackets the crossover: PDAF is below parity at I/O = 1 and well above it at I/O = 10. Markers are drawn at all nine swept ratios; the axis labels decades only.

D. Per-stage hardware allocation Table V shows the hardware configuration the search selects for each stage at a prefill/output ratio of 100, and explains the divergence between the two design spaces. Under traditional NPU technology, every stage uses HBM memory with only two exceptions: Llama-3.1-405B’s PA stage and Llama4-Scout’s PF stage, which instead use LPDDR. The specific HBM operating point still varies considerably from stage to stage. The decode stages (DA, DF) always claim the highest available capacity and bandwidth tier, but at opposite ends of the compute range – decode-attention is given the lowest compute tiers (25–250 TFLOPS) while decode-FFN takes 2.5k; the prefill stages (PA, PF), by contrast, primarily maximize compute at 20k TFLOPS while accepting the smallest capacity tiers. The GPU allocations show no such spread, because GPUs fix compute and memory bandwidth together rather than allowing them to be tuned independently per stage. The search has little room to re-balance: 31 of the 32 PDAF stage assignments are H100, the remaining one an A100, so nearly every stage receives the same compute-to-bandwidth ratio regardless of whether it is attention- or FFN-bound. With this reduced design-space flexibility, no single mode can consistently match each stage’s hardware to its own bottleneck, and the relative benefit of PD, AF, and PDAF instead varies with the specific compute/memory demands of each model and workload ratio. It also explains why the four-way split does not pay off on GPUs: splitting attention from FFN only helps when the two can be given genuinely different hardware, which is precisely what the NPU design space permits and the GPU catalogue does not.

At I/O = 100, the ranking shifts. Both disaggregated prefill/decode strategies clear parity almost universally: PDAF exceeds ND for all eight models (1.05–1.92×) and PD for seven of eight, with DeepSeek-V4-Pro (0.95×) the sole configuration still below ND. PDAF is the best mode for six of the eight models, reaching 1.81× and 1.77× for Llama4 Maverick and Scout. AF remains the weakest mode at every ratio tested, never exceeding ND; at I/O = 100 it spans 0.20– 0.65×, its best showing but still far from parity. The three ratios of Figure 6 bracket the crossover into a disaggregation-favoring regime but do not locate it. Figure 5 sweeps nine ratios and reports PDAF’s benefit averaged across models. The transition is abrupt and confined to a single decade: PDAF is still at 0.48× at I/O = 1 and already at 2.10× by I/O = 10. It is also not monotonic thereafter. The benefit plateaus between 1.5× and 1.8× through I/O = 500 and then falls to 1.27× at I/O = 1000, consistent with prefill growing large enough that the decode stages it feeds are no longer the system bottleneck, so hardware provisioned separately for them is increasingly idle. Agentic workloads sit near the middle of this range rather than at its extremes, which is where the four-way split is most productive. C. Comparison of disaggregations for heterogeneous GPUs We repeat the sweep across commercially available hardware: each stage pool is built from one or more identical AWS EC2 GPU instances (H100, A100, L40S, L4, A10G, T4, V100, M60), with datasheet-accurate compute, memory, and TDP, and is constrained by a fixed hourly cost budget (USD/hr) rather than a power budget. The interconnect bandwidth here is not uniform: single-instance pools use intra-node bandwidth (NVLink/PCIe), while multi-instance pools are bottlenecked by the slower inter-node fabric (EFA/NIC). As in the NPU search, each disaggregation stage may be assigned a different device – in PDAF, every stage (prefill-attention, prefill-FFN, decode-attention and decode-FFN) can use a different GPU type.

E. Model architecture ablation To isolate individual mechanisms, we ran a controlled ablation matrix: five architecture factors, each swept independently while holding every other model parameter fixed, repeated across three baseline models. Each factor is swept over four points, and every configuration is evaluated at a fixed prefill/output ratio of 100 under the same fixed hardware

8

(a) Decode-dominated.

2.5 2 1.5 1

(b) Balanced.

Qwen3

GPT-OSS

GLM

L4-Scout

L4-Mav

L-405B

0

DS-Flash

0.5 DS-Pro

Qwen3

GPT-OSS

GLM

L4-Scout

0

L4-Mav

Qwen3

GPT-OSS

GLM

L4-Scout

L4-Mav

L-405B

DS-Flash

0

1 0.5 L-405B

1 0.5

2 1.5

DS-Flash

2

I/O = 100 (I = 100k)

2.5

DS-Pro

PDAF

Throughput (×ND)

I/O = 1 (I = 1k) Throughput (×ND)

AF

1.5

DS-Pro

Throughput (×ND)

I/O = 0.01 (I = 10) PD

2.5

(c) Prefill-dominated.

Fig. 6: NPU throughput relative to No Disaggregation ND (×ND) for each disaggregation mode across models and workload profiles (O=1000 tokens fixed; I/O varies the prefill length). The dashed line marks ND parity, which is 1.00× by definition; bars above it beat non-disaggregated serving. All three panels share a common y-axis. Models, left to right: DeepSeek-V4-Pro, DeepSeek-V4-Flash, Llama-3.1-405B, Llama-4-Maverick, Llama-4-Scout, GLM-4.6, GPT-OSS, Qwen3-235B-A22B. TABLE V: Per-stage device allocation for PDAF disaggregation at I/O = 100 (I = 100k, O = 1k), comparing the NPU design space against commercial GPU clusters. Each stage lists the device count N , the per-device memory operating point (technology, capacity in GB / bandwidth in GB s−1 ), and peak compute in TFLOPS; the value beside each row label is the total device count for that cluster. The NPU search moves memory and compute independently – decode stages take the top memory tier (190/8.0k) at the two extremes of the compute range, prefill stages take the top compute tier (20k) at the smallest capacities. The GPU catalogue ties the two together, so every stage receives the same operating point and only N can vary. Model

PA N Memory

PF

DA

TFLOPS N Memory

TFLOPS N Memory

DF TFLOPS N Memory

TFLOPS

DS-V4-Pro

NPU (80) 16 HBM 16/1.0k GPU (100) 32 H100-80/3.4k

20k 16 HBM 140/4.8k 989 32 H100-80/3.4k

20k 24 HBM 190/8.0k 989 4 H100-80/3.4k

250 24 HBM 190/8.0k 989 32 H100-80/3.4k

2.5k 989

DS-V4-Flash

NPU (80) 24 HBM 16/1.0k GPU (112) 64 H100-80/3.4k

20k 8 HBM 80/3.5k 989 16 H100-80/3.4k

20k 24 HBM 190/8.0k 989 16 H100-80/3.4k

250 24 HBM 190/8.0k 989 16 H100-80/3.4k

2.5k 989

L-405B

NPU (88) 24 LPDDR 32/200 GPU (112) 64 H100-80/3.4k

20k 8 HBM 80/3.5k 989 16 H100-80/3.4k

20k 32 HBM 190/8.0k 989 16 H100-80/3.4k

100 24 HBM 190/8.0k 989 16 H100-80/3.4k

2.5k 989

L4-Mav

NPU (108) 16 HBM 16/1.0k GPU (112) 32 H100-80/3.4k

20k 4 HBM 140/4.8k 989 16 H100-80/3.4k

20k 64 HBM 190/8.0k 989 32 H100-80/3.4k

100 24 HBM 190/8.0k 989 32 H100-80/3.4k

2.5k 989

L4-Scout

NPU (104) 16 HBM 16/1.0k GPU (112) 32 H100-80/3.4k

20k 8 LPDDR 32/200 989 16 H100-80/3.4k

20k 64 HBM 190/8.0k 989 32 H100-80/3.4k

100 16 HBM 190/8.0k 989 32 H100-80/3.4k

2.5k 989

GLM-4.6

NPU (104) 16 HBM 16/1.0k GPU (112) 32 H100-80/3.4k

20k 8 HBM 80/3.5k 989 16 H100-80/3.4k

20k 64 HBM 190/8.0k 989 32 H100-80/3.4k

100 16 HBM 190/8.0k 989 32 H100-80/3.4k

2.5k 989

GPT-OSS

NPU (113) 16 HBM 16/1.0k GPU (120) 32 H100-80/3.4k

10k 989

1 HBM 190/8.0k 8 A100-40/1.6k

20k 64 HBM 190/8.0k 312 64 H100-80/3.4k

25 32 HBM 190/8.0k 989 16 H100-80/3.4k

2.5k 989

Qwen3-235B

NPU (92) 24 HBM 40/2.0k GPU (112) 64 H100-80/3.4k

20k 4 HBM 80/3.5k 989 16 H100-80/3.4k

20k 48 HBM 190/8.0k 989 16 H100-80/3.4k

100 16 HBM 190/8.0k 989 16 H100-80/3.4k

2.5k 989

search and installed-power budget used elsewhere in this work. The factors are the following:

testing whether capacity growth alone is the cheaper of the two. • FFN size - moves both the active work (as Sparsity does) and the resident footprint (as Capacity does). Since only the product ex · k enters the active terms, doubling ffn_expansion should match doubling num_active_experts unless the added footprint changes the outcome.

Precision - FP4 through FP32. The only lever that shifts every stage’s arithmetic intensity at once. • Decode attention arithmetic intensity - MLA latent width, which sets KV bytes per token and hence decode-attention arithmetic intensity. • Sparsity - experts activated per token, i.e. active FFN compute and the power contention it creates. • Capacity - total FFN weight footprint with active compute pinned. The deliberate counterpart to Sparsity: it separates weight capacity pressure from active compute pressure, •

We report four findings. Finding 1: PDAF’s benefit grows monotonically with decode-attention KV traffic. kv_lora_rank is the width of the compressed latent KV vector that MLA-style attention

9

(a) Decode-dominated.

(b) Balanced.

2 1.5 1

Qwen3

GPT-OSS

GLM

L4-Scout

L4-Mav

0

L-405B

0.5 DS-Flash

Qwen3

GPT-OSS

GLM

L4-Scout

0

2.5

DS-Pro

1 0.5 L4-Mav

Qwen3

GPT-OSS

GLM

L4-Scout

L4-Mav

L-405B

DS-Flash

0

2 1.5

L-405B

1 0.5

2.5

DS-Flash

2 1.5

I/O = 100 (I = 100k) Throughput (×ND)

I/O = 1 (I = 1k)

PDAF

DS-Pro

AF

Throughput (×ND)

PD

DS-Pro

Throughput (×ND)

I/O = 0.01 (I = 10) 2.5

(c) Prefill-dominated.

Fig. 7: GPU throughput relative to No Disaggregation ND (×ND) for each disaggregation mode across models and workload profiles (O=1000 tokens fixed; I/O varies the prefill length). Axes match Figure 6 so the two design spaces can be compared directly. Models are ordered as in Figure 6. TABLE VI: Architecture ablation: PDAF throughput relative to ND (×ND) at I/O = 100, sweeping five factors on three MoE baselines spanning low, mid, and high decodeattention arithmetic intensity. Each factor varies one parameter with all others held at the baseline value: Precision sweeps dtype_bytes (bytes per element), DA AI sweeps kv_lora_rank (latent width), Sparsity sweeps num_active_experts, Capacity sweeps num_experts, and FFN size sweeps ffn_expansion, the latter three as multiples of each model’s baseline. Bold marks the best swept value per model per factor. Factor

Precision

DA AI

Sparsity

Capacity

FFN size

Model GPT-OSS GLM-4.6 DS-V4-Flash GPT-OSS GLM-4.6 DS-V4-Flash GPT-OSS GLM-4.6 DS-V4-Flash GPT-OSS GLM-4.6 DS-V4-Flash GPT-OSS GLM-4.6 DS-V4-Flash

(1.17× → 1.96×). Note that this direction is one of decreasing decode-attention arithmetic intensity: FLOPs per decode step are independent of the rank while KV bytes scale with it, so over this sweep DA AI falls from 10.7 to 1.8. The mechanism is visible directly in the roofline structure of decode-attention. Its per-step byte count carries a KV term proportional to context length and a weight term that is independent of batch, so its arithmetic intensity is set by the ratio between them. At kv_lora_rank = 128 the KV term is small enough that decode-attention is weight-read dominated and therefore structurally indistinguishable from decode-FFN: the two stages want the same hardware, the attention/FFN split has nothing to separate, and PDAF falls below the non-disaggregated baseline for two of the three models (0.93×, 0.81×). As the rank grows, KV traffic comes to dominate and decode-attention’s requirement diverges from decode-FFN’s, which is precisely the asymmetry the four-way split exists to exploit. Finding 2: memory-capacity growth is essentially free. Growing num_experts while holding num_active_experts fixed leaves every stage’s pertoken compute and bandwidth demand unchanged; FLOPs and weight-bytes-read in the decode-FFN roofline depend only on the number of active experts, and increasing the total raises only the capacity needed to hold all experts’ weights resident. PDAF’s benefit is correspondingly insensitive to it. Quadrupling the expert count relative to baseline moves GLM-4.6 by +3% and DeepSeekV4-Flash by under 0.5% (flat at 1.41× at every point), indicating capacity never becomes the binding constraint for these configurations. GPT-OSS is non-monotonic, and notably its worst point is the smallest expert count, 41% below its baseline (1.07× versus 1.80× at 0.5× experts), while 2× and 4× sit within 6% of baseline. Finding 3: active-compute growth is the only factor that reduces PDAF’s benefit. Growing num_active_experts raises decode-FFN’s per-token compute and its per-step weight traffic together, and this is the one lever that collapses PDAF’s advantage. Relative to baseline, DeepSeekV4-Flash loses 69% at 4× active experts (1.41× → 0.43×) and 73% at 8× (0.38×). GLM-4.6 loses 17% at 4× (1.60×) before collapsing by 60% at 8× (0.78×). Only GPT-OSS is insensitive, staying within

Swept value → PDAF 0.5

1

2

4

2.43 1.29 0.85

1.63 1.29 0.95

1.80 1.92 1.41

1.54 2.20 1.47

128

512

1024

4096

0.93 0.81 1.17

1.23 0.87 1.41

1.64 1.50 1.80

1.85 2.00 1.96

0.25×

1×

4×

8×

1.77 1.80 1.98

1.80 1.92 1.41

1.88 1.60 0.43

1.69 0.78 0.38

0.5×

1×

2×

4×

1.07 1.92 1.42

1.80 1.92 1.41

1.70 1.92 1.41

1.75 1.98 1.41

0.25×

0.5×

1×

2×

1.75 2.00 1.92

2.00 1.87 1.64

1.80 1.92 1.41

1.57 1.63 1.33

caches per token, and thus sets KV bytes per token per layer. Sweeping kv_lora_rank from 128 to 4096 moves PDAF’s benefit monotonically upward in all three models, by +100% for GPT-OSS (0.93× → 1.85×), +147% for GLM4.6 (0.81× → 2.00×), and +67% for DeepSeekV4-Flash

10

TABLE VII: QWEN 32B quantization results on BFCL and GSM8k (20%).

6% of baseline across the whole range (1.69×–1.88×); it is also the model with the lowest baseline active-expert count, so the same relative multiple leaves it at a lower absolute compute demand. Sweeping ffn_expansion, the other lever on the same ex · k compute proxy, reproduces the ordering but not the magnitude: over 0.25× to 2×, DeepSeekV4-Flash declines steadily by 31% (1.92× → 1.33×) while GLM-4.6 and GPTOSS decline by 19% and 10% — an order of magnitude milder than the 73% collapse the active-expert lever produces. Finding 4: reduced precision is not uniformly beneficial. Sweeping dtype_bytes splits the three models rather than moving them together. Narrowing precision from 4 to 0.5 bytes per element improves GPT-OSS by +57% (1.54× → 2.43×) but costs GLM-4.6 41% (2.20× → 1.29×) and DeepSeekV4Flash 42% (1.47× → 0.85×), the latter falling below parity with the non-disaggregated baseline. Precision scales the weight-byte term of every stage simultaneously, so narrowing it raises the arithmetic intensity of all four stages at once; whether that helps PDAF depends on whether it widens or narrows the gap between decode-attention and decode-FFN, which is what the split monetizes. For models whose decodeattention is already KV-dominated, shrinking weight bytes leaves the KV term untouched and preserves the asymmetry; where weight traffic is what distinguished the stages, removing it collapses the distinction.

Configuration

BFCL

GSM8k (20%)

Baseline 8/8/8/8 4/4/4/4

18% 21% 6%

78% 75% 11%

8/4/8/8 8/8/4/8 4/8/8/8 8/8/8/4

20% 11% 12% 20%

47% 79% 80% 15%

VII. C ONCLUSIONS We present an event-driven simulator for stagedisaggregated LLM inference that treats prefill-attention, prefill-FFN, decode-attention and decode-FFN as independently provisionable stages, each with its own hardware, parallelism and numeric precision, and searches the resulting design space under a power or cost budget. We validate its component models against an 8×B200 node. Using the simulator, we evaluate stage disaggregation on both commercial GPUs and custom accelerators, and generalize the result across eight models and five orders of magnitude of workload profile. A factorial ablation isolates which architectural properties drive the benefit, and a quantization study shows that the precision-sensitive stage is a property of the workload. Several limitations bound these results. Validation covers the simulator’s components rather than its end-to-end serving behavior, leaving scheduling and batching effects unverified, and the quantization study covers one model and two tasks without repeated runs, so its stage asymmetry should be read as a direction rather than a calibrated magnitude. The results of the GPU experiment need to be considered with the fact that they are based on a single particular inference provider offer and can vary with different pricing models. These bounds point directly at the next steps. Validating the simulator against a deployed disaggregated serving stack would close the gap between component accuracy and end-to-end behavior, and is the prerequisite for trusting the scheduling and batching effects the current model abstracts away. The quantization search is the other open front: the accuracy evaluation, rather than the simulator, is what makes it expensive, so extending it beyond one model and four stagewise assignments depends on cheaper accuracy proxies rather than on faster simulation. Finally, the ablation suggests that PDAF’s benefit is largely predicted by a model’s decodeattention arithmetic intensity and its active FFN compute. If that relationship holds across a wider set of architectures, the four-way split could be ruled in or out from a model’s configuration alone, without a hardware search at all.

F. Quantization sensitivity Evaluating the quantization search is considerably more expensive than the throughput sweeps above, since each candidate assignment requires a full accuracy evaluation rather than a single simulator pass. We therefore restrict this analysis to a targeted sensitivity study on one smaller model, Qwen3.532B, and characterize the accuracy cost of reducing precision at each stage in isolation. We consider two prefill-dominated reasoning workloads, BFCL and a subset of GSM8K, which differ substantially in shape: BFCL averages 11,174 prefill and 1,472 decode tokens per request, against 995 and 313 for GSM8K, an approximately 11× difference in context length. Beginning from an 8-bit baseline applied uniformly to all four stages (75% on GSM8K, 21% on BFCL), we reduce exactly one stage to 4 bits and re-evaluate. Uniform 4-bit quantization collapses accuracy on both tasks (11% and 6%), confirming that 4 bits is too aggressive when applied globally. The single-stage reductions, however, expose a pronounced asymmetry between stage classes. Reducing either FFN stage, prefill or decode, is severely damaging on GSM8K (47% and 15% accuracy respectively, the latter comparable to the uniform 4-bit collapse) while leaving BFCL essentially unchanged (20% in both cases, at baseline). Reducing either attention stage inverts the pattern: BFCL accuracy is approximately halved (11–12%), whereas GSM8K remains at or marginally above the 8-bit baseline (79–80%). Which stage tolerates low precision is therefore a property of the workload rather than of the model alone, and a single global precision choice necessarily sacrifices one of the two.

11

R EFERENCES

E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783 [8] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu, “Webvoyager: Building an end-to-end web agent with large multimodal models,” in ACL, 2024. [9] Z. Li, J. Liu, Z. Xu, Y. Zhang, T. Rabbani, and C. Zhang, “Not all prefills are equal: Ppd disaggregation for multi-turn llm serving,” 2026. [Online]. Available: https://arxiv.org/abs/2603.13358 [10] Meta AI, “The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,” https://ai.meta.com/blog/llama-4-multimodalintelligence/, 2025, llama 4 Scout and Llama 4 Maverick. [11] J. Nie, H. Wu, Y. Lai, Z. Cao, C. Zhang, B. Lou, E. Wang, J. Cheng, T. M. Jones, R. Mullins, R. Antonova, and Y. Zhao, “Kernelcraft: Benchmarking for agentic close-to-metal kernel generation on emerging hardware,” arXiv preprint arXiv:2603.08721, 2026. [12] NVIDIA Corporation, “NVLink and NVLink switch,” https://www. nvidia.com/en-us/data-center/nvlink/, 2026, accessed: 2026-07-30. [13] OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025. [Online]. Available: https://arxiv.org/abs/2508.10925 [14] P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024. [Online]. Available: https://arxiv.org/abs/2311.18677 [15] S. G. Patil, H. Mao, C. Cheng-Jie Ji, F. Yan, V. Suresh, I. Stoica, and J. E. Gonzalez, “The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models,” in ICML, 2025. [16] R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,” in FAST, 2025. [17] Qwen Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388 [18] A. Ramachandran, S. Kundu, and T. Krishna, “Microscopiq: Accelerating foundational models through outlier-aware microscaling quantiza-

[1] K. Aubrey and F. Ghodsian, “Inside NVIDIA Groq 3 LPX: The LowLatency Inference Accelerator for the NVIDIA Vera Rubin Platform,” https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-lowlatency-inference-accelerator-for-the-nvidia-vera-rubin-platform/, Mar. 2026, nVIDIA Technical Blog. Accessed: 2026-05-13. [2] H. Chae, N. Kim, K. T.-i. Ong, M. Gwak, G. Song, J. Kim, S. Kim, D. Lee, and J. Yeo, “Web agents with world models: Learning and leveraging environment dynamics in web navigation,” in ICLR, 2025. [3] J. Cho, H. Choi, G. Heo, and J. Park, “Llmservingsim 2.0: A unified simulator for heterogeneous and disaggregated llm serving infrastructure,” 2026. [Online]. Available: https://arxiv.org/abs/2602. 23036 [4] DeepSeek-AI, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437 [5] DeepSeek-AI, “Deepseek-v4 technical report,” https://huggingface.co/ collections/deepseek-ai/deepseek-v4, 2026, preview release; includes DeepSeek-V4-Pro and DeepSeek-V4-Flash. [6] Google Research, “Coral npu: A full-stack platform for edge ai,” 2025. [7] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. AlDahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood,

12

tion,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 1193–1209. [19] S. Rando, L. Romani, A. Sampieri, L. Franco, J. Yang, Y. Kyuragi, F. Galasso, and T. Hashimoto, “Longcodebench: Evaluating coding LLMs at 1m context windows,” in COLM, 2025. [20] B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, S. Dusan, V. Elango, M. Golub, A. Heinecke, P. James-Roxby, D. Jani, G. Kolhe, M. Langhammer, A. Li, L. Melnick, M. Mesmakhosroshahi, A. Rodriguez, M. Schulte, R. Shafipour, L. Shao, M. Siu, P. Dubey, P. Micikevicius, M. Naumov, C. Verrilli, R. Wittig, D. Burger, and E. Chung, “Microscaling data formats for deep learning,” 2023. [Online]. Available: https://arxiv.org/abs/2310.10537 [21] StepFun, “Step-3 is large yet affordable: Model-system co-design for cost-effective decoding,” 2025. [Online]. Available: https://arxiv.org/ abs/2507.19427 [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008. [23] H. Wu, Z. Cao, Y. Lai, B. Lou, J. Nie, C. Xiao, T. Adeniran, P. Forys, K. Johar, C. Wright, J. Liu, K. Shi, N. D. Lane, R. Antonova, J. Cheng, T. Jones, A. Zhao, and R. Mullins, “Memexplorer: Navigating the heterogeneous memory design space for agentic inference npus,” 2026. [Online]. Available: https://arxiv.org/abs/2604.16007 [24] H. Wu, C. Xiao, J. Nie, X. Guo, B. Lou, J. T. H. Wong, Z. Mo, C. Zhang, P. Forys, C. Ai, T. Adeniran, W. Luk, H. Fan, J. Cheng, T. M. Jones, R. Antonova, R. Mullins, and A. Zhao, “Combating the memory walls: Optimization pathways for long-context agentic llm inference,” arXiv preprint arXiv:2509.09505, 2026. [25] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu, “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” in NeurIPS, 2024. [26] S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang, Y. Dai, J. Li, Z. Wang, R. Zhang, K. Wen, X. Ning, and Y. Wang, “Flightllm: Efficient large language model inference with a complete mapping flow on fpgas,” 2024. [Online]. Available: https://arxiv.org/abs/2401.03868 [27] Z. Zeng, P. Chen, S. Liu, H. Jiang, and J. Jia, “MR-GSM8K: A metareasoning benchmark for large language model evaluation,” in ICLR, 2025. [28] C. Zhang, J. Cheng, Z. Yu, and Y. Zhao, “Mase: An efficient representation for software-defined ml hardware system exploration.” [29] H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in Proceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 1080–1096. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00082 [30] Zhipu AI, “Glm-4.6,” https://z.ai/blog/glm-4.6, 2025. [31] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024. [32] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in OSDI, 2024.

13

Related documents

• MoSE: Mode-Switching Expander for Mixed LLM Training and Inference#1144105
• MoSE: Mode-Switching Expander for Mixed LLM Training and Inference#1136800
• NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference#259408
• Rethinking On-Policy Distillation of Large Language Models II: One Training Example#639704
• Understanding LLM Parameter Update Sparsity through the Lens of Fisher#1131104