HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

reason over complex data [14, 56–58]. As advances in quantization, distillation, and efficient model design push increasingly capable models from datacenters to user-facing devices, edge inference is becoming attractive not only for privacy and cost, but also for latency-sensitive applications such as personal assistants, AR/VR, robotics, local search, and industrial automation [36, 47, 66, 78, 80, 84–86]. These deployments are especially sensitive to time-to-firsttoken and sustained throughput because the model often sits inside an interactive control loop rather than serving one-shot requests. As a result, practical edge inference is constrained not only by model size, but also by the growing cost of serving long-context workloads under tight latency and power limits. Modern edge platforms attempt to meet this demand through heterogeneous system-on-chips (SoCs) that combine CPUs, integrated graphics processing units (iGPUs), and neural processing units (NPUs) [7, 15, 40, 68]. Yet current software stacks still expose this heterogeneity mostly through coarse backend choices rather than fine-grained cross-accelerator workload scheduling. AMD exposes GPU and NPU execution through separate software stacks, ROCm and ONNX Runtime [10, 13], respectively; Apple Core ML [16] allows developers to choose compute units while leaving dispatch to the runtime; and Qualcomm QNN / AI Engine Direct [71] likewise relies on explicit backend targets. On unifiedmemory edge SoCs such as those mentioned above, this abstraction still leaves performance on the table. Unlike datacenter settings with clear host-device memory separation [46, 76], unified memory eliminates host-device copy overheads and makes fine-grained cross-accelerator mapping practical. But this opportunity is rarely explicit in the original LLM graph: it must first be exposed through careful scheduling, and then mapped onto the right accelerator under tight thermal and power constraints. The key open question is whether heterogeneity itself improves end-to-end LLM inference once confounding factors are removed. Prior edge LLM systems show that heterogeneous SoC execution can be useful, but their gains are often coupled with model restructuring, sparsity, outlier routing, or quantization changes [89, 90]. GPU–NPU systems such as HeteroInfer [21] study heterogeneous mapping more directly, but still focus primarily on greedy tensorlevel placement on highly imbalanced mobile SoCs. More broadly, prior work does not jointly study two complementary sources of heterogeneous opportunity, overlapping independent work across the graph and splitting individual operations across accelerators. This gap matters most on unbalanced SoCs, where a policy that only optimizes local operation placement can either mostly reproduce

Modern edge system-on-chips (SoCs) increasingly combine CPUs, integrated graphics processing units (iGPUs), and neural processing units (NPUs) to meet the growing compute demands of edge AI. However, existing LLM runtimes do not fully optimize this heterogeneity. Production frameworks typically make only coarse device-level decisions, while prior research often focuses on optimizing operations locally rather than holistically accelerating the LLM task graph. As a result, much of the available heterogeneous opportunity remains unrealized, especially on unified-memory edge platforms where performance depends not only on where an operation runs, but also on how execution is coordinated across the task graph. We argue that edge LLM inference on such systems should be treated as a heterogeneity-first scheduling problem rather than a coarse-grained accelerator-mapping problem, and present HeteroMosaic to realize this view. We first develop a heterogeneous roofline model to characterize the potential benefit of heterogeneity across systems with different relative iGPU and NPU capabilities. Guided by this analysis, HeteroMosaic decomposes LLM execution into dependency-preserving micro-batches that expose additional opportunities to overlap work across accelerators. It then uses traceguided optimizations to jointly tune this new schedule with device allocation so that critical stages complete earlier despite real system effects such as device variation, unified-memory contention, DVFS, and NPU runtime behavior. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen™ AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On a balanced system, across select offthe-shelf models, HeteroMosaic achieves up to 1.73× speedup over an iGPU baseline, 1.78× over an NPU baseline, and up to 2.05× over strong existing frameworks such as llama.cpp, while using up to 45.3% less energy. Compared with prior heterogeneous edge AI solutions, HeteroMosaic improves performance by up to 2.35×.

Keywords edge AI, heterogeneous computing, large language models, neural processing units, runtime scheduling, unified memory

Mehdi Saeedi

Advanced Micro Devices, Inc. Advanced Micro Devices, Inc.

Introduction

Large language models (LLMs) are transforming computing systems from passive tools into active agents that interpret, generate, and ∗ To appear in the Proceedings of MICRO 2026.

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Figure 1: Overview of HeteroMosaic. Using off-the-shelf LLMs, HeteroMosaic restructures execution into dependency-preserving micro-batches and jointly optimizes cross-accelerator overlap and per-stage assignment using a heterogeneous roofline model and trace-guided critical-interval co-optimization. The resulting schedule executes across the CPU, iGPU, and NPU with unified memory, shared weights, low-overhead synchronization, and NPU-aware queue management. Ryzen™ AI offers a broad range of CPU, iGPU, and memory configurations, allowing us to study heterogeneous execution across diverse balance points while keeping the LLM model and framework fixed. Across these systems, we show that heterogeneity can be the winning solution when it is jointly scheduled at the graph and operation levels. Our key contributions are as follows.

the stronger accelerator’s behavior or offload too much work to the weaker one, obscuring when heterogeneity is genuinely beneficial. HeteroMosaic addresses this gap by treating edge LLM inference on heterogeneous SoCs as a scheduling problem, not just as an accelerator placement problem. We first develop a heterogeneous roofline model to characterize when heterogeneous execution should help across systems with different relative iGPU and NPU capabilities. Guided by this analysis, HeteroMosaic decomposes execution into dependency-preserving micro-batches that expose additional opportunities to overlap work across accelerators. Exposing this overlap is only the first step: once these new scheduling choices exist, their benefits become interdependent on real systems. On real systems, the best greedy accelerator assignment is not always the best global choice. A mapping that is faster in isolation can still degrade end-to-end performance if, because of accelerator asymmetry, unified-memory contention, DVFS, or NPU runtime behavior, it delays stages that are more critical to the overall execution graph. HeteroMosaic therefore uses trace-guided critical-interval co-optimization to jointly tune the schedule, decide which accelerator executes each stage, and adjust stage completion times around the measured critical path. We realize HeteroMosaic on three AMD Ryzen™ AI SoCs spanning NPU-heavy, balanced, and iGPU-heavy designs. We focus on AMD Ryzen™ AI for two reasons. First, AMD Ryzen™ AI exposes low-level software stacks for both the iGPU and NPU, enabling optimized custom kernels on each accelerator together with lowoverhead interoperability and control across them. Second, AMD

• Problem reformulation. We show that heterogeneous edge LLM inference on unified-memory SoCs is fundamentally a scheduling problem, not just an operation-placement problem. Much of the heterogeneous opportunity is not explicit in the original execution graph and must first be exposed before it can be exploited. • Analytical model. We develop a heterogeneous roofline model that characterizes when heterogeneous execution should help across systems with different relative iGPU and NPU capabilities, and use it as a principled target for the runtime. • Algorithmic. We show that decomposing LLM execution into dependency-preserving micro-batches exposes additional opportunities to overlap work across accelerators that are unavailable in the monolithic schedule. • Scheduling. We show that exploiting this exposed opportunity is inherently non-greedy on real systems, and introduce trace-guided critical-interval co-optimization to jointly tune the schedule, accelerator assignment, and stage timing under practical runtime effects such as DVFS, device asymmetry, and NPU runtime behavior. 2

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Jun et al.

C1 C1

(a) M Split

(b) K Split

Figure 2: Different tensor partitioning schemes in GEMM. Table 1: AMD Ryzen™ AI SoC specifications. All CPUs use Zen 5 cores. The table reports CPU core/thread count, iGPU RDNA generation and compute units (CUs), NPU generation and AIE core count, and peak memory bandwidth under the maximum LPDDR5X configuration. CPU Zen 5

iGPU RDNA / CUs

AMD Ryzen™ 8c/16t AI 7 350

RDNA 3.5, 8 CUs

SoC

NPU Gen.

NPU AIEs

is therefore often constrained by memory bandwidth rather than raw arithmetic throughput. The standard performance metric for this phase is tokens per second (TPS), which measures the sustained generation throughput after prefill.

2.3

Mem. BW

XDNA 2 32

128 GB/s

12c/24t RDNA 3.5, 16 CUs XDNA 2 32

128 GB/s

16c/32t RDNA 3.5, 40 CUs XDNA 2 32

256 GB/s

AMD Ryzen™ AI 9 HX 370 AMD Ryzen™ AI Max+ 395

• Systems and empirical. We realize this design on three AMD Ryzen™ AI SoCs spanning NPU-heavy, balanced, and iGPUheavy designs, and show that heterogeneity-first scheduling outperforms strong single-accelerator and prior heterogeneous baselines while reducing energy and improving tokens per watt by up to 45.3%. Compared with prior SOTA, we achieve an average 1.25× speedup across platforms and models, with a peak of 2.35× and a minimum of 1.07×.

2 Background 2.1 AMD Ryzen™ AI AMD Ryzen™ AI SoCs [2, 8, 9] contain an x86 CPU, an RDNA iGPU, and an XDNA NPU. These components are connected through AMD’s Infinity Fabric, which integrates the compute engines and connects them to a unified memory architecture. The AMD Ryzen™ AI product family spans multiple use cases and system requirements. Table 1 summarizes three current generation AMD Ryzen™ AI configurations, showing differences in CPU resources, iGPU size, NPU generation, and memory bandwidth.

2.2

LLM Micro-Batching

Micro-batching [4, 28, 31] reduces peak prefill memory by splitting a single long prompt into smaller contiguous prompt segments. Here, a micro-batch does not refer to a batch of independent requests, as in conventional training or batched inference. Instead, it refers to a sub-sequence of tokens from the same prompt. Rather than executing one monolithic prefill pass over the full prompt, the runtime processes these prompt segments sequentially. As each micro-batch is processed, the KV cache is incrementally extended, while temporary activations are materialized only for the current segment. Importantly, attention can still attend to preceding tokens through the KV cache, meaning the KV cache is used not only during decode but also during prefill. Micro-batch size determines the size of 𝑄, while 𝐾 and 𝑉 continue to grow through the KV cache. Because LLM models employ causal attention, later micro-batches see a deeper KV history and therefore incur higher attention cost than earlier micro-batches. Causality is what makes this decomposition exact. A later micro-batch may attend to tokens from earlier microbatches through the KV cache, but it does not depend on tokens that have not yet been processed. From a computational perspective, micro-batching does not change the final mathematical result, but it does change the execution schedule. It increases dispatch frequency, reuses weights across multiple smaller launches, and changes how attention cost evolves across micro-batches. Those properties make it useful not only as a memory optimization, but also as a scheduling primitive for heterogeneous execution.

2.4

LLM Inference

Tensor Parallelism

Tensor parallelism distributes a single operation across multiple compute units [77]. In LLM inference, the most common target operations are GEMM operations because they dominate the arithmetic cost of prefill. These dense linear operations are natural split targets because partitioning preserves regular independent subproblems with predictable communication. Figure 2 shows three natural split dimensions. 𝑀- and 𝑁 -splits partition the input / output space and require gathering of results, while a 𝐾-split produces partial sums that must be reduced. In unified-memory SoCs, these communication patterns still matter even though the gather or reduction ultimately resolves through unified memory rather than explicit multi-accelerator collectives.

LLM inference consists of two distinct phases. The first phase, prompt encoding or prefill, processes the entire prompt and produces the first output token. Prefill is dominated by large matrix multiplications and is therefore primarily constrained by dense general matrix multiply (GEMM) throughput. The commonly used metric time to first token (TTFT) captures the latency of processing a prompt of 𝑁 tokens and producing the first generated token. The second phase, token generation or decode, produces one token at a time by attending to both the encoded prompt and previously generated tokens. Decode relies more heavily on general matrixvector multiplication (GEMV) operations and KV-cache traffic, and 3

Jun et al.

2.5

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Community-Driven NPU Programming

3.1.2 Outlier-routed NPU inference. Fast On-device LLM Inference with NPUs [89] leverages heterogeneity by adapting the model execution path to better match the target hardware. During quantization, the original dense matrix computation is split into two paths: a low-precision dense main path and a high-precision shadowoutlier path for activation–weight interactions that cannot be safely represented within the low-precision format. At runtime, the lowprecision main path is mapped to the NPU, while the high-precision outlier path is executed on the CPU. In this sense, the system exploits heterogeneity by reshaping the model so that each backend handles the precision and compute intensity it supports best. This work is among the first to study outlier execution from a hardwaresystems perspective, while later work such as SVDQuant [48] studies the quantization side of outlier handling more directly. Although this work makes NPU execution practical under mobileSoC constraints, it also couples heterogeneity with a changed execution structure. The model is explicitly restructured so that the main low-precision path fits the NPU, while the numerically sensitive outlier path is separated onto the CPU. As a result, the reported benefit reflects both hardware-aware outlier decomposition and heterogeneous CPU–NPU mapping. This differs from HeteroMosaic, which does not introduce a separate numerical path or depend on outlier routing; instead, it keeps the off-the-shelf LLM execution structure intact and improves performance by exposing graph-level overlap, shaping stage latency, and coordinating existing work across accelerators. Consequently, this model-restructuring requirement [89] narrows the generality of the reported heterogeneous-execution result: because the method relies on separating the computation into an NPU-friendly low-precision main path and a CPU-routed shadowoutlier path, its benefit is tied to how cleanly that decomposition matches the target model and backend capabilities. If the computation does not separate in this way, or if the outlier path becomes too expensive, it is unclear whether the same heterogeneous strategy would remain effective. Moreover, as with activation sparsity, shadow-outlier selection can depend strongly on calibration representativeness: if the calibration data does not capture the activation outliers that appear during deployment, either accuracy can degrade or more work may be routed through the expensive shadow path.

IRON [39] is an open-source framework for programming AMD’s XDNA NPUs, combining fine-grained control over compute kernels with higher-level abstractions for data movement and synchronization. Its open-source nature has enabled the broader community to build optimized custom kernels on top of this stack. In particular, we leverage community-developed AWQ [49] W4A16 kernels [34] as a key component in enabling heterogeneous execution across the NPU and iGPU.

Related Work

Prior work on edge LLM inference on unified-memory systems has shown evidence of the potential benefits of heterogeneous SoC execution, but often conflates heterogeneity with other sources of improvement. Existing systems combine heterogeneous execution with model restructuring, quantization changes, activation sparsity, outlier routing, or accelerator capability imbalance, where practical compute throughput is heavily skewed toward one accelerator. Moreover, the closest GPU–NPU edge LLM studies are largely evaluated on Qualcomm Gen3-class mobile SoCs [70], where HeteroInfer reports a roughly 1:10 practical GPU-to-NPU throughput ratio [21]. As a result, it is difficult to determine whether the reported gains come from heterogeneity itself or from coupled factors such as reduced work, changed numerical formats, platform-specific quantization paths, or accelerator imbalance. Thus, while prior work establishes that SoC heterogeneity can be useful, it leaves open the key question: can heterogeneity itself, when optimized from first principles, become the winning strategy for off-the-shelf LLM architectures and standard quantization formats?

3.1

Sparse and Model-Restructured Inference.

3.1.1 Activation-sparse and storage-aware inference. PowerInfer2 [90] improves edge LLM inference by combining heterogeneous execution with activation sparsity and storage-aware execution. Rather than executing each linear layer as a dense operation, it decomposes computation into fine-grained neuron clusters, maps dense activation clusters to the NPU, and handles sparse clusters on the CPU. It further coordinates compute with flash I/O through segmented neuron caching and pipelining, allowing models that exceed DRAM capacity to execute more efficiently. The key distinction is that PowerInfer-2 changes the amount and structure of computation exposed to the hardware. Activation sparsity allows inactive or low-importance neuron clusters to be skipped or delayed, so part of the speedup may come from reducing the effective number of dense operations rather than from better scheduling the same dense workload across accelerators. This does not diminish the value of PowerInfer-2, but it makes the source of improvement different from HeteroMosaic. HeteroMosaic keeps the off-the-shelf LLM graph and standard weight-only quantization path intact, then improves performance by changing when and where existing work executes. In addition, sparsity policies are only as reliable as the calibration or profiling inputs used to construct them. If those inputs do not match deployment prompts, the active-neuron distribution can shift, causing the sparse execution plan to lose either performance predictability or accuracy.

3.2

NPU Optimizations, Runtime Optimizations, and Tensor Parallelism

3.2.1 NPU-oriented optimizations. ScalingNPU [35], although not a heterogeneous inference framework, makes evident the potential of mobile NPUs for efficient LLM execution when the runtime and kernels are co-designed around the hardware. Its key contribution is NPU-centric execution: it uses test-time scaling to expose additional decoding parallelism, then realizes that parallelism through hardware-tailored quantization layouts, fused NPU kernels, and carefully optimized runtime scheduling. In this sense, ScalingNPU shows that NPUs can serve as effective LLM execution engines, but only when the workload structure, data layout, and runtime are designed around the hardware rather than treated as a generic accelerator backend. 4

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

ScalingNPU thus demonstrates that practical NPU performance depends heavily on low-level runtime and kernel co-design. Specifically, it shows that Qualcomm Gen3 NPUs cannot efficiently consume the fine-grained weight-only quantization layouts commonly used in LLM deployment, such as AWQ-style W4A16 group quantization. From an accuracy standpoint, ScalingNPU further shows that QNN [71] per-channel W4A16 severely degrades Llama3.21B-Instruct accuracy compared with AutoAWQ per-group W4A16, especially on reasoning benchmarks. To address this mismatch, ScalingNPU introduces an NPU-tailored tile quantization layout, fused NPU kernels, and LUT-based implementations that reduce runtime dequantization overhead. However, optimizing specifically for the NPU can also reduce compatibility with GPU execution paths, since an NPU-tailored weight layout may not be directly usable by GPU kernels. HeteroMosaic targets this broader systems problem by building compatible iGPU and NPU execution kernels, then applying graph-level and operation-level heterogeneity. This is possible because the NPU and GPU execution paths use the same underlying LLM model and quantization scheme, enabling heterogeneous scheduling without requiring separate model representations for each accelerator.

Jun et al.

HeteroMosaic differs by treating heterogeneity as an end-to-end scheduling problem: it first exposes graph-level overlap through causal micro-batching, then uses heterogeneous partitioning to shape critical and non-critical stage latencies under measured system effects. 3.2.3 Portable GPU-oriented LLM runtimes. llama.cpp [27] is one of the most practical and portable single-accelerator frameworks for edge LLM inference, primarily targeting CPU or GPU execution, with more recent support for select NPU backends. Built around the ggml/GGUF [29] execution stack, llama.cpp emphasizes portability across a wide range of backend hardware rather than finegrained heterogeneous orchestration across accelerators. Its backend ecosystem spans Apple Silicon through Metal [17], NVIDIA GPUs through CUDA [53], AMD GPUs through HIP/ROCm [5], Intel and NVIDIA GPUs through SYCL [30, 44], generic GPUs through Vulkan and OpenCL [43, 45], CPUs through BLAS/BLIS and ZenDNN [3, 26, 51], and emerging accelerator targets such as OpenVINO, CANN, Hexagon, WebGPU, RPC, and VirtGPU [30, 33, 38, 41, 72, 88]. This broad backend support makes llama.cpp a strong community-driven deployment framework, but its primary abstraction is backend portability rather than heterogeneous graph scheduling. llama.cpp also provides its own GGUF quantization formats to balance model size, bandwidth, and performance across devices, including 4-bit, 5-bit, 6-bit, 8-bit, and 16-bit weight formats [32]. It also supports micro-batching to reduce peak memory pressure on lowmemory systems by splitting long prompt processing into smaller execution units [28, 31]. However, in llama.cpp, micro-batching is primarily a memory-management mechanism within a selected backend path, not a mechanism for exposing cross-accelerator overlap. Moreover, because llama.cpp prioritizes portability across diverse backends, its execution path can leave performance on the table on GPUs; for example, backend-agnostic tensor transitions may introduce unnecessary data movement or intermediate casting to preserve a common execution model across hardware targets. HeteroMosaic targets a different point in the design space: it extends micro-batching into a heterogeneous scheduling primitive, builds compatible iGPU and NPU kernels around a shared quantization path, and then combines graph-level overlap with operation-level partitioning to improve end-to-end latency and energy.

3.2.2 Heterogeneous Tensor Parallelism. HeteroInfer [21] studies heterogeneity more directly by mapping LLM inference across the GPU and NPU of a mobile SoC. It uses the NPU as the primary compute engine and the GPU as a secondary accelerator, combining layer-level and tensor-level partitioning with profiling-guided scheduling and lightweight synchronization. Its focus is therefore closer to heterogeneous accelerator mapping than sparsity- or outlier-based systems. At the same time, the evaluation leaves the source of the reported gains difficult to isolate. The evaluated platform is highly NPUskewed: HeteroInfer reports roughly a 1:10 practical GPU-to-NPU throughput ratio on Snapdragon 8 Gen 3 [70]. In other words, the NPU is approximately an order of magnitude faster than the GPU for the evaluated kernels. In this regime, assigning more work to the NPU should improve performance even without demonstrating complementary GPU–NPU execution. Thus, without an NPU-only baseline for the same model and software stack, it is difficult to determine whether the reported gains come from heterogeneity itself, from improved tensor placement, or simply from assigning more work to the stronger accelerator. This ambiguity is reinforced by Qualcomm AI Hub results, which report an optimized Llamav2-7B-Chat deployment for Snapdragon 8 Gen 3 using W4A16 weights with W8A16 in some layers [69] that meets or exceeds the heterogeneous performance numbers reported by HeteroInfer. There are also quantization and backend-equivalence issues that complicate attribution. Related work on Qualcomm Gen3 NPUs, such as ScalingNPU [35], notes that the available QNN [71] quantization path differs from blockwise LLM quantization schemes and can affect prediction metrics such as perplexity. Since HeteroInfer uses different backend stacks for different devices, QNN for the NPU and OpenCL for the GPU, the evaluation may conflate heterogeneous scheduling with backend-specific quantization and kernel behavior. These factors do not diminish HeteroInfer’s contribution as an important GPU–NPU study, but they limit how directly its results establish heterogeneity itself as the source of the gain.

Heterogeneous Roofline Model

We develop an operation-level heterogeneous roofline model for our target AMD Ryzen™ AI platforms, illustrated in Figure 3. The figure first contrasts two bounds: the red curve denotes the conventional single-accelerator roofline, while the green curve denotes the best-case operation-level heterogeneous roofline when multiple accelerators are active. The red curve is limited by the compute throughput and memory bandwidth of a single accelerator. The green curve raises the peak operation throughput as heterogeneous execution combines compute resources across accelerators. At the same time, both curves remain constrained by the unified off-chip memory system and by system effects such as DVFS. Figure 3 should therefore be interpreted as an operation-level view of attainable performance, not as a roofline for the full endto-end LLM workload. The blue point denotes an operation that is 5

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

directly suitable for heterogeneous execution, since, relative to the single-accelerator roofline, it remains sufficiently compute-limited that heterogeneous execution can raise its attainable performance toward the heterogeneous bound. The red point denotes an operation that is a poor heterogeneous candidate because of its proximity to the bandwidth-bound regime and therefore cannot benefit much from additional heterogeneous compute capability. The roofline model therefore serves two purposes. First, it quantifies the best-case upper bound of heterogeneous opportunity across different relative accelerator capabilities within a SoC. Second, it reveals why a placement-only view is insufficient. Some operations are good heterogeneous targets in the original graph, while others appear unsuitable under local operation placement alone. However, these seemingly unsuitable opportunities can become useful when scheduling moves bandwidth-bound work off the critical path or exposes new overlap that was not visible in the original monolithic graph. This distinction explains why an operation-level roofline is necessary but not sufficient for end-to-end LLM inference. Although it can identify which individual operations are promising heterogeneous candidates, it cannot determine whether accelerating a given operation shortens the global application-level schedule, nor can it expose overlap that is absent from the original execution graph. We therefore treat the roofline as an analytical target rather than an end-to-end predictor. The scheduling framework in the following section restructures execution through causal micro-batching and uses trace-guided critical-interval optimization to determine how much of that target can be realized in practice. Having established the operation-level intuition behind Figure 3, we now formalize the heterogeneous roofline. We start from the classical roofline model and extend it to AMD Ryzen™ AI platforms that combine CPU, iGPU, and NPU resources under a shared memory system. The classical roofline model [87] defines single-accelerator attainable performance as

induced by the heterogeneous tensor split, such as the 𝑀-, 𝐾-, and 𝑁 -split GEMM decompositions shown in Figure 2 and described in Section 2.4. For this fixed accelerator-side split, the average Ísystem-wide 𝑁 −1 𝑓𝑖 operational intensity, denoted as 𝐼 avg = 1/ 𝑖=0 𝐼𝑖 , is computed using the weighted harmonic mean of the individual intensities. For a fixed work split, the attainable heterogeneous performance is limited by the slowest assigned accelerator partition and by the shared-memory bandwidth cap: CPU iGPU NPU 𝑃 attainable = min 𝑃attainable , 𝑃 attainable , 𝑃 attainable , 𝐵 peak · 𝐼 avg . This minimum captures the bottleneck for one heterogeneous split. The full operation cannot complete faster than the slowest participating accelerator or the shared-memory system that feeds them. The heterogeneous roofline shown as the green curve in Figure 3 is then the best attainable envelope over feasible work splits: ∗ 𝑃 hetero =

Thus, the inner minimization gives the attainable performance for one split, while the outer maximization selects the split that best balances the CPU, iGPU, and NPU and produces the best-case heterogeneous bound. To account for practical inefficiencies, we introduce a performance efficiency parameter 𝜂𝑖 for accelerator 𝑖, where 0 < 𝜂𝑖 ≤ 1, and a bandwidth efficiency factor 𝛼𝑖 . Here, 𝜂𝑖 captures effective compute efficiency, measured as achieved TOP/s divided by published peak TOP/s, while 𝛼𝑖 captures effective memory-bandwidth efficiency after cache behavior, access-pattern effects, unified-memory contention, and runtime overheads. These values are not fixed constants during execution: on real SoCs, they vary with DVFS, thermal state, kernel shape sensitivity, NPU runtime behavior, and memory contention. The effective compute and bandwidth for each accelerator are

𝑃 = min(𝑃 peak, 𝐵 · 𝐼 ),

𝑃𝑖eff = 𝜂𝑖 · 𝑃 peak,𝑖 ,

where 𝑃peak is peak compute throughput, 𝐵 is peak off-chip memory bandwidth, and 𝐼 is operational intensity, defined as arithmetic operations per byte accessed. For floating-point kernels, this corresponds to FLOPs/byte; for quantized kernels, we report ops/byte to match the TOP/s-based throughput model. This model captures the trade-off between compute-bound and memory-bound regimes, which we adapt to reason about heterogeneous execution across multiple accelerators sharing a common memory system. We adopt Gables [37] to model 𝑁 compute units operating concurrently on a unified memory system. Each accelerator 𝑖 has peak compute throughput 𝑃 peak,𝑖 , memory bandwidth 𝐵𝑖 , and executes a fraction 𝑓𝑖 of the workload with operational intensity 𝐼𝑖 , where Í 𝑖 𝑓𝑖 = 1. We write the attainable whole-operation performance allowed by accelerator 𝑖 as 𝑖 𝑃attainable =

max 𝑃 attainable . Í { 𝑓𝑖 }: 𝑖 𝑓𝑖 =1

𝐵𝑖eff = 𝛼𝑖 · 𝐵𝑖 .

Using these effective terms, the attainable whole-operation performance allowed by accelerator 𝑖 becomes 𝑖 𝑃attainable =

min(𝐵𝑖eff · 𝐼𝑖 , 𝑃𝑖eff ) . 𝑓𝑖

This formulation preserves the roofline structure while allowing the model to account for software overheads, thermal throttling, runtime inefficiencies, and memory contention by scaling effective compute and bandwidth. In particular, when the iGPU and NPU contend for unified memory, effective bandwidth degrades, reducing the attainable performance of both devices and shifting the practical heterogeneous bound downward. We then use this model to project the best-case heterogeneous speedup as a function of achieved iGPU and NPU efficiency. Figure 4 plots this projected speedup surface.1 The figure should not be read as assigning a single fixed 𝜂 or 𝛼 to each device. Instead, it sweeps achieved iGPU and NPU efficiency values relative to each platform’s published peak capability [2, 8, 9]. In the idealized projection, runtime overheads and memory-contention losses are

min(𝐵𝑖 · 𝐼𝑖 , 𝑃 peak,𝑖 ) . 𝑓𝑖

The numerator is the local roofline limit for accelerator 𝑖 on its assigned partition, while the division by 𝑓𝑖 converts this local partition throughput into an equivalent whole-operation throughput. The workload fractions 𝑓𝑖 correspond to the accelerator-side partitions

1 The CPU is omitted because of its relatively limited dense-compute capabilities.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

is more sensitive to operation shape, kernel implementation, and runtime state than isolated GEMM benchmarks. Under the idealized assumption that all compute is heterogeneously mappable, and that software overheads, memory contention, and other runtime effects are absent, the model projects approximate upper bounds of 3×, 1.8×, and 1.25× over each iGPU baseline for the AMD Ryzen™ AI 7 350, AMD Ryzen™ AI 9 HX 370, and AMD Ryzen™ AI Max+ 395, respectively. These projected bounds indicate that heterogeneity can be beneficial in principle, but they do not by themselves explain how to realize that opportunity on a real system. In particular, the roofline model does not determine which parts of the graph are immediately profitable to map heterogeneously, which parts become profitable only after schedulable opportunity has been exposed, or how either should be scheduled under practical effects such as unified-memory contention, runtime overheads, and device-specific behavior. The remainder of the paper focuses on exactly this gap: exposing latent heterogeneous opportunity in the execution graph and developing a runtime that can realize as much of the analytical bound as possible in practice.

Figure 3: Illustration of the heterogeneous roofline model. Some operations are strong candidates for heterogeneous execution, while others remain bandwidth-bound when viewed in isolation. 10.50× 7.5 9.00× 6.00 0× ×

0.40 12.0 10.2 8.4 6.6 4.8 3.0 1.2

0×

0.25

Speedup (×)

0.30

3.0

4.5

NPU Efficiency

0.35

0.20 0.15 0.10 0.1

1.50×

0.2

0.3

0.4

0.5

0.6

iGPU Efficiency

0.7

0.8

0.9

5 6.00× 4.204×.80.4×0× 3.60 3.0 × 0×

Speedup (×)

1.80

0×

0.25

2.4

NPU Efficiency

0.30 0.20 0.15

0.2

0.3

0.4

0.5

0.6

iGPU Efficiency

0.7

0.8

0.9

1.0

(b) AMD Ryzen™ AI 9 HX 370

3.2 2 3.00×5× 2.252.50.7×5× × 2.00 1.7 × 5×

0.40

3.75 3.45 3.15 2.85 2.55 2.25 1.95 1.65 1.35 1.05

0.30

Speedup (×)

NPU Efficiency

0.35

0×

1.5

0.25

1.25×

0.20 0.15 0.10 0.1

0.2

0.3

0.4

0.5

0.6

iGPU Efficiency

0.7

0.8

0.9

HeteroMosaic Overview

Figure 1 illustrates the overall design of HeteroMosaic. HeteroMosaic starts from the fixed structure of off-the-shelf decoder-only LLMs and implements model execution as a compiled C++ functioncall path following the transformer architecture. PyTorch C++ [67] is used for tensor and weight management, while the execution graph seen by the scheduler is explicitly exposed through C++ stage boundaries in this custom C++ transformer path. These boundaries form coarse graph nodes, such as 𝐺 1 , Attention, and 𝐺 2 , with edges derived from program order and KV-cache causality. KV-cache updates, RoPE variants, and FlashAttention-style execution are represented as explicit C++/HIP stages, with the FlashAttentionstyle path implemented as a custom HIP kernel. HeteroMosaic then restructures this explicitly defined graph into causally valid microbatches and uses the heterogeneous roofline model, microbenchmarks, and trace-guided optimization to decide how work should be overlapped and assigned across the iGPU and NPU. The resulting schedule is executed through a unified runtime with shared weights, cross-device synchronization, and NPU-aware queue management. The central challenge is therefore not merely to map compatible operations onto multiple devices, but to expose and schedule heterogeneous opportunity within the compiled C++ execution path under real system constraints. Our roofline analysis shows why this distinction matters. If heterogeneous execution is restricted only to operations that are directly compatible across the iGPU and NPU, then the achievable speedup is fundamentally bounded by the fraction of runtime spent in those operations. While this limitation is less restrictive at shorter contexts, it becomes increasingly important for longer prompts. For example, in Llama3-8B [50], non-GEMM operations account for only a small fraction of runtime at short prompts, but at a prompt length of 16,384 they can grow to as much as 35% of total execution time. As a result, heterogeneous GEMM execution alone cannot fully realize the available opportunity, especially across SoCs with widely different iGPU-to-NPU balance points.

7.5 6.6 5.7 4.8 3.9 3.0 2.1 1.2

0.35

0.10 0.1

1.0

(a) AMD Ryzen™ AI 7 350

0.40

Jun et al.

1.0

Figure 4: Projected heterogeneous speedup over the iGPU baseline from the heterogeneous roofline model as a function of iGPU and NPU efficiency. The contours indicate the projected speedup for different iGPU–NPU efficiencies.

excluded; the measured microbenchmarks and traces later calibrate how far real execution falls below this ceiling. To ground the efficiency ranges used in Figure 4, we calibrate the NPU and iGPU terms separately rather than assuming that published peak TOPS are fully attainable. For the NPU, we conservatively use 0.4 as a favorable upper efficiency setting, consistent with prior academic studies of AMD NPUs [81–83]. For the iGPU, prior AMD GPU GEMM benchmarking reports optimized denselinear-algebra efficiency of roughly 0.7 [20]. However, we still leave iGPU efficiency as a broader sweep because iGPU performance 7

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

under unified-memory contention, DVFS, accelerator imbalance, and backend shape sensitivity [21]. This matters because 𝑀-, 𝐾-, and 𝑁 -splits stress the system differently. For a GEMM 𝐶 = 𝐴𝐵, where 𝐴 ∈ R𝑀 ×𝐾 and 𝐵 ∈ R𝐾 ×𝑁 , an 𝑀-split partitions rows of 𝐴 and 𝐶. This avoids cross-device reduction, but each accelerator must still read the same weight matrix 𝐵. When 𝐵 is large and DDR bandwidth is fixed, this duplicated weight traffic can reduce or eliminate the benefit of using both accelerators. An 𝑁 -split partitions columns of 𝐵 and 𝐶. This can avoid duplicate weight reads, but instead duplicates activation reads and can be sensitive to how downstream operators consume the output. If the split introduces extra gathers, transposes, or backend-specific layout conversions, local GEMM speedup may not translate into end-to-end benefits. A 𝐾-split partitions the reduction dimension, so each accelerator computes a partial result that must later be accumulated. This adds partial-𝐶 read/write traffic and a reduction step, but when 𝐾 is large, the extra merge cost can be amortized by the greater amount of parallel work. Since GEMM kernels are often highly sensitive to the reduction dimension and how it maps onto finite compute resources, 𝐾-split can become preferable for large-𝐾 or large-weight GEMMs despite being more computationally and memory intensive. The split choice also introduces accelerator-specific shape sensitivity. A tensor split changes the local GEMM shape seen by each backend, and that shape determines occupancy, tiling efficiency, vector utilization, memory coalescing, and launch overhead. A split that appears balanced by total FLOPs may still leave one backend with a subproblem that is too small, poorly aligned with its tile shape, or unable to saturate its available compute units. Data layout further complicates this trade-off: contiguous DDR access is important for both accelerators, but the iGPU and NPU may prefer different packed layouts, tiling orders, and vectorization patterns.

(a) Illustration of our Abstracted LLM Graph.

(b) Illustration of Serial Micro-Batching.

(d) Example of a Latency-Shaped Micro-Batched Schedule.

Figure 5: Illustration of Heterogeneous Micro-Batching. HeteroMosaic addresses this in two steps. It first restructures execution to expose additional graph-level heterogeneous opportunity, then applies operation-level partitioning and trace-guided co-optimization to realize that opportunity on the true critical path. Supporting this in practice requires a unified runtime with crossdevice and cross-stream synchronization, NPU schedule management, and dynamic iGPU/NPU execution optimizations. HeteroMosaic therefore combines PyTorch C++ tensor and weight management with open-source IRON-based fused NPU kernels and compatible iGPU kernels tailored to the shared heterogeneous execution path.

5.1

Heterogeneous Tensor Parallelism

5.1.2 Why tensor parallelism alone is insufficient. These split-level effects necessitate an empirical characterization of single-operation heterogeneity. A scheduler cannot make accurate global decisions if it assumes that all split-friendly GEMMs behave similarly. It must know which split dimensions are profitable for a given shape, how stable those gains are under DVFS and memory contention, and how much overhead is introduced by synchronization and gather operations. Our microbenchmarks in Section 6.1 provide this calibration by measuring 𝑀-, 𝐾-, and 𝑁 -split behavior across the three AMD Ryzen™ AI balance points. At the same time, tensor parallelism alone cannot fully realize the heterogeneous opportunity. Operation-level partitioning can improve dense GEMM- and GEMV-like operations either by sharing compute across accelerators or by matching a shape to the device that executes it best. However, this opportunity is not uniformly distributed across the LLM graph. Reduction-heavy, normalizationheavy, and fusion-sensitive operations are weaker targets for operation only sharing; for example, softmax introduces row-wise reductions and normalization, while flash attention [23] relies on fused tiling and on-chip reuse whose benefits can be diluted by partitioning or offload. As a result, HeteroMosaic treats tensor parallelism as a scheduling primitive rather than a standalone placement rule. The microbenchmarks provide candidate split dimensions and split ratios,

Tensor parallelism is the natural first step for heterogeneous execution because it distributes a single operation across multiple compute engines. As introduced in Section 2.4, GEMM has three natural split dimensions: 𝑀-, 𝐾-, and 𝑁 -splits. On heterogeneous SoCs, these splits can be extended across accelerators when an operation exposes sufficient data parallelism and both backends can execute the same computation. In such cases, a single GEMM can be partitioned across the iGPU and NPU rather than assigned entirely to one device. Although conceptually simple, heterogeneous splitting remains nontrivial on a unified-memory SoC because unified memory only removes data-copy overhead, not coordination overhead. The accelerators can access the same model buffers without full weight duplication, but the profitability of a split still depends on shared-memory contention, synchronization cost, accelerator balance, backend shape sensitivity, and runtime state. Thus, the best split cannot be inferred from peak TOPS or total arithmetic work alone; it must be characterized empirically and selected in the context of the global schedule. 5.1.1 Split-dimension trade-offs under unified memory. Prior heterogeneous LLM frameworks for edge SoCs use tensor-level partitioning, but do not fully characterize how split dimensions behave 8

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

but the final choice is made in the context of the global schedule. A split that is fastest for one GEMM in isolation may still hurt endto-end latency if it increases memory contention, delays a critical stage, or causes downstream layout overhead. HeteroMosaic therefore combines operation-level tensor partitioning with graph-level overlap and critical-path-aware scheduling, rather than greedily optimizing each split-friendly dense layer in isolation. This limitation motivates HeteroMosaic’s graph-level component: rather than treating heterogeneity exclusively as operationlevel mapping, we restructure execution to expose new overlap opportunities across the execution graph.

5.2

Jun et al.

𝜇𝐵 2 -Attention, creating heterogeneous overlap that is absent in the original monolithic schedule. HeteroMosaic realizes this overlap using a small pool of CPU work-dispatch threads that arbitrate dependency-ready micro-batch stages. Each dispatch thread selects ready work from the global schedule and forwards it to the appropriate backend. iGPU work is issued asynchronously through independent HIP streams, with hipEventRecord [11] used to enforce cross-stream dependencies at KV-cache update points. NPU work is not dispatched directly through additional general-purpose work threads; instead, it is enqueued to a dedicated NPU management thread, which serializes and coalesces NPU requests before issuing them to the NPU runtime, as described in Section 5.3.2. The number of work-dispatch threads is therefore a tunable runtime parameter rather than a fixed design choice. It depends on the available CPU cores, iGPU compute units, and platform power budget. In our evaluated systems, the AMD Ryzen™ AI 7 350 and AMD Ryzen™ AI 9 HX 370 use two work-dispatch threads, while the larger AMD Ryzen™ AI Max+ 395 uses three. The additional CPU resources, larger iGPU, and higher power budget on the AMD Ryzen™ AI Max+ 395 allow more independently dispatched GPU work to improve iGPU occupancy without overwhelming hostdevice coordination, while the NPU remains managed through its single dedicated queueing path. Importantly, parallel micro-batching alone does not necessarily reduce end-to-end LLM latency on a single-accelerator system, since all work still contends for the same execution resources. In our experiments, asynchronous parallel dispatch of micro-batches could even hurt performance due to cache-level interference; for example, 𝜇𝐵 1 -𝐺1 and 𝜇𝐵 2 -𝐺1 may simultaneously read duplicate QKV weights into the iGPU LLC, leading to cache thrashing. In a heterogeneous system, however, these same causally valid micro-batches can be separated across devices and converted into useful overlap. This connects directly back to the roofline model: the roofline identifies where heterogeneous opportunity exists in principle, while micro-batching reshapes the execution graph to expose schedulable opportunity that is hidden in the original monolithic schedule. In this sense, micro-batching does not change the analytical bound itself; rather, it makes more of the bound reachable in practice by uncovering previously hidden heterogeneous opportunity.

Heterogeneity-Aware Scheduling

Figure 5a shows the coarse graph abstraction used by the scheduler. In our implementation, this graph is derived from the compiled PyTorch C++ function-call structure of the model, where nested layer and operator calls are traced into coarse stage nodes such as 𝐺 1 , Attention, and 𝐺 2 . In the original monolithic prefill graph, each layer exposes a mostly serial stage sequence, 𝐺 1 → Attention → 𝐺 2 , over the full prompt. However, this initial graph contains little dependency-valid overlap because there are no independent nodes to schedule. Through micro-batching, we restructure the graph by replicating these nodes per 𝜇B and adding only the KVcache dependency edges required for causal correctness. As a result, nodes such as 𝜇𝐵 1 -𝐺 2 and 𝜇𝐵 2 -Attention can become independent once the required KV-cache update is complete, exposing overlap that is absent from the original monolithic execution graph. 5.2.1 Causal Parallel Micro-Batching. HeteroMosaic uses microbatching not merely as a memory optimization, but as a mechanism for exposing concurrent heterogeneous opportunity. As illustrated in Figure 5a, we view each LLM transformer block as three coarse stages: operations before attention that generate and write entries into the KV cache (𝐺 1 ), the attention computation, and operations that follow attention (𝐺 2 ). This abstraction lets us describe prefill not as one monolithic pass over the full prompt, but as a sequence of smaller causally ordered chunks, or micro-batches (𝜇Bs). Each 𝜇B processes only a subset of prompt tokens, writes its new keys and values into the KV cache, and allows later 𝜇Bs to attend to the tokens that have already been processed. This preserves the same causal semantics as the original prefill, because a later 𝜇B can read earlier KV-cache entries but never depends on future tokens. Figure 5b shows the conventional serial form used by frameworks such as llama.cpp: 𝜇𝐵 1 executes first and extends the KV cache, then 𝜇𝐵 2 executes using the deeper causal history created by 𝜇𝐵 1 , and the final output is produced after the last micro-batch. Later 𝜇Bs therefore attend over a longer KV history and incur longer attention time. Our first scheduling insight is that these micro-batches can themselves execute in parallel. However, to preserve attention causality, a subsequent 𝜇𝐵 may only read from the KV cache after the preceding 𝜇𝐵 has written its entries. Figure 5c illustrates the resulting schedule. Compared with the serial schedule in Figure 5b, 𝜇𝐵 2 does not need to wait for all of 𝜇𝐵 1 to finish; once 𝜇𝐵 1 has produced the required KV-cache entries, 𝜇𝐵 2 can begin its attention stage while 𝜇𝐵 1 continues into 𝐺 2 . Thus, work such as 𝜇𝐵 1 -𝐺 2 can overlap with

5.2.2 Heterogeneous Critical-Interval Latency Shaping. Once causal parallel micro-batching has exposed additional graph-level overlap, HeteroMosaic uses heterogeneous allocation as a latency-shaping mechanism that controls when a node in the task graph completes relative to the global schedule. The key control is the allocation choice for that node. A tensor-parallel split close to the deviceoptimal balance point can decrease the node’s execution time by using both accelerators effectively. For non-critical nodes, however, HeteroMosaic can choose slower but less interfering allocations: single-accelerator execution on the fastest accelerator, singleaccelerator execution on the second-fastest accelerator, or a deliberately less aggressive split. Thus, the allocation space spans fast tensor-parallel configurations for critical nodes and slower single-device or weakly split configurations for non-critical nodes. Algorithm 1 formalizes this policy for a single node within a critical interval, a window in time in which dependency-ready stages 9

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Algorithm 1 Runtime-Aware Latency Shaping 1: Input: node 𝑛, competing stages 𝐶 𝑤 2: Output: edit 𝑞𝑛 3: 𝑆 ← GetSystemInfo ⊲ DVFS, device balance, NPU runtime 4: 𝑞𝑛 ← NoEdit ⊲ default edit 5: 𝑏𝑛 ← 0 ⊲ no bubble by default 6: if not IsHeteroCompatible(𝑛) then 7: if not IsCritical(𝑛) then 8: 𝑏𝑛 ← ChooseBubble(𝑛, 𝐶 𝑤 , 𝑆) 9: 𝑞𝑛 ← InsertBubble(𝑛, 𝑏𝑛 ) 10: return 𝑞𝑛 11: 𝛽𝑛 ← TargetBalancePoint(𝑛, 𝑆) ⊲ e.g., turbo → 0.5, sustained → 0.6 12: 𝑠𝑛 ← QuantizeSplit(𝑛, 𝛽𝑛 ) 13: if IsCritical(𝑛) then 14: 𝑠𝑛 ← BiasTowardMaxResources(𝑛, 𝑠𝑛 , 𝑆) 15: else 16: 𝑠𝑛 , 𝑏𝑛 ← BiasBubbleAwayFromCriticalNodes(𝑛, 𝑠𝑛 , 𝑆) 17: 𝑞𝑛 ← SetSplitAndBubble(𝑛, 𝑠𝑛 , 𝑏𝑛 ) 18: return 𝑞𝑛

(a) Limits of Micro-Batching and Latency Shaping.

(b) Example of an Asymmetric Schedule.

Figure 6: Example of Asymmetric Micro-Batching Schedules.

The concrete latency-shaping decision can therefore change across SoC variations and runtime regimes. In particular, the balance point can shift between a short-lived turbo regime, in which the SoC operates at temporarily elevated power and frequency, and a sustained regime, in which the chip settles to a lower long-term operating point under thermal and power constraints. A split that is favorable while the SoC remains in turbo may no longer be optimal once execution enters the sustained-power regime, and NPU-side effects such as reconfiguration pressure can further bias the preferred allocation. HeteroMosaic therefore does not use a fixed split rule; instead, as shown in Algorithm 1, it selects a runtime-aware target balance point and then biases the final split differently depending on whether the node is critical or non-critical. This distinction matters because graph-level overlap becomes substantially more valuable when combined with operation-level latency shaping. By adjusting the allocation of work across the iGPU and NPU, HeteroMosaic can control the execution time of stages such as 𝐺 1 and 𝐺 2 so that they better align with the global graph schedule. Critical stages are biased toward allocations that minimize completion time under the current system state, while non-critical stages may use less aggressive splits, single-accelerator execution, or bubbles so that they do not interfere with more important work. In this way, heterogeneous allocation is used not merely to distribute computation, but to shape node latency around the critical path under practical runtime effects.

from overlapping micro-batches execute concurrently and contend for the same SoC resources. Within such an interval, HeteroMosaic contracts nodes on the critical path while relaxing non-critical nodes through weaker allocations or bubbles, freeing resources for the stages that determine interval completion. Figure 5c shows the overlapping micro-batch structure that creates these intervals, and Figure 5d illustrates how latency shaping changes stage completion times within them. The key idea in Algorithm 1 is that heterogeneous allocation is both runtime-aware and dependency-preserving. Candidate nodes are considered in topological order over the traced micro-batched DAG, while the competing set 𝐶 𝑤 captures the stages that overlap with the current node inside the critical interval. This topological order preserves producer–consumer constraints, such as KV-cache writes before later attention reads; it does not serialize execution, since dependency-ready nodes from different micro-batches may still execute concurrently. Figure 5d illustrates how the policy uses this dependency-valid overlap. In the constrained region, critical stages such as 𝜇𝐵 1 -𝐺 1 are biased toward faster allocations so that they complete earlier and unblock dependent work. In the relaxedlatency region, non-critical stages such as 𝜇𝐵 2 -𝐺 2 may use a less aggressive allocation or an inserted bubble so that they finish just in time without extending the global critical path. Given a node and its competing stages, the algorithm first checks whether the node is heterogeneity-compatible. If not, the only remaining control is whether to insert a bubble for a non-critical node so that it interferes less with the critical path, as shown by the delayed 𝜇𝐵 2 -𝐺 1 stage in Figure 5d. If the node is compatible, the runtime queries the current system state and selects a target balance point before quantizing that decision into a concrete split. This system state includes DVFS behavior, effective iGPU–NPU asymmetry, and NPU runtime effects such as queueing and configuration pressure.

5.2.3 Asymmetric Micro-Batching for Long Contexts. The fixed micro-batch schedules discussed so far implicitly assume that heterogeneity provides sufficient dynamic range to align node completion times with the global schedule. This assumption becomes increasingly fragile at long context lengths and SoCs with accelerator asymmetry. In Transformer-based graphs, the root cause is the different scaling behavior of linear projections and attention. For a fixed model width, the computational complexity of linear projections scales linearly with prompt length, while the computational complexity of attention scales quadratically. More concretely, projection layers scale as 𝑂 (𝑁𝑑 2 ), whereas attention scales as 𝑂 (𝑁 2𝑑), where 𝑁 is the prompt length and 𝑑 is the embedding dimension. As a result, when 𝑁 is small, non-attention computation occupies a larger share of the schedule, but as 𝑁 grows, attention increasingly 10

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Jun et al.

dominates. As illustrated in Figure 6a, exaggerated here for clarity, naive micro-batching of long prompts can produce schedules in which the growing attention time, driven both by the intrinsic complexity of attention and by the deeper KV history seen by later micro-batches, overwhelms the latency-shaping range available through heterogeneous tensor parallelism. This challenge becomes even more severe on unbalanced systems, where the achievable heterogeneous dynamic range may be insufficient to satisfy the schedule induced by the graph. The limitation is that, for long prompts, a fixed micro-batch schedule can create stage imbalance that heterogeneous latency shaping cannot fully absorb. As later micro-batches see a deeper KV history, their attention stages grow and begin to dominate the microbatched schedule. The maximum acceleration each micro-batch can achieve is bounded by the dynamic range of the latency-shaping mechanism. Here, dynamic range refers to how much the runtime can change a node’s execution time through tensor-parallel splits, single-accelerator assignment, or bubbles. This range is larger on balanced systems: if the iGPU and NPU have comparable capability, using both accelerators can ideally provide close to a 2× latency reduction over either single-accelerator baseline. On a skewed system, however, the range relative to the stronger accelerator is much smaller. For example, with a 1:4 NPU-to-iGPU compute ratio, the ideal heterogeneous speedup over the iGPU is only (1+4)/4 = 1.25×. Thus, when attention growth exceeds this limited shaping range, the runtime cannot contract the long-attention micro-batches enough to achieve the same degree of speedup observed at shorter prompt lengths. HeteroMosaic addresses this limitation by expanding the search space to include not only heterogeneous allocation, but also microbatch size selection. Rather than assuming a fixed, equal-size microbatch schedule, it identifies a micro-batch strategy that better matches the available latency-shaping range. The intuition is illustrated in Figure 6b, which replaces the fixed schedule in Figure 6a with an example gradually decreasing micro-batch size. In the fixed schedule, later micro-batches spend disproportionate time in attention because attention cost grows with the deeper KV history. The decreasing schedule counteracts this by making later micro-batches smaller, reducing the attention time that would otherwise dominate those stages. Larger earlier micro-batches preserve useful projection work when the attention history is still shallow. This better balances attention time across micro-batches and reduces the amount of linear projection work that remains on the critical path for deeper micro-batches. Although this strategy is not always the winning solution, we find it particularly useful on systems with limited heterogeneous range and for long prompt lengths, where fixed schedules can otherwise become too rigid.

into global gains. HeteroMosaic therefore introduces a trace-guided critical-interval co-optimization algorithm that leverages its explicit C++ runtime to emit fine-grained execution traces. Each trace record contains the stage identifier, micro-batch identifier, backend assignment, tensor split ratio, start timestamp, end timestamp, and measured latency. These records are then used to evaluate graph-level overlap and operation-level resource allocation against measured end-to-end latency. Algorithm 2 summarizes this procedure. We adopt a trace-driven approach because graph nodes in real systems can skew in time due to scheduler race conditions, DVFS, and runtime-specific behaviors. Since the algorithm depends on accurately identifying the true critical interval, relying on a static schedule or purely analytical timing model proved insufficient. As shown in lines 3–8 of Algorithm 2, the method first evaluates candidate micro-batch schedules, executes each one, and constructs an internal DAG from the resulting runtime trace. It then selects the schedule that minimizes measured critical-path pressure. After this initialization step, the algorithm repeatedly builds critical intervals, i.e., time windows in which the same set of stage instances are simultaneously active and therefore contend for the same GPU/NPU environment. Lines 11–17 of Algorithm 2 then iterate over nodes in the current traced DAG and invoke Algorithm 1 to propose a node-level edit. Each edit applies the latency-shaping mechanism described in Section 5.2.2 in the context of the measured global schedule, adjusting a node’s split, accelerator assignment, or bubble so that the node completes earlier or later as needed to reduce critical-path pressure rather than optimizing the node in isolation. In this way, Algorithm 1 provides the local runtime-aware policy, while Algorithm 2 decides whether that local edit is globally beneficial. Within each interval, HeteroMosaic jointly optimizes two decisions: the heterogeneous allocation, and optional bubbles for noncritical nodes, which intentionally delay execution when doing so protects the global critical path. Every proposed edit is re-executed and accepted only if it improves measured end-to-end latency, as shown in lines 13–16 of Algorithm 2. This structure is important because a node-level improvement from Algorithm 1 does not necessarily reduce overall latency once the full schedule is re-executed. This tuning is performed offline once per model/device configuration. In our current implementation, the full trace-guided search takes on the order of eight hours and produces reusable configurations for representative prompt-length ranges. At deployment time, HeteroMosaic selects the corresponding configuration and does not repeat the full search. Thus, the search cost is paid during offline calibration rather than on the latency-critical inference path.

5.2.4 Critical-Interval Co-Optimization. The exposed overlap creates a non-greedy schedule optimization problem: locally faster mappings can worsen end-to-end latency by perturbing more critical stages. In a heterogeneous micro-batched schedule, graph-level overlap and operation-level allocation are inherently coupled. For example, in Figure 5c, greedily optimizing 𝜇𝐵 2 -𝐺 2 can negatively impact 𝜇𝐵 1 -𝐺 1 , and vice versa, because both execute within a shared runtime environment and contend for the same heterogeneous resources. As a result, local improvements do not necessarily translate

5.3

AMD Ryzen™ AI Runtime Realization

The previous subsection described the scheduling concepts underlying HeteroMosaic. We now describe the AMD Ryzen™ AI-specific runtime mechanisms required to realize them efficiently in practice. These mechanisms are not the primary algorithmic contribution of the paper; rather, they are the platform-specific enablers that allow the scheduling framework to execute effectively on unified-memory AMD Ryzen™ AI SoCs. 11

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Algorithm 2 Trace-Guided Critical-Interval Co-Optimization 1: Input: initial configuration 𝐶 0 , micro-batch schedule 𝑆 0 2: Output: tuned configuration (𝐶 ★, 𝑆 ★ ) 3: for all 𝑠 ∈ S0 do 4: (𝑇𝑠 , 𝐿𝑠 ) ← RunModelAndCollectTrace(𝐶 0, 𝑠) ⊲ Execute schedule 𝑠 and record trace 𝑇𝑠 and latency 𝐿𝑠 5: 𝐺𝑠 ← BuildInternalDAG(𝑇𝑠 ) ⊲ Build a stage DAG from traced dependencies and timing 6: 𝑃𝑠 ← MeasureCriticalPathPressure(𝐺𝑠 ) ⊲ Critical-path pressure metric 7: if 𝑃𝑠 < 𝑃 ★ then ⊲ 𝑃 ★ initialized to ∞ 8: 𝑆 ★, 𝐶 ★,𝑇 ★, 𝐺 ★, 𝑃 ★, 𝐿★ ← 𝑠, 𝐶 0,𝑇𝑠 , 𝐺𝑠 , 𝑃𝑠 , 𝐿𝑠 9: for 𝑖 = 1 to 𝐵 do ⊲ Search budget 𝐵 10: 𝑊 ← BuildCriticalIntervals(𝐺 ★) 11: for all 𝑛 ∈ 𝐺 ★ do 12: 𝑞𝑛 ← LatencyShaping(𝑛,𝑊 ) 13: 𝐶𝑞 ← ApplyEdit(𝐶 ★, 𝑞𝑛 ) 14: (𝑇𝑞 , 𝐿𝑞 ) ← RunModelAndCollectTrace(𝐶𝑞 , 𝑆 ★) 15: 𝐺𝑞 ← BuildInternalDAG(𝑇𝑞 ) 16: if 𝐿𝑞 < 𝐿★ then 17: 𝐶 ★,𝑇 ★, 𝐺 ★, 𝐿★ ← 𝐶𝑞 ,𝑇𝑞 , 𝐺𝑞 , 𝐿𝑞

where, without careful alignment, the NPU may thrash between configurations across adjacent micro-batches. To address this, we implement a dedicated NPU management thread that services requests generated by the main micro-batch schedule shown in Figure 1. Independent micro-batches enqueue NPU operations into this thread, which then coalesces requests with compatible datapath configurations. In addition, the same operation often appears across multiple micro-batches and differs only in its activation inputs. Without optimization, operations such as 𝜇𝐵 1 -𝐺1 and 𝜇𝐵 2 -𝐺1 would be dispatched separately, causing the NPU to reread the same 𝑄 weights multiple times. When such operations recur within a suitable interval, the NPU management thread can coalesce these otherwise independent but structurally identical operations, thereby reducing both NPU dispatch overhead and redundant weight reads. 5.3.3 Custom Heterogeneous GPU Kernels. The IRON-based NPU kernels available through the open-source fork in [34] rely on a custom weight layout. As a result, to share weights effectively between the iGPU and NPU, HeteroMosaic cannot rely on existing GPU kernels. We therefore develop optimized iGPU kernels whose weight layout is compatible with the NPU layout, eliminating weight duplication across accelerators. Beyond GEMM, we also implement framework-tailored iGPU kernels for non-GEMM stages such as normalization, KV-cache handling, and FlashAttention-style attention [23]. In our experiments, these optimized iGPU kernels outperform available W4A16 implementations such as llama.cpp [27], largely because they are specialized for RDNA-based iGPUs [6] and HeteroMosaic’s execution path.

18: return (𝐶 ★, 𝑆 ★ )

5.3.1 Unified Memory and Synchronization. A primary challenge in mapping compute across the iGPU and NPU on AMD Ryzen™ AI is reconciling separate memory and runtime models. On AMD Ryzen™ AI running Linux, we implement unified memory across different runtimes through the Direct Rendering Manager (DRM) [42] stack, which allows us to create shareable buffers across devices. A second challenge is synchronization across heterogeneous runtimes. On the iGPU, execution follows an asynchronous streambased model through ROCm [5], where the host dispatches work non-blockingly and dependencies are preserved within each stream. The NPU, by contrast, follows a host-dispatch-and-return model, where the host submits work and blocks until completion. HeteroMosaic reconciles these differing runtimes by using independent HIP streams to preserve correctness within a micro-batch, while combining hipEventRecord and hipStreamWaitValue32 [11, 12] as host-mediated signaling mechanisms to preserve dependencies across multiple iGPU streams and the NPU.

Evaluation

To evaluate HeteroMosaic, we study three current-generation AMD Ryzen™ AI devices spanning NPU-heavy, balanced, and iGPU-heavy designs: the AMD Ryzen™ AI 7 350, AMD Ryzen™ AI 9 HX 370, and AMD Ryzen™ AI Max+ 395, respectively. Our evaluation is organized to test the two-stage argument of this paper. First, we use microbenchmarks to measure the directly accelerable core of inference and to evaluate whether the achievable behavior of GEMM aligns with the roofline-based opportunity identified in Section 4. Second, we use end-to-end LLM inference to test whether HeteroMosaic can expose and recover that opportunity at the graph level under realistic runtime constraints. We then contextualize these results against relevant prior heterogeneous frameworks, analyze the contribution of HeteroMosaic’s individual components through ablations, and study whether the resulting speedups translate into lower energy under the TDP-constrained behavior of modern SoCs.

5.3.2 Runtime NPU Management. AMD Ryzen™ AI NPUs, when implemented well, can exhibit lower GEMM shape sensitivity than prior mobile NPU paths such as HeteroInfer, where fixed systolicarray tiling and tensor order/shape alignment introduce stage-, order-, and shape-sensitive performance variation [21]. However, this comes at the cost of fixed reconfiguration overhead. Prior work [75] has shown that NPU reconfiguration can negatively impact end-to-end performance. On AMD Ryzen™ AI, the NPU exposes two levels of configuration: (i) datapath reconfiguration, which is expensive because it reprograms tile internals and switchbox routes, and (ii) DMA reconfiguration, which updates the DMA engines and is effectively negligible even for microsecond-scale kernels. The central challenge is therefore to choose an NPU schedule that maximizes reuse of datapath configurations. This issue becomes even more important in parallel micro-batched schedules,

6.1

Microbenchmarks

Microbenchmarks isolate the directly accelerable GEMM core of inference and provide the cleanest measured view of the rooflineidentified opportunity. They also quantify the split-dimension design space introduced in Section 2.4 and Section 5.1 by answering four questions. First, do the measured gains match the opportunity predicted by the roofline model? Second, how do those gains vary across SoCs with different iGPU–NPU balance points? Third, which tensor-parallel split dimension is most effective for different GEMM shapes? Fourth, how much of the observed behavior reflects 12

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

1.0 0.5 0.0

0.1

0.2

0.3

0.4

0.5

0.6

Fraction of GEMM M on GPU

K-split

Speedup over iGPU

3.5

0.8

0.9

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

3.0 2.5 2.0 1.5 1.0 0.5 0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fraction of GEMM K on GPU

0.8

0.9

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

3.0

N-split

0.7

2.5 2.0 1.5 1.0 0.5 0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fraction of GEMM N on GPU

0.8

0.9

1.0

Speedup over iGPU

1.5

1.4

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.1

0.2

0.3

0.4

0.5

0.6

Fraction of GEMM M on GPU

2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4

1.2 1.0 0.8 0.6

0.8

0.9

1.0

0.0

1.2

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fraction of GEMM K on GPU

0.8

0.9

1.0

1.0 0.8

1.0

0.0

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.1

0.2

0.3

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Fraction of GEMM M on GPU

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.6 0.4 0.2 0.0 1.4

1.75 1.50 1.25 1.00 0.75 0.50 0.25

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.4 0.7

Speedup over iGPU

2.0

2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.0

AMD Ryzen™ AI Max+ 395

Speedup over iGPU

2.5

Speedup over iGPU

M-split

Speedup over iGPU

3.0

Speedup over iGPU

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

3.5

Speedup over iGPU

AMD Ryzen™ AI 9 HX 370 Speedup over iGPU

AMD Ryzen™ AI 7 350

Jun et al.

1.2 1.0 0.8

0.1

0.2

0.3

Fraction of GEMM K on GPU

2048_2048_2048 4096_4096_4096 8192_4096_14336 8192_8192_8192 8192_14336_4096

0.6 0.4 0.2

0.4

0.5

0.6

0.7

0.8

Fraction of GEMM N on GPU

0.9

1.0

0.0

0.1

0.2

0.3

Fraction of GEMM N on GPU

Figure 7: Speedup of heterogeneous execution for GEMMs of varying sizes on the NPU and iGPU, normalized to the iGPU baseline. Results are shown for M-, K-, and N-dimension tensor splits across three AMD Ryzen™ AI systems. AMD Ryzen™ AI 9 HX 370, where DDR bandwidth and iGPU compute capability are more limited than on the AMD Ryzen™ AI Max+ 395, 𝐾-split can outperform 𝑀-split for larger GEMMs. Using the notation 𝑀×𝐾×𝑁 for GEMM shapes, with 𝐴 ∈ R𝑀 ×𝐾 , 𝐵 ∈ R𝐾 ×𝑁 , and 𝐶 ∈ R𝑀 ×𝑁 for 𝐶 = 𝐴𝐵, examples of large GEMMs where 𝐾-split becomes favorable include 8192×8192×8192 and 8192×14336×8192. In these cases, splitting along 𝐾 creates subproblems that better utilize both accelerators, despite the additional reduction step. The 𝐾-split results also illustrate why empirical calibration is necessary. Several GEMM shapes, including 2048×2048×2048, 4096× 4096×4096, and 8192×4096×14336, perform noticeably worse than what a simple FLOP-balanced split would suggest. This behavior is consistent with backend shape sensitivity: the split creates local GEMM shapes that map awkwardly onto the backend tiling and compute resources, reducing occupancy or tiling efficiency even when the total arithmetic work appears balanced. This effect is most pronounced on the AMD Ryzen™ AI 9 HX 370 for the 4096×4096×4096 GEMM, which shows sharp performance drops at certain split ratios, such as around 37% and 56% GPU assignment. Thus, the best split cannot be inferred from FLOP balance alone; it depends on how the partitioned shape maps onto the physical compute resources of each backend.

genuine heterogeneous opportunity rather than shape sensitivity, memory contention, or kernel artifacts [1, 59]? We map GEMM operations heterogeneously using the 𝑀-, 𝐾-, and 𝑁 -split strategies described in Section 2.4. Figure 7 summarizes the results. Each point is averaged over 1024 GEMM runs, with error bars indicating variation across those repeated runs. This variation primarily reflects SoC power-management behavior, where execution can move between a transient turbo/high-power state and a sustained TDP-limited state. Shorter GEMMs are especially sensitive to which regime they execute under, which is evident across platforms for shapes such as 2048×2048×2048. Longer GEMMs can also show variation when a run spans a turbo-to-sustained transition or begins from a different thermal/power state, as seen for larger shapes such as 8192×14336×4096 on the AMD Ryzen™ AI 9 HX 370. Overall, the measured trends follow the roofline intuition: the NPU-heavy AMD Ryzen™ AI 7 350 exposes the largest heterogeneous speedups over the iGPU, while the balanced AMD Ryzen™ AI 9 HX 370 shows moderate but still substantial gains, and the iGPU-heavy AMD Ryzen™ AI Max+ 395 shows the smallest headroom because the iGPU baseline is already strong. Across the full sweep, 𝑀-split is the most consistently effective strategy. 𝑀-split avoids cross-device reduction and works well for small and moderate GEMMs where duplicated weight reads do not dominate execution. However, on the AMD Ryzen™ AI 7 350 and 13

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

𝑁 -split is generally the weakest strategy in our measurements. Although it can reduce duplicate weight reads in principle, it interacts poorly with the packed weight layout used by our heterogeneous kernels. Because 𝑁 -split partitions columns of 𝐵 and 𝐶, it can require strided weight reads and scatter-like writes into the final 𝐶 tensor. These non-contiguous accesses reduce memory coalescing and introduce gather/scatter overhead, often preventing 𝑁 -split from converting its reduced weight traffic into measured speedup. On the AMD Ryzen™ AI Max+ 395, 𝑀-split remains best across the evaluated shapes because higher memory bandwidth and a stronger iGPU make simple output-row partitioning more effective than the reduction overhead of 𝐾-split or the gather/scatter overhead of 𝑁 -split. These results provide the empirical bridge between the roofline model and the HeteroMosaic scheduler. For compute-heavy GEMMs, the measured iGPU–NPU speedups track the roofline model’s predicted opportunity, while deviations mainly reflect shape sensitivity, memory contention, and runtime overheads. However, the best tensor-parallel strategy still depends on GEMM shape, split ratio, SoC balance, memory bandwidth, layout compatibility, and accelerator-specific shape sensitivity. HeteroMosaic therefore uses these microbenchmarks to calibrate candidate splits, while the global scheduler ultimately determines which split that improves the end-to-end critical path.

6.2

bubbles on the iGPU path so that the effective ratio matches the performance characteristics described in HeteroInfer.4 Our iGPU baseline uses the same micro-batching schedule as llama.cpp to keep the comparison fair, yet it still consistently outperforms llama.cpp, largely because our iGPU kernels are more tightly optimized for AMD Ryzen™ AI. Our NPU-oriented baseline follows a HeteroInfer-like placement strategy, in which linear projections run on the NPU while operations such as attention and normalization remain on the iGPU. 6.2.1 Analysis of Prefill Results. Figure 8 shows a clear devicelevel trend across all evaluated models. HeteroMosaic performs best on the left column (AMD Ryzen™ AI 7 350), remains clearly beneficial in the middle column (AMD Ryzen™ AI 9 HX 370), and still provides consistent gains in the right column (AMD Ryzen™ AI Max+ 395). This trend follows the underlying SoC balance point: as the system shifts from NPU-heavy to balanced and then to iGPUheavy, the total heterogeneous headroom above the iGPU baseline shrinks. Averaged across the raw prefill results, HeteroMosaic achieves about 2.30×, 1.50×, and 1.17× speedup over the iGPU baseline on the AMD Ryzen™ AI 7 350, AMD Ryzen™ AI 9 HX 370, and AMD Ryzen™ AI Max+ 395, respectively. This ordering is consistent with the roofline and GEMM microbenchmarks, but the absolute end-to-end gains are lower than the operation-level peaks because full inference also includes non-partitionable operators, attention and KV-cache bandwidth, synchronization overhead, finite split granularity, NPU reconfiguration, unified-memory contention, shape sensitivity, and DVFS/thermal effects. The HeteroInfer-style tensor-partitioning baseline falls even further below the GEMM microbenchmark peaks, showing that local operation partitioning alone does not preserve GEMM-level opportunity once embedded in the full execution graph. A key point in Figure 8 is that HeteroMosaic consistently outperforms both comparison strategies across all three devices. Relative to our NPU-oriented baseline, HeteroMosaic is still about 1.35× faster on the AMD Ryzen™ AI 7 350, 1.60× faster on the AMD Ryzen™ AI 9 HX 370, and 2.86× faster on the AMD Ryzen™ AI Max+ 395 on average. This comparison makes clear that HeteroMosaic’s advantage is not just a byproduct of favoring the NPU. Instead, these gains come from exposing additional graph-level overlap and then co-optimizing accelerator allocation around the actual critical path. HeteroMosaic also remains ahead of HeteroInfer across the full figure, with average improvements of about 1.22×, 1.28×, and 1.32× on the same three devices. The rightmost column is especially instructive: even though the absolute iGPU-normalized headroom is smaller there, HeteroMosaic still retains an advantage because schedule-level overlap and latency shaping continue to matter after simple tensor placement has largely run out of room. Across models, HeteroMosaic also shows that the amount of recoverable heterogeneous opportunity depends on model structure. Models with larger embedding dimensions, such as Qwen2.5-14B and Llama3-70B, naturally expose more operation-level heterogeneous work. Phi-3.5-3.8B is also relatively favorable because its

End-to-End LLM Evaluation

Thus, to evaluate whether and to what extent the heterogeneous gains observed in the microbenchmarks translate to the full endto-end application, we evaluate HeteroMosaic across a series of LLMs with different model sizes and prompt lengths, as shown in Figure 8. This setting is substantially more challenging than the microbenchmark setting because full-model inference includes attention and KV-cache traffic, non-partitionable operators, layerwise shape variation, synchronization overhead, NPU runtime behavior, unified-memory contention, and DVFS/thermal effects. For each model–device pair, we report results across several prompt lengths and normalize performance to the corresponding iGPU baseline. Our comparison includes five configurations: our iGPU-only baseline, our NPU-oriented baseline, llama.cpp [27]2 , a HeteroInferstyle baseline, and HeteroMosaic. For HeteroInfer [21], we implement a faithful re-creation based on the published design, since the original system is not open source and is not natively compatible with AMD Ryzen™ AI.3 To validate that this re-creation captures the published behavior, we emulate the reported iGPU:NPU imbalance of roughly 1:10 by inserting 2We base these results on llama.cpp commits from March 2026 and use the ROCm

backend with Q4_K_S, the closest available proxy to AWQ-style W4A16 execution in llama.cpp. Q4_K_S is a llama.cpp-specific 4-bit K-quant format that keeps the linear-layer weights in 4-bit form, whereas Q4_K_M uses a mixed variant that retains selected tensors at higher precision. We therefore use Q4_K_S to better approximate a uniform 4-bit weight-only baseline. 3When details are ambiguous, we use conservative implementation choices that avoid penalizing the baseline. For example, AMD Ryzen™ AI NPUs exhibit less shape sensitivity than reported in the original paper, and Qualcomm’s QNN stack differs from AMD Ryzen™ AI in both quantization support and kernel availability [35, 73, 74]. We therefore use the strongest corresponding implementation available on our platform: fused IRON-based NPU kernels together with our heterogeneity-compatible iGPU AWQ W4A16 kernels, rather than directly mirroring QNN’s channel-wise quantization path.

4 Under this calibration, on AMD Ryzen™ AI 7 350 with Llama3-8B at prompt length

256, our re-creation reaches roughly 6.1× versus the reported 5.6× relative to its respective iGPU baseline. We attribute the remaining gap to differences in device characteristics and implementation details not fully specified in the paper. 14

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

Phi3.5-3.8B

AMD Ryzen™ AI 7 350

AMD Ryzen™ AI 9 HX 370

2.25

1.6

2.00

1.4

1.75 1.50

llama.cpp(iGPU) iGPU-baseline NPU-baseline

1.25

HeteroInfer HeteroMosaic

Llama3-8B Qwen2.5-14B Llama3-70B

2.4 2.2 2.0 1.8 1.6 1.4 1.2 1.0 0.8

1.0

0.8 llama.cpp(iGPU) iGPU-baseline NPU-baseline

HeteroInfer HeteroMosaic

2048

4096

8192

16384

0.2

2048

4096

8192

16384

llama.cpp(iGPU) iGPU-baseline NPU-baseline

1.4

1024

2048

4096

HeteroInfer HeteroMosaic

0.8

2048

4096

8192

16384

4096

1.0

8192

16384

2.5 2.0 llama.cpp(iGPU) iGPU-baseline NPU-baseline

1.5

HeteroInfer HeteroMosaic

1.0 1024

2048

4096

8192

16384

1024

2048

HeteroInfer HeteroMosaic

4096

16384

8192

16384

8192

16384

0.8 llama.cpp(iGPU) iGPU-baseline NPU-baseline

0.6

HeteroInfer HeteroMosaic

0.4 1024

1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8 0.7

8192

HeteroInfer HeteroMosaic

1.0 llama.cpp(iGPU) iGPU-baseline NPU-baseline

0.8 2048

16384

1.2

1024

8192

0.4 1024

1.4

HeteroInfer HeteroMosaic

llama.cpp(iGPU) iGPU-baseline NPU-baseline

0.6

1.6

llama.cpp(iGPU) iGPU-baseline NPU-baseline

4096

0.8

1.0 16384

2048

1.0

1.2

8192

1024 1.2

1.6

HeteroInfer HeteroMosaic

0.4 1024

1.8

llama.cpp(iGPU) iGPU-baseline NPU-baseline

0.6

0.4

0.50

2.75 2.50 2.25 2.00 1.75 1.50 1.25 1.00 0.75

1.2

0.6

0.75

AMD Ryzen™ AI Max+ 395 1.2

0.8

1.00

1024

Jun et al.

2048

4096

8192

16384

1024

2048

4096

1.1 llama.cpp(iGPU) iGPU-baseline NPU-baseline

1.0

HeteroInfer HeteroMosaic

0.9 0.8 0.7

llama.cpp(iGPU) iGPU-baseline NPU-baseline

0.6

HeteroInfer HeteroMosaic

0.5 0.4 1024

2048

4096

8192

16384

1024

2048

4096

Figure 8: End-to-end LLM prompt latency speedup across AMD Ryzen™ AI devices and across multiple AWQ W4A16 models from Hugging Face. Columns correspond to devices and rows to models. In each subplot, the x-axis shows input prompt length in tokens, and the y-axis shows speedup over the iGPU baseline, computed as baseline latency divided by method latency; higher is better. We note that for Llama3-70B, the 16K runs without micro-batching failed due to OOM. Table 2: Normalized TPS decode performance (HeteroMosaic / iGPU) across three AMD Ryzen™ AI platforms.

MHA design retains larger 𝐾 and 𝑉 projections than the GQAbased models. Even so, the central pattern is consistent across rows: HeteroMosaic remains the strongest method because it combines operation-level partitioning with graph-level overlap and runtimeaware schedule shaping, rather than relying only on a placement policy for split-friendly dense layers. At long contexts, especially for Llama3-70B, the gains of all methods become more muted as execution moves deeper into the sustained-power regime, where the SoC has moved past short-lived turbo behavior and settles into a lower steady-state throughput under thermal and power limits. Even in this constrained regime, HeteroMosaic maintains the strongest overall performance. At 16K, it is the only heterogeneous method among these baselines that completes across all three devices while preserving positive speedup over the iGPU baseline. Beyond exposing heterogeneous overlap, its micro-batching also manages memory footprint, allowing large cases such as Llama370B at 16K to avoid OOM failures. Taken together, the prefill results show that tensor placement alone recovers only part of the available heterogeneous opportunity, whereas HeteroMosaic more fully

Model Phi3.5-3.8B Llama3-8B Qwen2.5-14B Llama3-70B

AMD Ryzen™ AI 7 350

AMD Ryzen™ AI 9 HX 370

AMD Ryzen™ AI Max+ 395

1.00× 1.13× 1.07× 1.09×

1.11× 1.02× 0.99× 0.98×

1.11× 0.995× 0.996× 1.00×

realizes that opportunity by jointly reasoning about graph structure, operation shape, SoC balance, and runtime system effects. 6.2.2 Analysis of Decode Results. Decode performance is fundamentally memory-bound. However, similar to HeteroInfer [21], we find that heterogeneous GEMV can still be beneficial when the iGPU alone does not fully utilize the available memory system or when GEMV shapes underutilize the iGPU. As shown in Table 2, this occurs most consistently on the AMD Ryzen™ AI 7 350 for the 15

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

turbo

Current (A)

16.6% less J

2.75 2.50

0.33x 2.45x

Time (s)

2.25

2.13x

2.00

2.08x 0.24x

1.75

0.28x 0.28x

Ryzen AI 7 350 Ryzen AI 9 HX 370 Ryzen AI MAX+ 395

larger evaluated models, where the weaker iGPU benefits modestly from additional CPU/NPU parallelism. On the AMD Ryzen™ AI 9 HX 370 and AMD Ryzen™ AI Max+ 395, heterogeneous decode is usually neutral or slightly harmful because the stronger iGPU already approaches the useful memory-bandwidth limit and the added synchronization overhead can outweigh the benefit. Phi3.53.8B is an exception: its smaller GEMV dimensions leave more room for shape- and launch-efficiency effects, allowing modest heterogeneous decode gains on the larger platforms as well. These results are consistent with the broader thesis of the paper: decode offers limited heterogeneous headroom on unified-memory SoCs once the iGPU can already drive sufficient memory traffic, whereas prefill remains the primary setting in which graph-level scheduling and latency shaping recover substantial heterogeneous benefit.

1.50

1.46x

1.25

1024

2048

4096

8192

1.00

ize

Prompt s

Figure 10: Ablation of HeteroMosaic for Llama3-8B across AMD Ryzen™ AI devices and prompt sizes. Bar height indicates speedup over the iGPU baseline, while colored segments show the contribution of each optimization.

the globally optimal schedule. In particular, tensor parallelism, asymmetric 𝜇𝐵, DVFS-aware scheduling, critical-interval microbatching, and NPU scheduling build on one another rather than contributing independently. For example, the full benefit of tensor parallelism cannot be realized to the same extent without criticalinterval micro-batched scheduling, which exposes additional graphlevel parallelism for heterogeneity to exploit. We therefore perform ablation by removing each component, re-co-optimizing the remaining system, and then measuring the resulting speedup so that each component’s contribution is isolated as fairly as possible. Figure 10 helps explain where HeteroMosaic’s gains come from. First, tensor parallelism provides the raw heterogeneous leverage by allowing compute to be mapped across devices; across systems and prompt lengths, this is the dominant source of speedup. Second, critical-interval scheduling determines whether that leverage survives globally, since locally beneficial mappings do not necessarily translate into lower end-to-end latency without schedule-level coordination. Third, as prompt length grows and attention becomes more dominant, the contribution of asymmetric 𝜇𝐵 becomes more visible because it expands the schedule space and exposes additional schedulable work. Fourth, DVFS-aware scheduling matters because realized performance is not nominal performance: by partitioning work between the NPU and iGPU in a DVFS-aware manner, HeteroMosaic can keep the SoC in a higher turbo state for longer, increasing effective performance beyond what a static mapping would suggest. Finally, NPU scheduling matters because runtime effects such as reconfiguration overhead and redundant weight movement can otherwise erase a meaningful fraction of the gain.

Power Study

A natural concern with heterogeneous execution is that activating more accelerators will simply increase power. However, on TDP-constrained edge SoCs, as shown in Figure 9, the iGPU alone often already operates near the system power limit.5 Instead, the more relevant question is whether heterogeneous scheduling can complete more useful work within roughly the same power envelope. Under this view, any speedup achieved without materially increasing peak power should reduce the total energy required to complete inference. We validate this experimentally by measuring current using a lab bench power supply [60] at a sampling rate of 10 Hz. The results, summarized in Figure 9, support this intuition: HeteroMosaic improves performance without introducing a corresponding increase in peak current. We also observe that AMD Ryzen™ AI SoCs exhibit a turbo regime in which the chip operates at an elevated power level for a short interval before settling into a lower sustained operating point. HeteroMosaic can extend this high-performance interval, most clearly on the AMD Ryzen™ AI Max+ 395, which is consistent with our broader thesis that heterogeneity is most effective when it is co-optimized against real system behavior.

6.4

2.30x

Figure 9: Measured current (A) over time during the same LLM inference workload across three platforms.

6.3

DVFS

3.00

45.3% less J

Asymmetric B NPU Scheduling

0.27x

29.5% less J

Tensor Parallel Critical Interval B

Ryzen AI 7 350-HeteroMosaic Ryzen AI 7 350-iGPU Ryzen AI 9 HX 370-HeteroMosaic Ryzen AI 9 HX 370-iGPU Ryzen AI Max+ 395-HeteroMosaic Ryzen AI Max+ 395-iGPU

Speedup over iGPU

Jun et al.

Ablation Study

An ablation study of HeteroMosaic is by nature non-trivial as its optimizations are co-dependent: removing one component changes

6.5

5 In heterogeneous execution, adding iGPU participation can cause the package to

Comparison with Other Frameworks

To contextualize HeteroMosaic against llama.cpp iGPU-ROCm [27] and production AMD LLM frameworks, including FastFlowLM

approach its TDP limit through iGPU DVFS regardless of the exact NPU fraction, so heavier NPU use does not necessarily translate into lower instantaneous power. 16

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

AMD Ryzen AI 9 HX 370

18.32s

AMD Ryzen AI 7 350 20.0

llama.cpp(iGPU)

AMD Ryzen AI MAX+ 395

FLM

RyzenOGA

HX 370, HeteroMosaic continues to provide measurable throughput advantages over FLM and Ryzen-OGA while remaining competitive with, or exceeding, the llama.cpp baseline.

HeteroMosaic

0.0

1024

4096

1024

4096

1024

2048

3.40s

2.45s

4.24s 3.74s 1.19s

1.60s

4.97s

2048

0.81s 2.90s 1.83s 0.59s

8.17s 8.86s

2048

4.70s 5.14s 3.61s 2.43s

2.5

2.21s 2.54s 1.91s 1.42s

5.0

6.40s

7.5

5.19s 3.60s 2.66s

8.05s

10.0

8.07s 8.88s

10.28s 8.02s 9.80s

15.0

3.72s 2.96s 1.91s 1.39s

Prefill / TTFT (seconds)

17.5

12.5

4096

40.00

37.18

38.92

10.36

8.82

2048

4096

1024

2048

4096

Discussion and Future Work

Although the evaluation in this paper uses AMD Ryzen™ AI as a concrete implementation target, the broader goal is to provide a principled methodology to reason about when fine-grained heterogeneity becomes useful. In its current realization, HeteroMosaic benefits from three platform properties: programmable accelerators, shared or low-cost memory access across devices, and lowoverhead synchronization mechanisms. These properties are not unique to AMD Ryzen™ AI, but they are also not guaranteed across all heterogeneous systems. Thus, future systems that build on the insights of HeteroMosaic would need to develop the corresponding platform-specific runtime mechanisms, including GPU/NPU kernels, memory-layout compatibility, queue management, and synchronization, while preserving the same scheduling abstraction. Beyond LLM inference, the same scheduling formulation should be applicable to other transformer-based workloads, including vision-language-action models for robotics such as OpenPI/𝜋0 , 𝜋0.5 , and NVIDIA GR00T [52, 55, 63–66], as well as transformer-based diffusion and flow models [18, 19, 24, 62]. These workloads contain fundamentally similar mixtures of dense projections, attention-like stages, memory-sensitive operators, and cross-stage dependencies, making them natural candidates for graph-level heterogeneous scheduling. HeteroMosaic, when applied to these similar workloads, may expose additional opportunities because their modalityspecific components can create more heterogeneous scheduling choices than text-only LLMs. However, these workloads also introduce new constraints, such as image-token preprocessing, vision encoder stages, control-loop latency requirements, and diffusionstep dependencies. Thus, extending HeteroMosaic to these settings would require extending the current model from a single transformer execution path to multiple heterogeneous model stages that connect and execute together, while also accounting for how modality-specific stages interact with the critical path. The cross-platform TTFT results also suggest a hardware-design implication where a smaller heterogeneous platform may have the potential to outperform a stronger iGPU-only baseline if the software can expose enough parallel work and coordinate the accelerators efficiently. This does not imply that larger iGPUs are unhelpful; rather, it suggests that balanced accelerator design, shared-memory efficiency, and low-overhead synchronization can matter as much as raw single-accelerator scale for transformer inference. Finally, extending this formulation to heterogeneous datacenter systems is another possible direction, where the same heterogeneous scheduling idea could be applied across a mixture of accelerators spanning different CPU and GPU generations, compute capabilities, and more unconventional accelerators such as fieldprogrammable gate arrays (FPGAs). However, this extension would require explicit communication modeling. Unlike unified-memory edge SoCs, datacenter accelerators often communicate through PCIe [61], CXL [22], NVLink [54], or network-level links. In that

26.33

33.35 10.59

14.68

1024

14.06 15.68

4096

15.62 16.44

2048

9.89

15.33

15.57 10.63 15.55 16.94

1024

9.94

11.51 11.25 10.58 13.48

12.18 11.49 12.36 14.19

29.89

HeteroMosaic

37.50

AMD Ryzen AI MAX+ 395

RyzenOGA

FLM

39.24

AMD Ryzen AI 9 HX 370

llama.cpp(iGPU)

39.78

AMD Ryzen AI 7 350

12.55 12.45 12.91 14.59

Decode Throughput / TPS (tokens/s)

(a) TTFT comparison. Lower is better. 40

Jun et al.

(b) TPS comparison. Higher is better.

Figure 11: Cross-framework comparison for Llama3-8B across AMD Ryzen™ AI platforms. (FLM) [25] and the Ryzen ONNX Hybrid Runtime (Ryzen-OGA)6 [10], we evaluate Llama3-8B [50] across AMD Ryzen™ AI platforms. We report raw TTFT and TPS in Figure 11 so that cross-platform and cross-device performance can be compared more directly. These comparisons are not numerically identical across implementations: for example, llama.cpp on the iGPU uses integer computation, Ryzen-OGA uses BFP16 [79] activations, and HeteroMosaic uses BF16, while the exact internal behavior of FLM is not openly documented. Therefore, these experiments should be interpreted as a coarse cross-framework and cross-platform comparison rather than a strict apples-to-apples evaluation. For TTFT, HeteroMosaic is consistently the strongest solution across the evaluated platforms, with the largest benefits appearing when heterogeneous resources can be effectively exposed and scheduled at the graph level. A particularly notable result is that HeteroMosaic on the AMD Ryzen™ AI 7 350 achieves a 2048-token TTFT of 2.66 s, which is not only substantially faster than the same-platform llama.cpp baseline of 8.05 s, but also faster than the stronger llama.cpp iGPU baseline on the AMD Ryzen™ AI 9 HX 370 of 4.70 s. This corresponds to a 1.76× TTFT advantage over the AMD Ryzen™ AI 9 HX 370 llama.cpp baseline, demonstrating that well-orchestrated heterogeneous execution can outperform stronger iGPU-only baselines. This suggests that balanced heterogeneous compute, low-overhead shared memory, and fine-grained synchronization can provide more practical benefit for transformer inference than simply increasing iGPU resources alone. For TPS, the picture is more nuanced because decode is more strongly constrained by memory bandwidth. HeteroMosaic remains competitive and often leads, but the gains are smaller than for TTFT, especially on the AMD Ryzen™ AI Max+ 395, where both HeteroMosaic and llama.cpp already operate near the bandwidth ceiling. In contrast, on the AMD Ryzen™ AI 7 350 and AMD Ryzen™ AI 9 6 Ryzen-OGA runs prompt encoding on the NPU and decode on the iGPU.

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

setting, accelerator placement must account for activation movement, synchronization latency, collective overheads, and interconnect contention. The same critical-interval optimization principle may still apply, but the cost model would first need to treat communication as a first-class scheduling constraint and then apply heterogeneous scheduling around that constraint.

[10] AMD. 2026. OnnxRuntime GenAI (OGA) Flow — Ryzen AI Software. https: //ryzenai.docs.amd.com/en/latest/hybrid_oga.html. Accessed: 2026-03-23. [11] AMD ROCm Team. 2024. HIP Runtime API: hipEventRecord. https://rocm.docs.amd.com/projects/HIP/en/develop/doxygen/html/group___ event.html#ga5df2309c9f29ca4c8e669db658d411b4 Accessed: 2025-07-28. [12] AMD ROCm Team. 2024. HIP Runtime API: hipStreamWaitValue32. https://rocm.docs.amd.com/projects/HIP/en/docs-develop/reference/hip_ runtime_api/modules/stream_memory_operations.html Accessed: 2025-07-28. HIP Runtime API: Streams and Synchro[13] AMD ROCm Team. 2025. nization. https://rocm.docs.amd.com/projects/HIP/en/latest/understand/ programming_model.html Accessed: 2025-07-28. [14] Anthropic. 2025. Claude Code. https://www.anthropic.com. AI coding assistant built on Claude; accessed 31 Jul 2025. [15] Apple. [n. d.]. The Most Powerful Neural Engine Ever. https://www.apple.com/ newsroom/2024/05/apple-introduces-m4-chip. [16] Apple. 2026. Core ML | Apple Developer Documentation. https://developer.apple. com/documentation/coreml. Accessed: 2026-03-23. [17] Apple Inc. 2026. Metal | Apple Developer Documentation. https://developer. apple.com/documentation/metal. Accessed: 2026-05-17. [18] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. 2023. All are Worth Words: A ViT Backbone for Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22669– 22679. https://doi.org/10.48550/arXiv.2209.12152 arXiv:2209.12152 [cs.CV] [19] Black Forest Labs. 2024. FLUX. https://github.com/black-forest-labs/flux. Official inference repository for FLUX.1 models. [20] Cade Brown, Ahmad Abdelfattah, Stanimire Tomov, and Jack Dongarra. 2020. Design, Optimization, and Benchmarking of Dense Linear Algebra Algorithms on AMD GPUs. In 2020 IEEE High Performance Extreme Computing Conference (HPEC). 1–7. https://doi.org/10.1109/HPEC43674.2020.9286214 [21] Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. https://doi.org/10.1145/3731569.3764808 [22] Compute Express Link Consortium. [n. d.]. Compute Express Link Specification. CXL Consortium Specification. https://www.computeexpresslink.org Accessed: 2026-06-17. [23] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems, Vol. 35. 16344–16359. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135 [24] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M"uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. https://doi.org/10.48550/arXiv.2403.03206 arXiv:2403.03206 [cs.CV] [25] FastFlowLM. 2026. FastFlowLM. https://github.com/FastFlowLM/FastFlowLM. GitHub repository, accessed 2026-04-04. [26] FLAME Project. 2026. BLIS: BLAS-like Library Instantiation Software Framework. https://github.com/flame/blis. Accessed: 2026-05-17. [27] GGML-ORG. [n. d.]. llama.cpp. https://github.com/ggml-org/llama.cpp. [28] ggml-org. 2024. What’s the difference between batch-size and ubatch-size? Discussion #6328. https://github.com/ggml-org/llama.cpp/discussions/6328. Accessed: 2026-03-24. [29] ggml-org. 2026. ggml: Tensor library for machine learning. https://github.com/ ggml-org/ggml. Accessed: 2026-05-17. [30] ggml-org. 2026. llama.cpp Build Documentation. https://github.com/ggml-org/ llama.cpp/blob/master/docs/build.md. Accessed: 2026-05-17. [31] ggml-org. 2026. llama.cpp common.h. https://github.com/ggml-org/llama.cpp/ blob/master/common/common.h. Accessed: 2026-03-24. [32] ggml-org. 2026. llama.cpp Quantization Tool Documentation. https://github. com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md. Accessed: 2026-05-17. [33] ggml-org. 2026. llama.cpp RPC Backend Documentation. https://github.com/ ggml-org/llama.cpp/blob/master/tools/rpc/README.md. Accessed: 2026-05-17. [34] glassescrab. 2026. mlir-aie: An Open-Source Fork of the IRON API and MLIRBased AI Engine Toolchain. https://github.com/glassescrab/mlir-aie. GitHub repository, accessed 2026-03-27. [35] Zixu Hao, Jianyu Wei, Tuowei Wang, Minxing Huang, Huiqiang Jiang, Shiqi Jiang, Ting Cao, and Ju Ren. 2026. Scaling LLM Test-Time Compute with Mobile NPU on Smartphones. In Proceedings of the 21st European Conference on Computer Systems (EuroSys ’26). 2157–2172. https://doi.org/10.1145/3767295.3769382 [36] Jyothi Hariharan, Rahul Rama Varior, and Sunil Karunakaran. 2023. Real-time Driver Monitoring Systems on Edge AI Device. arXiv preprint arXiv:2304.01555 (2023). https://doi.org/10.48550/arXiv.2304.01555 [37] Mark Hill and Vijay Janapa Reddi. 2019. Gables: A roofline model for mobile socs. In 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 317–330.

Conclusion

Edge AI is increasingly defined by heterogeneous SoCs, yet current inference systems still treat heterogeneity too coarsely and therefore underutilize the full combination of on-chip compute resources. In this paper, we present HeteroMosaic, a heterogeneity-first framework that treats edge LLM inference primarily as a scheduling problem rather than merely a placement problem. HeteroMosaic first uses a heterogeneous roofline model to determine when heterogeneity should help in principle, then restructures execution through causal micro-batching to expose latent cross-device overlap, and finally realizes that opportunity through latency shaping and trace-guided critical-interval co-optimization under real system effects. Across three AMD Ryzen™ AI platforms, HeteroMosaic improves end-to-end LLM inference over strong baselines without increasing peak power. This shows that, on TDP-constrained edge SoCs, cooptimized heterogeneity can complete more work within the same practical power envelope, improving both energy efficiency and tokens per watt. Although the exact implementation depends on platform-specific runtime mechanisms, the underlying scheduling principles are broader than any one SoC family. Taken together, HeteroMosaic shows that fine-grained heterogeneity, when exposed and controlled through scheduling, is a practical path toward faster and more energy-efficient edge LLM inference.

Acknowledgment We thank Paul Hartke from AMD, and other AMD colleagues who provided valuable feedback and technical guidance throughout this work. This work is supported in part by the AMD Center of Excellence at UIUC.

References [1] 2024. Matrix Multiplication Performance Guide. Technical Report. NVIDIA. https://docs.nvidia.com/deeplearning/performance/dl-performancematrix-multiplication/index.html [2] Advanced Micro Devices, Inc. 2025. AMD Ryzen™ AI 9 HX 370. https://www.amd.com/en/products/processors/laptop/ryzen/ai-300series/amd-ryzen-ai-9-hx-370.html [3] Advanced Micro Devices, Inc. 2026. AMD Zen Deep Neural Network Library. https://www.amd.com/en/developer/zendnn.html. Accessed: 2026-05-17. [4] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–136. [5] AMD. [n. d.]. ROCm. https://github.com/ROCm. [6] AMD. 2023. How to accelerate AI applications on RDNA 3 using WMMA. https: //gpuopen.com/learn/wmma_on_rdna3/ Describes RDNA 3 WMMA instructions and supported data types (FP16, BF16, INT8, INT4). Accessed: 2025-11-02. [7] AMD. 2025. AMD XDNA Architecture. https://www.amd.com/en/technologies/ xdna.html. Accessed: 2025-07-28. [8] AMD. 2026. AMD Ryzen™ AI 7 350. https://www.amd.com/en/products/ processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-7-350.html. Product page, accessed 2026-04-02. [9] AMD. 2026. AMD Ryzen™ AI Max+ 395. https://www.amd.com/en/products/ processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html. Product page, accessed 2026-04-02. 18

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

[38] Huawei. 2026. CANN: Ascend AI Computing Platform. https://www.hiascend. com/en/cann. Accessed: 2026-05-17. [39] Erika Hunhoff, Joseph Melber, Kristof Denolf, Andra Bisca, Samuel Bayliss, Stephen Neuendorffer, Jeff Fifield, Jack Lo, Pranathi Vasireddy, Phil JamesRoxby, and Eric Keller. 2025. Efficiency, Expressivity, and Extensibility in a Close-to-Metal NPU Programming Interface. arXiv:2504.18430 [cs.SE] https: //arxiv.org/abs/2504.18430 [40] Intel. [n. d.]. Quick overview of Intel’s Neural Processing Unit (NPU). https: //intel.github.io/intel-npu-acceleration-library/npu.html. [41] Intel. 2026. OpenVINO Documentation. https://docs.openvino.ai/. Accessed: 2026-05-17. [42] kernel.org. [n. d.]. DRM Memory Management. https://www.kernel.org/doc/ html/v4.15/gpu/drm-mm.html. [43] Khronos Group. 2026. OpenCL Registry. https://registry.khronos.org/OpenCL/. Accessed: 2026-05-17. [44] Khronos Group. 2026. SYCL 2020 Specification. https://registry.khronos.org/ SYCL/specs/sycl-2020/html/sycl-2020.html. Accessed: 2026-05-17. [45] Khronos Group. 2026. Vulkan Specification. https://registry.khronos.org/vulkan/ specs/latest/html/vkspec.html. Accessed: 2026-05-17. [46] Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim. 2025. LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 544–558. https://doi.org/10.1145/3695053.3731092 [47] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. https://doi.org/10.48550/arXiv.2406.09246 arXiv:2406.09246 [cs.RO] [48] Muyang Li, Yujun Lin, Zhekai Zhang, et al. 2024. SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. https://arxiv.org/abs/ 2411.05007 [49] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Chen Wang, Wei-Ming Chen, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2023. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv preprint arXiv:2306.00978 (2023). [50] Meta. [n. d.]. Llama 3. https://ai.meta.com/blog/meta-llama-3. [51] Netlib. 2026. BLAS: Basic Linear Algebra Subprograms. https://www.netlib.org/ blas/. Accessed: 2026-05-17. [52] NVIDIA. 2025. NVIDIA Isaac GR00T N1. https://developer.nvidia.com/isaac/gr00t. Official product page. [53] NVIDIA. 2026. CUDA Toolkit Documentation. https://docs.nvidia.com/cuda/. Accessed: 2026-05-17. [54] NVIDIA. 2026. NVIDIA NVLink and NVLink Switch. NVIDIA Data Center Technology Overview. https://www.nvidia.com/en-us/data-center/nvlink/ Accessed: 2026-06-17. [55] NVIDIA, Johan Bjorck, Fernando Casta neda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. 2025. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. https://doi.org/10.48550/arXiv.2503.14734 arXiv:2503.14734 [cs.RO] [56] OpenAI. 2023. ChatGPT. https://openai.com/chatgpt. Large-language-model conversational agent; accessed 31 Jul 2025. [57] OpenAI. 2023. GPT-4 Technical Report. (2023). arXiv:2303.08774 [cs.CL] https: //arxiv.org/abs/2303.08774 [58] OpenAI. 2023. Khan Academy. https://openai.com/index/khan-academy/ Announces GPT-4 powering Khanmigo as a tutor and classroom assistant. [59] Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D. Owens. 2023. Stream-K: Work-Centric Parallel Decomposition for Dense MatrixMatrix Multiplication on the GPU. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Montreal, QC, Canada) (PPoPP ’23). Association for Computing Machinery, New York, NY, USA, 429–431. https://doi.org/10.1145/3572848.3577479 [60] OWON Technology. 2026. OWON SPE Series 1 CH 100W–300W DC Power Supply. https://www.owon.com.hk/products_owon_spe_series_1_ch_100w-300w_ dc_power_supply. Product page, accessed 2026-04-03. [61] PCI-SIG. 2025. PCI Express Base Specification Revision 7.0. PCI-SIG Specification Library. https://pcisig.com/specifications Accessed: 2026-06-17. [62] William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4195–4205. https://doi.org/10.48550/arXiv.2212.09748 arXiv:2212.09748 [cs.CV]

Jun et al.

[63] Physical Intelligence. 2025. Open Sourcing 𝜋 0 . https://www.pi.website/blog/ openpi. Project blog post. [64] Physical Intelligence. 2025. openpi: Open-Source Models and Packages for Robotics. https://github.com/Physical-Intelligence/openpi. GitHub repository. [65] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. 2025. 𝜋 0.5 : A Vision-Language-Action Model with Open-World Generalization. https://doi.org/10.48550/arXiv.2504.16054 arXiv:2504.16054 [cs.LG] [66] Physical Intelligence, Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian LiBell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. 2024. 𝜋0 : A Vision-Language-Action Flow Model for General Robot Control. https: //doi.org/10.48550/arXiv.2410.24164 arXiv:2410.24164 [cs.LG] [67] PyTorch. 2025. PyTorch. https://pytorch.org/. Accessed: 2025-07-28. [68] Qualcomm. [n. d.]. A new era of possibility with on-device AI. https://www. qualcomm.com/products/technology/artificial-intelligence. [69] Qualcomm. 2026. Llama-v2-7B-Chat. https://aihub.qualcomm.com/models/ llama_v2_7b_chat?domain=Generative+AI&useCase=Text+Generation& chipsets=qualcomm-snapdragon-8gen3. Qualcomm AI Hub model card, accessed 2026-03-27. Snapdragon 8 Gen 3 Mobile Plat[70] Qualcomm Technologies, Inc. 2023. form. https://www.qualcomm.com/smartphones/products/8-series/snapdragon8-gen-3-mobile-platform. Accessed: 2026-05-17. [71] Qualcomm Technologies, Inc. 2026. Qualcomm AI Engine Direct SDK Documentation. https://docs.qualcomm.com/nav/home/QNN_general_overview.html? product=1601111740009302. Accessed: 2026-03-23. [72] Qualcomm Technologies, Inc. 2026. Qualcomm Hexagon SDK Documentation. https://docs.qualcomm.com/nav/home?product=1601111740010422. Accessed: 2026-05-17. [73] Qualcomm Technologies, Inc. 2026. Quantization — Qualcomm AI Engine Direct SDK Documentation. https://docs.qualcomm.com/bundle/publicresource/topics/ 80-63442-10/quantization.html. Accessed: 2026-03-26. [74] Qualcomm Technologies, Inc. 2026. Writing QNN HTP Op Package — Qualcomm AI Engine Direct SDK Documentation. https://docs.qualcomm.com/bundle/ publicresource/topics/80-63442-10/writing_op_package.html. Accessed: 202603-26. [75] André Rösti and Michael Franz. 2025. Unlocking the AMD Neural Processing Unit for ML Training on the Client Using Bare-Metal-Programming Tools. In 33rd IEEE Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) 2025, Fayetteville, AR, USA, May 4–7, 2025. IEEE, 271. https: //doi.org/10.1109/FCCM62733.2025.00031 [76] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML’23). JMLR.org, Article 1288, 23 pages. [77] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053 (2019). https://doi.org/10.48550/arXiv.1909.08053 arXiv:1909.08053 [cs.CL] [78] Yushan Siriwardhana, Pawani Porambage, Madhusanka Liyanage, and Mika Ylianttila. 2021. A Survey on Mobile Augmented Reality With 5G Mobile Edge Computing: Architectures, Applications, and Technical Aspects. IEEE Communications Surveys & Tutorials 23, 2 (2021), 1160–1192. https://doi.org/10.1109/ COMST.2021.3061981 [79] Zhiyi Song et al. 2018. Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Design. In AAAI Conference on Artificial Intelligence. [80] Nazish Tahir and Ramviyas Parasuraman. 2025. Edge Computing and its Application in Robotics: A Survey. arXiv preprint arXiv:2507.00523 (2025). https://doi.org/10.48550/arXiv.2507.00523 [81] Endri Taka, Andre Roesti, Joseph Melber, Pranathi Vasireddy, Kristof Denolf, and Diana Marculescu. 2025. Striking the Balance: GEMM Performance Optimization Across Generations of Ryzen AI NPUs. arXiv:2512.13282 [cs.AR] https://arxiv. org/abs/2512.13282 [82] Chengyue Wang, Wesley Pang, Xinrui Wu, Gregory Jun, Luis Romero, Endri Taka, Diana Marculescu, Tony Nowatzki, Pranathi Vasireddy, Joseph Melber, Deming Chen, and Jason Cong. 2025. Can Asymmetric Tile Buffering Be Beneficial? arXiv:2511.16041 [cs.DC] https://arxiv.org/abs/2511.16041 [83] Erwei Wang, Samuel Bayliss, Andra Bisca, Zachary Blair, Sangeeta Chowdhary, Kristof Denolf, Jeff Fifield, Brandon Freiberger, Erika Hunhoff, Phil James-Roxby, 19

Jun et al.

HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3626772.3657662 [87] Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM 52, 4 (April 2009), 65–76. https://doi.org/10.1145/1498765.1498785 [88] World Wide Web Consortium. 2026. WebGPU Specification. https://www.w3. org/TR/webgpu/. Accessed: 2026-05-17. [89] Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2025. Fast On-Device LLM Inference with NPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’25). 445–462. https: //doi.org/10.1145/3669940.3707239 [90] Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv preprint arXiv:2406.06282 (2024). https://doi.org/10.48550/arXiv.2406.06282

Jack Lo, Joseph Melber, Stephen Neuendorffer, Eddie Richter, André Rosti, Javier Setoain, Gagandeep Singh, Endri Taka, Pranathi Vasireddy, Zhewen Yu, Niansong Zhang, and Jinming Zhuang. 2026. From Loop Nests to Silicon: Mapping AI Workloads onto AMD NPUs with MLIR-AIR. ACM Trans. Reconfigurable Technol. Syst. (Jan. 2026). https://doi.org/10.1145/3785670 Just Accepted. [84] Xubin Wang, Zhiqing Tang, Jianxiong Guo, Tianhui Meng, Chenhao Wang, Tian Wang, and Weijia Jia. 2025. Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models. Comput. Surveys 57, 9 (2025), 1–39. https: //doi.org/10.1145/3724420 [85] Yichuan Wang, Zhifei Li, Shu Liu, Yongji Wu, Ziming Mao, Yilong Zhao, Xiao Yan, Zhiying Xu, Yang Zhou, Ion Stoica, Sewon Min, Matei Zaharia, and Joseph E. Gonzalez. 2025. LEANN: A Low-Storage Vector Index. https://doi.org/10.48550/ arXiv.2506.08276 arXiv:2506.08276 [cs.IR] [86] Zijie J. Wang and Duen Horng Chau. 2024. MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation. In Proceedings of

Record · ID 366223 · SHA-256 b922d036d21849df
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.