Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

arXiv:2608.08730v1 [cs.LG] 9 Aug 2026

Jędrzej Maczan1 1 Independent Researcher jedrzej@maczan.pl

Abstract: Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet the overhead of WebGPU per-operation dispatch remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. Therefore, we conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference. Keywords: inference,llm,webgpu,edge,browser

1. Introduction Large Language Models (LLMs) are increasingly deployed outside the datacenters, like in web browsers [2] and on the edge [3, 4]. WebGPU has emerged as the cross-platform standard that makes this possible by exposing a common interface that supports multiple hardware. The performance engineering for these systems has so far followed familiar paths, like quantization [5–7], kernel optimization [8–10], and memory efficiency [11]. To our knowledge, however, no study characterizes the cost of WebGPU operation dispatch in the context of machine learning inference. WebGPU-based inference is hard in ways native execution is not. WebGPU code, like any code running in a browser, must meet high security standards that clash with the need for raw performance: every operation is validated before execution [12]. The execution environment is sandboxed and cannot be controlled by the inference engine, because it inherits whatever limits the browser imposes. And because the engine cannot reason about the target hardware, kernel (WGSL shader [13]) hyperparameters must be chosen from a “safe” set of general heuristics rather than tuned per device, as in LlamaWeb [1]. Factors like these give rise to performance challenges distinct from native inference, and require new ways of measuring performance. We find that the wall-clock measurements of operation performance overestimate per-dispatch cost by ∼20x, by conflating dispatch with synchronization. Each operation, when measured in isolation, carries a significant GPU-CPU synchronization cost that real autoregressive decoding incurs only once per token, not once per operation. Concretely, we introduce1 a sequential-dispatch measurement method: N operations are dispatched together and synchronized only once at the end, which isolates the true cost of a single dispatch. On Dawn [15], the 497 µs single-operation measurement is dominated by synchronization 1 An earlier, preliminary version of this work, appears in [14].

1

UK AI Conference 2026

Per-dispatch cost (µs, log scale)

104

103

Naive single-operation measurements overestimate the per-dispatch cost Naive single-operation Sequential (true per-dispatch)

63×

21× 8×

102 24

101

Dawn (RTX 5090)

1×

36

wgpu (RTX 5090)

1×

32

24

wgpu (AMD iGPU)

Safari (Apple M2)

1×

48

wgpu (Apple M2)

33

Chrome (RTX 5090)

Figure 1. Naive single-operation measurement overestimates per-dispatch cost by conflating dispatch with synchronization. The figure compares the single-operation measurement (one dispatch end to end, including a full GPU-CPU synchronization) with our sequential measurement (Section 4) which amortizes that one-time synchronization away. The overestimation factor is annotated above each naive bar. The effect is implementation-specific. Firefox is omitted, because we identify it to be rate-limited, as confirmed by [1].

(450 µs), with 24–36 µs dispatch cost on Vulkan and 32–71 µs on Metal. These times are identical for float32 and float16. Knowing how to measure dispatch cost, we ask how much does dispatch actually affect performance? We test this in the area of kernel fusion, which [1] explicitly leaves for future work. We find that fused kernels, using the same WGSL shaders and saving a negligible amount of memory traffic, improve throughput by 53% while cutting dispatches from 876 to 564. That fusion improves throughput is well known [16, 17]. Our contribution is the controlled isolation of dispatch count as the cause, and ruling out the kernels (shaders) quality and memory-bandwidth explanations that fusion is usually credited with. At batch size 1, the effective approach to LLM inference optimization in WebGPU is to reduce dispatch count, not to optimize kernels [18, 19]. Contributions. • A sequential-dispatch measurement method that isolates true per-dispatch cost. Using our method we show that naive single-operation measurements overestimate it by ∼20x, by conflating dispatch with synchronization. • Measurement of per-dispatch cost - 24–36 µs in Vulkan, 32–71 µs in Metal, independent of data type. • We identify the dispatch count as the main performance bottleneck for WebGPU LLM inference at batch size 1. We rule out the compute and memory-traffic explanations.

2. Related Work Browser-based LLM inference. The adoption of WebGPU and the rise of LLMs has produced several inference engines targeting browsers, including WebLLM [2], Transformers.js [20], and most recently LlamaWeb. The engines focus on quantization, kernels and their tradeoff between performance and portability, and memory efficiency. [1] reports state-of-the-art decode throughput and provides an extensive study on multiple kinds of hardware. They explicitly leave kernel fusion to reduce dispatch overhead as future work, which is the gap we address in our experiment. WeInfer [21] targets the synchronization stage by postponing GPU-CPU read across tokens, while our work characterizes and reduces the dispatch overhead within each forward pass. The synchronization WeInfer optimizes is the one our sequential method amortizes. Thus, it makes our two studies complementary. Unlike WeInfer, which reports the aggregated speedups, we isolate dispatch count as the bottleneck through a controlled experiment. 2

Jędrzej Maczan

WebGPU dispatch model: naive timing folds synchronization into dispatch cost Encode operation

Bind resources

Submit to queue

GPU executes

GPU CPU sync

Naive single-op timing measures dispatch + one full synchronization

overestimates ~20×

Sequential method N dispatches, sync once at end

isolates true per-dispatch cost

Per-dispatch cost

24 36 µs (Vulkan) · 32 71 µs (Metal)

Figure 2. The WebGPU dispatch model and the two measurement methods. In WebGPU, each operation is encoded, has its resources bound, and is submitted to the queue, after which the GPU executes and a GPU-CPU synchronization returns the results. Naive single-operation timing spans one encode-through-synchronize round trip and folds the one-time synchronization into the per-dispatch measurement, overestimating the results by up to ∼20×. Our sequential method issues N dispatches and synchronizes only once at the end, isolating the true marginal cost of a single dispatch.

Dispatch and launch overhead. On native APIs, kernel launch overhead is well studied: for example a latency of CUDA kernels launch is a few microseconds and can be amortized further with CUDA Graphs [22]. Vulkan’s design [23] tries to reduce the CPU overhead relative to OpenGL. A recent work quantifies its impact on LLM inference. [24] characterize kernel launch overhead on GPU-CPU line and find that systems remain CPU-bound at low batch sizes. [19] reports that launch overhead alone can account for 14.6% of end-to-end decode latency in production serving. We provide the analogous characterization for WebGPU, where per-operation validation and command submission shows dispatch cost of tens of microseconds per dispatch rather than single-digit, and dominant at batch size 1. Kernel fusion and megakernels. In deep learning compilers and inference engines, operation/kernel fusion is a standard technique for reducing memory traffic and launch overhead [8, 16, 17, 25]. Pushed to its limit, the megakernels line of work [26] fuses the entire computation pass into a single kernel to eliminate launch overhead [18, 19, 27]. [24] shows the potential of this method to mitigate the bottleneck in LLM inference specifically. In our study, we use fusion as an instrument in isolating dispatch count as the cause of the observed throughput gains.

3. Background Cross-platform GPU standards have a long lineage in graphics - OpenGL, OpenCL, and, for the browser, WebGL. WebGPU is its modern successor and provides a common interface for the major native APIs (Vulkan, Metal, and Direct3D). As of 2026 it is supported across all major browsers (Chrome, Safari, Firefox) and also provided as standalone implementations that can run natively - Dawn [15] and wgpu [28] - with bindings to low-level programming languages used for high performance, like C++ and Rust. Another browser standard, WebNN [29], provides a graph-based abstraction in which the application submits a network graph and the browser lowers it to an operating system-level runtime that handles the scheduling and kernel selection. This graph-submission model amortizes per-operation dispatch in a way WebGPU API does not. WebGPU follows a deferred command-buffer model. Operations are encoded into command buffers, connected using bind groups, submitted to a queue, and synchronized with explicit waits or buffer mappings. Each dispatch adds an overhead from encoder creation, bind group setup, and queue submission. Autoregressive 3

UK AI Conference 2026

Cutting dispatch count raises throughput, even when computation stays the same 19.4

800 600

+53% throughput

636 13.5

20.5

20.6

588

564

25 20

Throughput (tok/s)

Dispatches per forward pass

876

15

400

10

200

5

0

No fusion

+RMSNorm

+MLP gate/up/SiLU

+K+V projection

0

Figure 3. Our controlled fusion experiment isolates dispatch count and identifies it as the performance lever. Each fusion stage reduces the dispatch count per forward pass, while raising throughput for a total +53% from 876 to 564 dispatches. Fusion changes only the “packaging” of the computation into dispatches, without changing what gets computed and how efficiently (the underlying operations are not re-tiled or otherwise optimized). We classify the eliminated intermediate memory traffic as negligible (∼1.8 MB, <1 µs). The dispatch count reduction accounts for the throughput improvement, isolating per-operation dispatch overhead, rather than kernel compute quality, as the bottleneck at batch size 1.

LLM decoding at batch size 1 is built from hundreds of such small dispatches per token, with a required GPU-CPU synchronization for token selection. In such setup, per-dispatch overhead is a significant factor of LLM inference performance.

4. Method We make all the measurements using torch-webgpu [30], an open-source out-of-tree PrivateUse1-based [31] PyTorch [32] backend and compiler for WebGPU we built. It compiles models to WGSL shaders and executes them with Dawn. Its simplicity makes per-dispatch measurements and fusion experiment straightforward and reproducible by the research community.

4.1. Sequential-dispatch measurement The naive way to measure dispatch cost is to time a single operation end to end, from encode, through resources binding, submit, actual computation to a synchronization. This measurement conflates two distinct costs of the dispatch itself and a full GPU-CPU synchronization, and attributes both to the dispatch. In real decoding, however, synchronization happens only once per token. Our sequential-dispatch method mirrors this behavior. We submit N identical dispatches with a single synchronization at the end and then divide the total time by N . The one-time synchronization is amortized and leaves the true marginal cost of one dispatch. We apply this across four GPU vendors (NVIDIA, AMD, Apple, Intel) and multiple WebGPU implementations (Dawn, wgpu-native), on Vulkan and Metal backends.

4.2. Measuring the performance impact of dispatch To determine whether dispatch overhead rather than kernel quality limits throughput at batch size 1, we run a controlled kernel fusion experiment. We fuse groups of operations so that the number of dispatches in a forward pass drops from 876 to 564, while keeping the underlying computation unchanged. Because the WGSL shaders are essentially unchanged, any throughput difference cannot be attributed to improved kernel quality. Similarly, since the fusion eliminates only a small amount of intermediate memory traffic (about 1.8 MB, under 1 µs), it cannot be attributed to reduced memory traffic as well. The only variable 4

Jędrzej Maczan

Per-dispatch Timing Breakdown Submit

12.9 µs (40%)

Encoder create

6.4 µs (20%)

Encoder finish

6.1 µs (19%)

Pass begin

Primary bottleneck: Command buffer submission

3.2 µs (10%)

Set pipeline

1.4 µs (4%)

Set bind group

1.0 µs (3%)

Pass end

0.7 µs (2%)

Dispatch call

0.6 µs (2%)

GPU sync time 0.5 µs (2%) 0.0

2.5

5.0

7.5 10.0 12.5

Per-dispatch overhead (µs)

Figure 4. Per-dispatch overhead breakdown. Command buffer submission (Submit) is the dominant source of overhead - taking about 40% of total CPU time - 12.9 µs per dispatch. Encoder creation and completion contribute an additional 12.5 µs combined, while pipeline setup, bind-group binding, and dispatch invocation are a comparatively small costs (<1.5 µs each). We identify command buffer submission as the primary bottleneck. This result supports our claim that reducing dispatch count is more impactful than further optimizing pipeline or binding operations.

that changes meaningfully is the number of dispatches. We report this experiment on our reference setup (NVIDIA RTX 5090 32 GB, Dawn, Vulkan, float32).

5. Results 5.1. True per-dispatch cost is far below naive estimates Figure 1 compares single-operation and sequential measurements. Single-operation measurements overestimate per-dispatch cost by roughly 20x on native Dawn and more on Chrome and Safari, since it folds in a 450 µs synchronization cost that decoding pays only once per token. The sequentially measured cost is 24–36 µs on Vulkan and 32–71 µs on Metal, and is consistent across four GPU vendors on the same backend. Because a dispatch cost reflects command encoding and submission rather than the arithmetic the kernel performs, it is independent of data type (identical for float32 and float16), and we attribute it to the WebGPU API.

5.2. Dispatch count, not kernel quality, is the bottleneck Figure 3 shows the controlled fusion experiment results. A reduction of dispatches from 876 to 564, while keeping computationally the same WGSL shaders and with negligible memory savings, improves throughput by 53% (Time to First Token (TTFT) drops from 71.4 ms to 41.6 ms). Since kernel quality and memory traffic are both held constant by construction, we attribute the 312 fewer dispatches as the cause of the improvement. This isolates per-operation dispatch overhead, not kernel quality, as the bottleneck at batch size 1. This result is consistent with the broader observation that highly optimized individual kernels do not proportionally improve end-to-end throughput at batch size 1 - when per-operation overhead dominates, kernel speed is not the binding constraint.

5.3. Command buffer submission dominates the dispatch To understand the characteristics of dispatch itself, we measured different steps of WGSL shaders dispatch. In figure 4 we present the breakdown of averaged measurements of operations specific to WebGPU during 5

UK AI Conference 2026

dispatch. We find it informative in context of future decisions regarding WebGPU API design and its implementations.

6. Discussion The conventional performance optimization levers, such as writing better kernels, quantization, improving the memory layout, are not sufficient in case of inference using WebGPU. The binding constraint is the number of dispatches and the per-operation overhead added by each of them. The practical implication is that fusion, which reduces dispatch count, even if without any other additional computation improvements, is among the high value WebGPU-specific optimizations available. With this finding, we fill the missing kernel fusion research impact for WebGPU, left for the future work by [1]. Because per-dispatch cost is a property of the WebGPU API rather than of any particular model or shader, the deeper improvements might require changes in WebGPU specification itself. Dispatch amortization, mechanisms analogous to command-graph capture and replay (as in CUDA Graphs), would help avoid re-validating and re-submitting every operation. We recognize that such changes are non-trivial and have to be weighed against the browser security model, that motivates per-operation validation in the first place. Scope. We conducted the causal fusion experiment on a single reference platform (RTX 5090, Dawn, Vulkan) at batch size 1 with float32. The per-dispatch measurements were conducted using hardware from four vendors - NVIDIA, AMD, Intel, Apple - using two WebGPU backends - Dawn, wgpu - and is dtype-independent - we got the same results for both float32 and float16. The fusion result should be read as an evidence on this particular platform and regime rather than a universal claim. Batch sizes greater than one, where kernel compute begins to dominate, are left for the future work.

7. Conclusion We characterize WebGPU per-dispatch overhead for LLM inference. Our sequential-dispatch measurement method reveals that naive single-operation benchmarks overestimate dispatch cost by ∼20x, and measures a true per-dispatch cost, independent of data type - 24–36 µs on Vulkan and 32–71 µs on Metal. A controlled fusion experiment isolates dispatch count as the cause of a 53% throughput gain. We recognize the peroperation dispatch overhead as the major performance bottleneck in LLM inference on WebGPU at batch size 1. Our results point to dispatch amortization, both in the implementation of inference engines, as well as in the WebGPU specification, as a path to more performant inference.

References [1] R. Levine, R. Sharma, N. Jain, A. Ramesh, Z. Chen, N. Abbas, J. Contini, and T. Sorensen, “Llamas on the web: Memory-efficient, performance-portable, and multi-precision llm inference with webgpu,” 2026. [2] MLC-AI, “WebLLM: High-performance in-browser LLM inference engine.” https://github.com/ mlc-ai/web-llm, 2024. [3] G. Cai, R. Tian, L. Yang, Y. Jia, L. Li, and J. Wang, “Efficient inference for edge large language models: A survey,” Tsinghua Science and Technology, vol. 31, no. 3, pp. 1365–1380, 2026. [4] S. Kodavanti, S. Vajrala, S. Miriyala, U. Tiwari, U. Kumar, U. M. Kumar, A. P. Singh, A. D, N. Mutyala, V. N. Rajendiran, S. K. Allur, E. Lee, D. Kim, H. Lee, G. Cho, and J. Kim, “Unlocking the edge deployment and ondevice acceleration of multi-lora enabled one-for-all foundational llm,” 2026. [5] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in International Conference on Learning Representations (ICLR), 2023. 6

Jędrzej Maczan

[6] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” in Proceedings of Machine Learning and Systems (MLSys), 2024. [7] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022. [8] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022. [9] M. Hidaka, “webgpu-blas: WebGPU-accelerated BLAS operations.” https://github.com/milhidaka/ webgpu-blas, 2024. [10] Z. Nuss, “Optimizing a WebGPU matmul kernel for 1TFLOP+ performance.” https://www. nuss-and-bolts.com/p/optimizing-a-webgpu-matmul-kernel, 2024. [11] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023. [12] W3C GPU for the Web Working Group, “WebGPU specification.” W3C Working Draft, World Wide Web Consortium. https://www.w3.org/TR/webgpu/, 2024. [13] W3C GPU for the Web Working Group, “WebGPU shading language (WGSL) specification.” W3C Working Draft, World Wide Web Consortium. https://www.w3.org/TR/WGSL/, 2024. [14] J. Maczan, “Characterizing webgpu dispatch overhead for llm inference across four gpu vendors, three backends, and three browsers,” 2026. [15] Google, “Dawn: WebGPU implementation for Chromium.” https://dawn.googlesource.com/dawn, 2024. [16] D. Snider and R. Liang, “Operator fusion in XLA: Analysis and evaluation,” 2023. [17] Y. Shi, Z. Yang, J. Xue, L. Ma, Y. Xia, Z. Miao, Y. Guo, F. Yang, and L. Zhou, “Welder: Scheduling deep learning memory access via tile-graph,” in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), (Boston, MA), pp. 701–718, USENIX Association, 2023. [18] X. Cheng, Z. Zhang, Y. Zhou, J. Ji, J. Jiang, Z. Zhao, Z. Xiao, Z. Ye, Y. Huang, R. Lai, H. Jin, B. Hou, M. Wu, Y. Dong, A. Yip, S. Wang, W. Yang, X. Miao, T. Chen, and Z. Jia, “Mpk: A compiler and runtime for mega-kernelizing tensor programs,” 2025. [19] W. Dong, M. Hu, G. Yu, Q. Fu, P. Xu, H. Xu, Y. Xing, X. Jiao, S. Li, and L. Liu, “Ada-MK: Adaptive megakernel optimization via automated DAG-based search for LLM inference,” 2026. [20] Hugging Face, “Transformers.js: State-of-the-art machine learning for the web.” https://huggingface. co/docs/transformers.js, 2024. [21] Z. Chen, Y. Ma, H. Shen, and M. Liu, “WeInfer: Unleashing the power of WebGPU on LLM inference in web browsers,” in Proceedings of the ACM Web Conference (WWW), 2025. [22] NVIDIA, “CUDA graphs.” https://developer.nvidia.com/blog/cuda-graphs/, 2024. [23] Khronos Group, “Vulkan 1.3 specification.” https://registry.khronos.org/vulkan/specs/1.3/ html/, 2024. [24] P. Vellaisamy, T. Labonte, S. Chakraborty, M. Turner, S. Sury, and J. P. Shen, “Characterizing and optimizing LLM inference workloads on CPU-GPU coupled architectures,” 2025. 7

UK AI Conference 2026

[25] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “TVM: An automated end-to-end optimizing compiler for deep learning,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018. [26] H. Jin, B. Hou, G. Wang, R. Lai, J. Chen, Z. Ye, Y. Cai, Y. Dong, X. Cheng, Z. Zhang, Y. Zhao, Y. Huang, L. Yang, J. Jiang, G. Oliaro, J. Ji, X. Miao, V. Grover, T. C. Mowry, J. Zhihao, and T. Chen, “Event tensor: A unified abstraction for compiling dynamic megakernel,” 2026. [27] B. Spector, J. Juravsky, S. Sul, O. Dugan, D. Lim, D. Fu, S. Arora, and C. Ré, “Look Ma, no bubbles! Designing a low-latency megakernel for Llama-1B.” https://hazyresearch.stanford.edu/ blog/2025-05-27-no-bubbles, May 2025. Hazy Research blog, Stanford University. [28] gfx-rs Community, “wgpu: Safe and portable GPU abstraction in Rust.” https://wgpu.rs/, 2024. [29] W3C Web Machine Learning Working Group, “Web neural network API,” w3c candidate recommendation draft, World Wide Web Consortium (W3C), Jan. 2026. [30] J. Maczan, “torch-webgpu - PyTorch compiler and WebGPU runtime,” Oct. 2025. [31] PyTorch Contributors, “Facilitating new backend integration by PrivateUse1.” PyTorch Tutorials, 2023. Created Oct 3, 2023; last updated May 14, 2026; accessed June 11, 2026. [32] J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, et al., “PyTorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024.

8

Related documents

• Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU#216801
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference#189269
• SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference#238580
• LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling#155264
• HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing#120464