Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment Muhammad Azlan Qazi
Alexandros Iosifidis
Qi Zhang
Aarhus University Aarhus, Denmark [email protected]
Tampere University Tampere, Finland [email protected]
Aarhus University Aarhus, Denmark [email protected]
Abstract
Latency
Energy
Bandwidth
Batch Size
CCS Concepts • Computing methodologies → Cooperation and coordination; • Hardware → Hardware validation; • Networks → Network performance evaluation.
Keywords distributed inference, edge computing, Transformer, NVIDIA Jetson, adaptive inference, inter-device communication, embedded systems, performance profiling, system-on-chip (SoC) ACM Reference Format: Muhammad Azlan Qazi, Alexandros Iosifidis, and Qi Zhang. 2026. ProfilingDriven Adaptive Distributed Transformer Inference on Embedded Edge Deployment. In The 24th Annual International Conference on Mobile Systems, Applications and Services (MobiSys Workshop ’26), June 21–25, 2026, Cambridge, United Kingdom. ACM, New York, NY, USA, 6 pages. https: //doi.org/10.1145/3812836.3814999
This work is licensed under a Creative Commons Attribution 4.0 International License. MobiSys Workshop ’26, Cambridge, United Kingdom © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2712-2/2026/06 https://doi.org/10.1145/3812836.3814999
Job 4
Store MetaData
Job 2
Job 3
Job 1
Decision Making using MetaData
Distributed
Non-Distributed Terminal Device Input Embedding Positional Embedding
Edge Devices
Terminal Device
O2 FFN
Positional Embedding
InterDevice Com.
Self-Attention Partitioning FFN
Output
Terminal Device
Input Embedding
Self Attn.
O1 O2 O3
O1 FFN
Segment Mean (SM)
Self Attn.
SM
arXiv:2605.25682v1 [cs.DC] 25 May 2026
Profile
Distributing Transformer inference across embedded edge devices can alleviate individual memory and compute constraints, yet practical benefits on real hardware remain unclear: prior work relies largely on simulations that overlook hardware-specific communication overheads. We present a hardware prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi. Our key finding is that the dominant bottleneck is not just network bandwidth but also the CPU–GPU staging during communication. Because Jetson’s integrated GPU architecture lacks the PCIe/NVLink pathway that NCCL requires, all inter-device data communication should be routed through GLOO and staged in CPU memory; an overhead that scales with communication data volume and makes full-tensor exchange slower than single-device inference across the batch sizes for medium sized models such as ViT. We therefore evaluate Prism by combining Segment Means compression with lightweight offline profiling to adaptively select between local and distributed execution at runtime. Experiments show that this strategy reduces latency by 65%–77% and energy consumption by 34%–52% relative to full-tensor exchange in static distributed execution setup, demonstrating that profiling-driven adaptation is essential for practical distributed Transformer inference on embedded hardware.
O3 FFN Self Attn.
Output
Figure 1: Overview of Prism, showing offline profiling for adaptive local or distributed inference with compact Segment Means communication.
1
Introduction
Transformer models used in vision and language tasks are increasingly deployed on embedded AI systems. On-device image classification, local NLP inference, and real-time object detection all increasingly demand Transformer models that run without cloud connectivity, on hardware such as the NVIDIA Jetson Orin Nano: capable, affordable, but constrained in memory and compute. One natural strategy is to pool the resources of several such devices through distributed inference. Position-wise partitioning [3] is a promising approach: the input token sequence is split along the sequence dimension, each device processes its own partition independently, and a single collective communication operation per transformer block synchronizes intermediate features. Simulation studies show strong throughput gains compared to Tensor Parallelism (TP) [7], but translating these to real embedded hardware introduces challenges that prior work has not addressed. The GLOO staging problem. Jetson boards come with the integrated GPU, i.e. there are no PCIe-attached or NVLink-connected GPUs. The NVIDIA Collective Communications Library (NCCL) backend, which performs direct GPU-to-GPU transfers in server deployments, is therefore unavailable. The only feasible option is GLOO [5], which requires every communicated tensor to be staged through host CPU memory before transmission and copied back to GPU memory on receipt. This staging overhead scales with communication volume and dominates latency in full-tensor exchange. Batch-size sensitivity. Edge applications operating at small batch sizes for mid-sized models (such as ViT), where the fixed cost
MobiSys Workshop ’26, June 21–25, 2026, Cambridge, United Kingdom
of communication and staging may outweigh the compute savings from distributing the workload. Network variability. Network bandwidth fluctuates in practice. The benefit of distributed inference depends on whether network or staging latency dominates, which can only be determined through profiling on real hardware. We address these challenges on Prism [6], and report its hardware prototype study on a Jetson Orin Nano boards. Fig. 1 shows the distributed inference setup of Prism, where input tokens are partitioned across edge devices and synchronized using compact Segment Means intermediate features. An offline profiling phase collects performance metadata, which is queried at runtime to adaptively switch between local and distributed inference based on batch size and network conditions. While Prism [6] introduces Segment Means communication and evaluates it primarily in simulation, its behavior on real embedded hardware remains unknown. This paper therefore uses Prism as a case study to characterize how distributed Transformer inference behaves under the hardware constraints of integrated-GPU edge devices. Our contributions are: • Develop a working prototype. We develop a working prototype using Jetson Orin Nano instrumented for latency breakdown and per-sample energy measurement. Using this prototype, we quantify the CPU–GPU staging overhead imposed by the GLOO [5] communication backend and show that it renders full-tensor exchange (Voltage [3]) counterproductive at all tested batch sizes. • Profiling-driven adaptive inference policy. An offline profiling sweep across batch sizes and network conditions builds a performance map that enables runtime selection between single-device and distributed execution. We identify the batch-size crossover point below which local execution is preferred. • Bandwidth sensitivity measurements. Hardware experiments at bandwidths from 200 to 900 Mbps reveal that Prism’s compact Segment Means communication maintains low latency across all tested conditions, while Voltage degrades severely at lower bandwidth.
2
Background and Related Work
Parallelism strategies for distributed Transformer inference span a range of communication–computation trade-offs. Model parallelism [1] splits the model into sub-graphs (groups of layers) assigned to compute nodes, passing activations sequentially and causing pipeline bubbles (device idle time) that are especially costly at small batch sizes. Pipeline parallelism [4] mitigates idle time with micro-batches but provides minimal benefit for edge computation because devices still experience idle time. Tensor parallelism [7] splits individual weight matrices and requires two AllReduce operations per Transformer block; prior work found this demands at least 1000 Mbps to outperform single-device inference with six participating devices [3]. Voltage [3] uses position-wise partitioning: the input sequence is split across devices and a single AllGather per block collects intermediate features, reducing data communication per transformer block to (𝑃−1)𝑁 𝐷/𝑃, where 𝑃 is the number of data partitions
Muhammad Azlan Qazi, Alexandros Iosifidis, and Qi Zhang
or available edge devices, 𝑁 is the number of input tokens or sequences, and 𝐷 is the dimensionality of the attention features. While efficient relative to tensor parallelism, it still transmits full intermediate features, which causes increased in staging overhead on Jetson. Prism [6] extends position-wise partitioning by replacing full intermediate feature with compact Segment Means: column-wise averages of non-overlapping token segments. Communication reduces to (𝑃−1)𝐿𝐷 elements, where 𝐿 ≪ 𝑁 /𝑃 is controlled by the compression rate CR = 𝑁 /(𝐿 · 𝑃). A scaling-aware softmax reformulation further eliminates redundant Key/Value recomputation across devices, reducing per-device GFLOPs by up to 50.11% with two devices. The present paper focuses on prototype design, profiling, and hardware evaluation.
3 System Overview 3.1 Segment Means Communication Prism follows a master–worker architecture as illustrated in Fig. 1. A terminal device partitions the input X ∈ R𝑁 ×𝐷 into 𝑃 equal parts {X1, . . . , X𝑃 } along the sequence dimension. For each partition X𝑝 ∈ R𝑁𝑝 ×𝐷 , it computes 𝐿 Segment Means by dividing the partition into 𝐿 equal segments and taking the column-wise mean of each: Z𝑝 = [𝝁 0 ; 𝝁 1 ; . . . ; 𝝁 𝐿−1 ] ∈ R𝐿×𝐷 . (1) Each device receives its own partition X𝑝 and the Segment Means Z𝑝 ′ of every other device 𝑝 ′ ≠ 𝑝, constructing the augmented representation: X̂𝑝 = concat(X𝑝 , Z 𝑗 , Z𝑘 , . . .),
( 𝑗, 𝑘, . . .) ∈ I \ {𝑝}.
(2)
Query is computed from X𝑝 ; Key and Value use X̂𝑝 . A scaling-aware softmax reformulation [6] exploits the repetitive structure of Eq. (2) to further reduce redundant computations. The compression rate (CR) controls the trade-off between communication volume and accuracy. Crucially for edge deployment, it also directly controls the volume of data that must be staged through CPU memory, making it the primary tuning knob for the adaptive policy. The selected CR values {3.3, 4.95, 9.9} correspond to 𝐿 ∈ {30, 20, 10} segment means for 𝑁 =197 tokens. These represent low, medium, and high compression regimes while keeping segment sizes integer, and allow us to study how staging volume scales with communication reduction.
3.2
The GLOO Staging Bottleneck
As the Jetson Orin Nano integrates its GPU with the shared memory controller, there are no PCIe-attached or NVLink-connected GPUs. On Jetson-class integrated GPUs, NCCL falls back to host-staged TCP transfers due to the lack of PCIe/NVLink. Consequently, it provides no practical benefit over GLOO, because all tensors must be copied GPU→CPU before transmission and CPU→GPU on receipt. Thus, NCCL behaves effectively similar to GLOO in our deployment. As a result, the only practical backend is GLOO, which operates on CPU memory and introduces a two-step staging process for every inter-device tensor: (1) Device→Host: The tensor is copied from GPU memory to pinned CPU memory before transmission.
Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
MobiSys Workshop ’26, June 21–25, 2026, Cambridge, United Kingdom
Offline profiling phase (one-time) Load models (Single, Prism, Voltage)
Init GLOO backend
Sweep configurations 𝐵 ∈ 1,2,4,8,16,32 , 𝐶𝑅 ∈ 3.3, 4.95, 9.9 𝐵𝑊 ∈ {200, … , 1000}
For each (B, CR, BW) combination Warm-up (T=20)
Measure Per mode (Latency, Power, Energy)
Record to JSON
Next configuration
Performance map (Stored on terminal device)
Runtime decision Observe (B, BW)
Query map
Select best mode (Single or Distributed)
Figure 3: Jetson Orin Nano hardware prototype connected over WiFi 6E. Figure 2: Profiling flow in Prism for adaptive runtime execution. Table 1: NVIDIA Jetson Orin Nano hardware. (2) Host→Device: The received tensor is copied from CPU memory back to GPU memory.
Component
Specification
Staging latency is proportional to tensor size and largely independent of network bandwidth. For Voltage [3], which exchanges full embedding vectors per layer, this staging completely dominates the latency budget. Prism reduces the staged data volume elements, directly shrinking the staging cost by the factor (𝐶𝑅) as the communication reduction.
SoC GPU cores Memory WiFi module Comm. backend Power budget
NVIDIA Orin (Ampere GPU + ARM Cortex) 1024 CUDA cores (Ampere) 8 GB LPDDR5, unified CPU/GPU Intel AX210, WiFi 6E (up to 1 Gbps) GLOO (CPU-staged; NCCL unavailable) 7–25 W (configurable)
3.3
Profiling-Driven Adaptive Inference
A static execution mode is suboptimal because the optimal choice depends on batch size, network bandwidth, and CR; which are unknown at design time. Prism uses a light offline profiling phase (Fig. 2) executed once before deployment. The profiler sweeps: • Batch size 𝐵 ∈ {1, 2, 4, 8, 16, 32}, • Compression rate CR ∈ {3.3, 4.95, 9.9}, • Bandwidth 𝐵𝑊 ∈ {200, 300, 400, 500, 600, 700, 800, 900} Mbps (throttled via tc netem on the terminal device). For each combination it records total latency (ms), per-sample latency (ms), per-sample energy (J), and the three-way breakdown among computation, communication, and CPU–GPU I/O. These measurements are stored as a performance map on the terminal device, a lightweight JSON file requiring negligible storage. At runtime, when a batch of size 𝐵 arrives under observed network conditions, the terminal device queries the profiled map and selects the execution mode (non-distributed or distributed with the best-performing CR) that minimizes per-sample latency or energy, depending on the application objective. The one-time profiling cost is approximately |B| × |CR| × |BW| × 𝑇 inference passes, where we use 𝑇 =20 warm-up runs per configuration.
4
Prototype Setup
Hardware. Our prototype comprises NVIDIA Jetson Orin Nano development kits connected to an ASUS ROG Rapture GT-AXE16000
WiFi 6E router (Fig. 3). Table 1 summarizes the board specifications. All boards are homogeneous Jetson Orin Nano units, and experiments use a two-board configuration. Although we evaluate two devices, staging overhead grows linearly with the number of participants because each device must stage additional received tensors. Therefore, the crossover batch size where distributed execution becomes beneficial would shift to larger values as more nodes are added. Software. We use PyTorch 2.x with torch.distributed and the GLOO backend. Communication and computation latencies are profiled with the PyTorch Profiler (Fig. 7 shows flame graph produced by the tracer). Per-sample energy is recorded by Jetson Stats (jtop), which reads the on-board INA3221 power monitor. Bandwidth throttling. Network bandwidth is throttled using Linux tc netem on the terminal device to simulate conditions from 200 to 900 Mbps. Model and workload. We evaluate Prism on ViT [2] with CIFAR-10 inputs (224×224×3, 𝑁 =197 tokens). Batch size is swept from 1 to 32. All distributed experiments compare Prism (CR = 9.9 unless stated) against Voltage [3] (full-tensor exchange) and singledevice inference as a lower-bound baseline. It is worth noting that, ViT is used as a representative Transformer workload because its communication pattern mirrors that of encoder and decoder Transformer blocks used in vision, language, and multimodal foundation models.
MobiSys Workshop ’26, June 21–25, 2026, Cambridge, United Kingdom
Muhammad Azlan Qazi, Alexandros Iosifidis, and Qi Zhang
Table 2: Latency breakdown across execution modes.
Table 3: Comparison of Computation and Communication Efficiency For ViT Model
Latency (ms) Model
𝑃 𝑁
𝐿
𝐶𝑅 Batch
Strategy
𝑃
GFLOPs /dev
Comp. SU (%)
𝐶𝑅
Comm. SU CIFAR-10 (%) Acc.
No partition Voltage [3]
1 2
35.15 20.37
42.05
-
-
98.01 98.01
Prism
2 2 2
17.54 17.86 18.18
50.11 49.20 48.29
9.90 4.95 3.30
89.90 79.80 69.70
95.64 96.84 97.06
2
17.54
50.11
9.90
89.90
97.43
Comp. Other Comm. Total 1 197 – 1 197 – 1 197 – 1 197 – 1 197 – 1 197 –
– – – – – –
1 2 4 8 16 32
80.6 141.3 249.8 485.0 946.0 1864.8
2 99 2 99 2 99 2 99 2 99 2 99
10 10 10 10 10 10
9.9 9.9 9.9 9.9 9.9 9.9
1 2 4 8 16 32
123.0 26.5 140.2 29.8 179.5 34.4 272.0 52.3 494.0 86.7 936.1 182.0
2 99 2 99 2 99 Voltage 2 99 2 99 2 99
99 99 99 99 99 99
1 1 1 1 1 1
1 2 4 8 16 32
176.0 94.0 81.0 351.0 240.5 111.0 146.0 497.5 385.0 145.0 276.0 806.0 561.0 213.0 514.0 1288.0 970.0 344.0 960.5 2274.5 1454.0 533.0 1856.0 3843.0
Prism
Prism
5
0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0
80.6 141.3 249.8 485.0 946.0 1864.8
18.6 168.1 26.4 196.4 39.0 252.9 90.4 414.7 124.0 704.7 221.7 1339.8
Evaluation
We evaluate Prism on two Jetson Orin Nano boards using ViT with CIFAR-10. Section 5.1 analyses the latency breakdown and identifies the staging bottleneck. Sections 5.2 examine per-sample efficiency. Section 5.3 validates prediction correctness, and Section 5.4 describes the profiling instrumentation. Section 5.5 concludes with practical guidelines. All measurements are averaged over 20 runs.
5.1
Latency Breakdown and the Staging Bottleneck
Fig. 4a shows the latency breakdown across batch sizes at ≈ 400 Mbps, with detailed numbers in Table 2. Three key observations emerge. Staging dominates for Voltage. CPU–GPU I/O transfer (“Others” in Fig. 4a) is the dominant latency bottleneck for Voltage at all batch sizes, scaling linearly with the staged data volume. At batch 1, staging alone reaches 94 ms (Table 2), already exceeding the single-device latency of 80.6 ms, confirming that GLOO-imposed staging negates the benefits of distributed execution. Prism suppresses staging cost. At CR = 9.9, Prism exchanges only 𝐿=10 Segment Means vectors per partition per block instead of 𝑁 /𝑃=98 full token embeddings, reducing the staged tensor volume by approximately 90%. This brings the staging latency down to 26.5 ms at batch 1 (Table 2), shifting the bottleneck from communication back to computation. Adaptive crossover at batch 8. Below batch 8, even Prism’s reduced communication does not fully offset the fixed staging and synchronisation cost, so the profiling policy selects single-device execution (orange region in Fig. 4a). At batch 1, this yields 80.7 ms— a 77% reduction over Voltage’s 351 ms. From batch 8 onward, distributed Prism consistently outperforms single-device inference.
Prism (Finetuned)
Table 4: Prism vs. Voltage on Jetson Orin Nano boards (ViT, CIFAR-10, CR = 9.9, 400 Mbps). Orange rows mark single-device execution. Latency (ms)
Energy (J)
Batch Voltage 1 2 4 8 16 32
Prism
351.0 80.7 497.5 141.3 806.0 249.8 1288.0 414.7 2274.5 704.7 3843.0 1339.8
Gain Voltage Prism
Gain
77.0% 71.6% 69.0% 67.8% 69.0% 65.1%
51.8% 39.6% 36.2% 34.1% 38.8% 34.8%
1.05 1.59 2.74 5.02 9.78 17.67
0.51 0.96 1.75 3.31 5.98 11.52
Table 4 quantifies the overall gains. Prism achieves 65.1%–77.0% latency reduction and 34.1%–51.8% energy reduction over Voltage across all tested batch sizes. These gains come with a minor accuracy trade-off: as shown in Table 3, the CIFAR-10 accuracy at CR = 9.9 drops from 98.01% to 95.64%, which recovers to 97.43% after fine-tuning. Although lower compression rates (CR = 3.3) retain 97.06% accuracy without fine-tuning, it slightly compromises communication speed-up (i.e. 69.7% vs. 89.9%).
5.2
Per-Sample Latency and Energy
Fig. 4b and Fig. 4c present per-sample latency and energy across batch sizes. Latency. Voltage’s per-sample latency exceeds single-device inference at every batch size, confirming that GLOO staging overhead is never amortized. Prism combined with the profiling-driven adaptive policy, it reduces per-sample latency as batch size grows because the communication cost is spread over more samples. At batch 32, Prism achieves approximately 42 ms per sample, compared to 120 ms for Voltage and 58 ms for single-device execution. Energy. Per-sample energy closely tracks latency, since the boards operate at a fixed power mode. Prism saves 51.8% energy at batch 1 and 34.8% at batch 32 relative to Voltage (Table 4). From batch 8 onward, Prism’s per-sample energy closely approaches the single-device lower bound, indicating that overheads become smaller once communication volume is compressed. Bandwidth sensitivity. Fig. 6 sweeps network bandwidth at 𝐵=8. Voltage remains bandwidth-bound across 200–900 Mbps, while Prism crosses the single-device baseline near 340 Mbps and continues to improve above it, confirming bandwidth as a meaningful parameter of the profiling.
Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
Total Latency (ms)
3000
Compute Communication Others Prism (1 Device) Prism (2 Devices) CR=9.9 L=10 Voltage (2 Devices)
Prism (1 Device) Prism (2 Devices) CR=9.9 L=10 Voltage (2 Devices)
350
300
2500
2000
1500
1000
250
200
150
100
Prism (1 Device) Prism (2 Devices) CR=9.9 L=10 Voltage (2 Devices)
1.0
Energy per Sample (Joules)
3500
Latency per Sample (ms)
4000
MobiSys Workshop ’26, June 21–25, 2026, Cambridge, United Kingdom
0.8
0.6
0.4
0.2 50
500
0
0 1
2
Batch Size
4 Batch Size
(a)
(b)
4
8
16
32
1
2
8
16
32
0.0
1
2
4 Batch Size
8
16
32
(c)
Figure 4: Latency and energy scaling of distributed ViT inference on two Jetson Orin Nano devices. (a) Total latency breakdown. (b) Per-sample latency. (c) Per-sample energy consumption.
Figure 5: Batch-based inference comparison between Prism and Voltage on 16 CIFAR-10 images, showing predicted and ground-truth labels with per-batch latency and energy. 2250
Prism (2 Devices, CR=9.9) Voltage (2 Devices) Single Device (Baseline)
2000
Latency (ms)
1750
Prism overtakes single-device ( ≈ 340 Mbps)
1500 1250 1000
fixed costs dominate and the profiling policy falls back to singledevice execution. This matters for edge deployments where realtime applications often operate at 𝐵 ∈ [1, 4]; however, the crossover also depends on sequence length 𝑁 , so workloads with large 𝑁 (e.g., LLM inference) can benefit from distributed execution even at 𝐵=1. The decision to distribute therefore depends on both batch size and sequence length, motivating profiling-driven adaptation.
750 500
5.3 250 200
300
400
500
600
700
800
900
Network Bandwidth (Mbps)
Figure 6: Per-sample latency vs. network bandwidth at 𝐵=8 on two Jetson Orin Nano boards (ViT, CIFAR-10).
Limitations. Despite these gains, distributed Prism does not reach the theoretical ideal of half the single-device latency, residual GLOO staging and synchronization costs remain non-negligible even with compressed communication. At small batch sizes (1–4), these
Qualitative Inference Validation
Beyond efficiency metrics, we verify that Prism’s compression preserves prediction correctness. Fig. 5 shows a batch of 16 CIFAR10 images processed by both Prism and Voltage on two Jetson Orin Nano boards. Both methods produce identical predicted labels across all 16 samples. Two images are misclassified by both methods; single-device inference produces the same errors. These failures stem from visually ambiguous content (cluttered backgrounds) and are attributable to the ViT model itself, not to the Segment Means approximation or the partitioning scheme. In terms of end-to-end efficiency, Prism completes this batch in 1.10 s with 8.61 J, compared to 2.30 s and 18.50 J for Voltage. While
MobiSys Workshop ’26, June 21–25, 2026, Cambridge, United Kingdom
this represents a substantial improvement, the distributed configuration still incurs non-negligible staging overhead compared to ideal scaling, reinforcing that Prism mitigates (rather than resolves) the fundamental GLOO staging bottleneck.
5.4
Profiling Instrumentation
Communication Staging
Muhammad Azlan Qazi, Alexandros Iosifidis, and Qi Zhang
6
Conclusion
We presented a hardware prototype study of distributed Transformer inference on a set of NVIDIA Jetson Orin Nano boards connected over WiFi. Our central finding is that the GLOO backend imposes a CPU–GPU staging overhead that renders methods relying on full-tensor exchange like Voltage slower than single-device inference at every tested batch size. Prism’s Segment Means compression reduces this staging volume by up to 90%, delivering 65%– 77% latency and 34%–52% energy reductions over Voltage, while residual staging costs are not completely eliminated and remain a limiting factor at small batch sizes. A profiling-driven adaptive policy addresses this gap by querying an offline performance map at runtime to select single-device or distributed execution as appropriate. We validated these findings and confirmed that Prism’s compression preserves the model’s prediction behavior. We believe that this developed testbed, profiling methodology, and rigorous characterization of both the benefits and remaining limitations of GLOO-based distributed inference provide a practical foundation for deploying foundation models at the edge and motivate future research on improved communication backends compatible with integrated GPUs.
Acknowledgments This research was supported by the PANDORA project, funded by the European Union’s Horizon Europe Framework Programme under Grant Agreement No 101135775, and by NordForsk Nordic University Cooperation on Edge Intelligence (Grant No. 168043). Figure 7: PyTorch Profiler trace of distributed Prism inference on a Jetson Orin Nano, illustrating computation, GLOO communication, and CPU–GPU staging. Fig. 7 shows a PyTorch Profiler trace captured during distributed inference. The trace reveals three distinct phases within each Transformer block: GPU computation kernels, GLOO collective communication calls, and CPU–GPU memory staging operations. This confirms the three-way decomposition used in Table 2 and validates that the offline profiling mechanism (Section 3.3) can accurately attribute latency to each component when populating the performance map.
5.5
Summary of Profiling Insights
Our prototype experiments yield two concrete guidelines for practitioners. Always assume GLOO on Jetson; design for staging. Any distributed inference system on Jetson-class hardware must consider CPU–GPU staging bottleneck. Full-tensor exchange methods for transformer models will likely degrade performance regardless of bandwidth. While Prism reduces the staging volume by up to 90%, the residual overhead is not eliminated and remains a limiting factor at small batch sizes. Profile; do not estimate. The crossover batch size at which distributed execution becomes beneficial cannot be estimated accurately through modeling FLOPs analytically. A one-time profiling sweep of ∼200 inference passes is sufficient to populate the performance map.
References [1] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng. 2012. Large Scale Distributed Deep Networks. In Advances in Neural Information Processing Systems, F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (Eds.), Vol. 25. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/ paper/2012/file/6aca97005c68f1206823815f66102863-Paper.pdf [2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929 [cs.CV] https://arxiv.org/abs/2010.11929 [3] Chenghao Hu and Baochun Li. 2024. When the Edge Meets Transformers: Distributed Inference with Transformer Models. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). 82–92. doi:10.1109/ICDCS60910. 2024.00017 [4] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA. [5] PyTorch Contributors. 2026. Gloo: Collective Communications Library. https: //github.com/pytorch/gloo [6] Muhammad Azlan Qazi, Alexandros Iosifidis, and Qi Zhang. 2025. PRISM: Distributed Inference for Foundation Models at Edge. arXiv preprint arXiv:2507.12145 (2025). [7] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. ArXiv abs/1909.08053 (2019). https: //api.semanticscholar.org/CorpusID:202660670