1
Communication-Efficient Collaborative LLM Inference over LEO Satellite Networks
arXiv:2604.04654v1 [cs.DC] 6 Apr 2026
Songge Zhang, Graduate Student Member, IEEE, Wen Wu, Senior Member, IEEE, Liang Li, Member, IEEE, Ye Wang, Member, IEEE, and Xuemin (Sherman) Shen, Fellow, IEEE
Abstract—Low Earth orbit (LEO) satellites play an essential role in intelligent Earth observation by leveraging artificial intelligence models. However, limited onboard memory and excessive inference delay prevent the practical deployment of large language models (LLMs) on a single satellite. In this paper, we propose a communication-efficient collaborative LLM inference scheme for LEO satellite networks. Specifically, the entire LLM is split into multiple sub-models, with each deployed on a satellite, thereby enabling collaborative LLM inference via exchanging intermediate activations between satellites. The proposed scheme also leverages the pipeline parallelism mechanism that overlaps sub-model inference with intermediate activation transmission, thereby reducing LLM inference delay. An adaptive activation compression scheme is designed to mitigate cumulative errors from multi-stage model splitting while preserving inference accuracy. Furthermore, we formulate the LLM inference delay minimization problem by jointly optimizing model splitting and compression ratios under onboard memory and inference accuracy constraints. The problem is transformed into a shortest-path search problem over a directed acyclic graph that edge weights explicitly quantify the inference delay induced by model splitting and compression strategies, which is solved via a modified A⋆ based search algorithm. Extensive simulation results indicate that the proposed solution can reduce inference delay by up to 42% and communication overhead by up to 71% compared to stateof-the-art benchmarks, while maintaining the inference accuracy loss of less than 1%. Index Terms—Collaborative inference, model spliting, activation compression.
I. I NTRODUCTION Low Earth orbit (LEO) satellites have become an essential network infrastructure for the sixth generation (6G) networks, providing communication, sensing, and computing services [1], [2]. An increasing number of LEO satellites have been launched into space to provide ubiquitous connectivity. For example, Starlink has launched over 4,000 satellites, and other mega-constellations, such as OneWeb and Telesat, are also rapidly expanding [3]. Recent advances in satellite hardware have endowed LEO satellites with substantial on-board computing capabilities, e.g., graphics processing units (GPUs), enabling on-orbit deployment of artificial intelligence (AI) Songge Zhang is with the School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, Shenzhen, 518055, China, and also with the Frontier Research Center, Pengcheng Laboratory, Shenzhen, 518055, China (email: [email protected]); Wen Wu, Liang Li, and Ye Wang are with the Pengcheng Laboratory, Shenzhen, 518055, China (email: {wuw02, lil03, wangy02}@pcl.ac.cn). Xuemin (Sherman) Shen is with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, N2L 3G1, Canada (email: [email protected]).
models. The onboard computing facilitates real-time data processing and rapid responses to downstream tasks, such as Earth observation, disaster response, navigation, and environmental monitoring [4]. The on-orbit paradigm shift significantly reduces communication overhead by processing raw sensing data on board, while transmitting only the computed results to ground stations [5]. Deploying large language models (LLMs), such as vision transformers (ViT), on satellites offers good adaptability to the downstream tasks [6]. LLM inference on LEO satellites is constrained by limited onboard memory and computational resources, since LLMs typically contain billions of parameters [7]–[9]. The massive footprint may exceed onboard memory capacity and render deployment infeasible, or incur long inference delay when deployable. Researchers have developed model compression techniques, including structured weight sparsification [10], pruning [11], and low-bit quantization [12]. These methods substantially reduce model size, enabling execution within the tight resource envelope of satellites. However, such parameter reduction methods may inevitably lead to performance degradation, potentially compromising the inference accuracy. Model splitting is a promising technique for deploying LLM sub-models across multiple satellites to address the limitations of model compression [13]. This strategy alleviates the resource constraints on individual satellites and enables collaborative inference. Nevertheless, existing studies typically adopt a single-split scheme, which is insufficient for satellite networks requiring distributed satellite execution. In multi-split scenarios, the heterogeneity of satellite computing resources can result in significant idle time and waiting time, increasing the overall delay of the serial inference [14], [15]. Moreover, inter-satellite transmissions become more frequent, resulting in increased communication overhead. Therefore, it is essential to explore a collaborative and communication-efficient scheme in heterogeneous satellite environments. In this paper, we first propose a collaborative inference scheme across multiple LEO satellites to accelerate the LLM inference. Specifically, we split an LLM into multiple submodels and deploy them across different satellites, thereby meeting the memory constraints of individual satellites, as illustrated in Fig. 1. The multiple satellites are executed collaboratively, with intermediate activation sequentially transmitted via inter-satellite links to improve inference efficiency. We reduce the collaborative inference delay through pipeline parallelism, which allows each satellite to overlap the activation communication with the computation of its sub-model. Second, an adaptive compression scheme is proposed to reduce
2 Add & norm
TABLE I: List of Important Notations
l1 ×
Transformer block
l2 ×
Transformer block
l3 ×
Transformer block
lK ×
Transformer block Classification
Embedding Input
S0 Sensor Image
S1
S2
Immediate Activation
S3
SK
Ground Station
Inference Result
Fig. 1: An overview of the collaborative LLM inference scheme in satellite networks. inter-satellite communication overhead incurred by transmitting intermediate activations. A learnable Gumbel-mask module is trained end-to-end with a straight-through estimator and sparsity regularization to select informative activations, producing near-binary masks that sparsify the transmitted activations. The selected features are then quantized into lowbit formats and further compressed through entropy-based coding. Furthermore, we formulate a minimization problem to optimize the LLM splitting strategy and compression ratios subject to constraints on onboard memory and inference accuracy. The objective is the total inference delay, and the formulation is a mixed integer nonlinear programming problem. To solve it, the minimization problem is reformulated as a shortest-path search over a directed acyclic graph (DAG), where each path represents a valid layer assignment to satellites, and the path cost corresponds to the total inference delay. An iterative A⋆ based algorithm is then designed, where the inner loop solves for the optimal compression ratios, and the outer loop searches for the optimal LLM splitting strategy. Experimental results demonstrate that the proposed scheme reduces inference delay by up to 42%, and communication overhead by up to 71.7% compared to state-of-the-art benchmarks, while maintaining inference accuracy loss within 1%. The main contributions of this paper are summarized as follows: 1) We propose a collaborative LLM inference scheme across multiple LEO satellites through model splitting and pipeline parallelism; 2) We propose an adaptive and learnable compression scheme, which can reduce inter-satellite communication overhead while preserving inference accuracy; 3) We formulate a delay minimization problem to jointly optimize the model splitting strategy and activation compression ratios; 4) We design a directed acyclic graph-based solution and propose a modified A⋆ algorithm to obtain the optimal decision variables through an outer–inner search. The remainder of this paper is organized as follows. Section II reviews related work. Section III presents the proposed collaborative LLM inference scheme and activation compression scheme. Section IV formulates the end-to-end inference delay minimization problem. Section V describes the proposed solution. Section VI presents the experimental results, and
Feed forward
Notation Add & norm L K Multi-head attention lk fk Ck (lk ) comp Tk qk Sk Tkcomm Tkeff Ttotal θ Mk (lk ) Mkmax Acck (qk ) Accmin T t yK S D P E rsat rgs
Description Total number of layers in the LLM Number of computing satellites P Layers assigned to satellite k, K k=1 lk = L Computational capacity of satellite k (FLOPs/s) FLOPs required at satellite k with lk layers Computation delay at satellite k Compression ratio at satellite k Feature size at output of satellite k Communication delay from satellite k to k+1 Steady-state delay at satellite k considering overlap Total inference delay Auxiliary variable Memory usage at satellite k under lk layers Maximum memory capacity of satellite k Accuracy at satellite k under compression qk Minimum required inference accuracy Number of input samples Predicted result for input dt Sequence length of token Hidden dimension of transformer features Number of pixels per image Number of output classes Inter-satellite data rate Ground-to-satellite data rate
Section VII concludes the paper. Table I summarizes the main symbols. II. R ELATED W ORK To meet the growing demands for low-latency Earth observation and autonomous decision-making in space, on-orbit computing has emerged as a promising paradigm that shifts AI inference from ground stations to satellites themselves. Typically, satellite systems primarily acted as passive data collectors, transmitting raw data to the ground station for processing [16]. However, the ground-based processing paradigm suffers from severe downlink bottlenecks, especially when dealing with high-resolution imagery or continuous video streams. To alleviate the communication burden, recent efforts have focused on integrating lightweight onboard filtering techniques that prioritize or discard data based on relevance. For instance, Giuffrida et al. demonstrated that edge-deployed convolutional neural networks (CNNs) could effectively identify low-utility images in real time [17]. Similarly, Cratere et al implemented FPGA-accelerated CNNs onboard satellites to filter irrelevant image content before transmission [18]. These approaches ensured that only mission-critical data is sent to the ground, thereby significantly improving bandwidth utilization. The advancement of AI-optimized embedded hardware, including central processing units (CPUs), GPUs, and neural processing units (NPUs), has enabled the direct deployment of inference models onboard satellites. These hardware systems support real-time deep neural network inference without requiring a ground server. Nonetheless, deploying a large AI model in orbit remains challenging due to limitations in energy and memory. To address these constraints, researchers have developed model compression techniques, including structured weight sparsification [10], pruning [11], and low-bit quantization [12], which substantially reduce model size, enabling onboard execution within the resource-limited
3
satellites. Zhang et al. proposed an object knowledge distilled joint detection and tracking framework, to perform object detection in onboard satellite videos [19]. Sun et al. proposed a pruning method that integrates graph topology information with model pruning to optimize the graph neural network structure and communication overhead in distributed satellite communications [20]. Since model compression inevitably reduces parameter capacity and may degrade performance, collaborative inference across multiple nodes emerges as a promising solution to support LLM inference on onboard satellites without compromising accuracy. Collaborative inference has emerged as an effective paradigm for accelerating AI inference tasks by leveraging the computational resources of multiple distributed nodes [21]. Existing collaborative inference methods can be broadly categorized into data partitioning and model splitting. Among these, model splitting has gained significant attention due to its ability to decouple the AI model and distribute sub-models across different nodes. In the most common two-node setting, the model is divided into two parts: the front-end layers are deployed on the user-side device (e.g., satellite or edge terminal), while the remaining layers reside on the server-side or ground station [22], [23]. The split paradigm allows for early feature extraction on resource-constrained nodes, followed by highcapacity processing downstream. Sun et al. proposed a twopart model splitting scheme, which dynamically adjusts the splitting point and allocates resources in real time to meet userspecific AI inference tasks [24]. Li et al. introduced a delayaware DNN inference framework that improves throughput in mobile edge computing environments by jointly optimizing model splitting and parallel multi-thread execution [25]. In more complex multi-node scenarios, the model can be divided into multiple parts, each assigned to a different node in the system [26]. Cao et al. proposed a multi-layer partitioned multi-access edge computing system, which aims to meet users’ computation latency requirements through the joint optimization of communication and computing resource allocation [27]. Xie et al. investigated the design of multilayer edge computing architectures and the co-scheduling of heterogeneous edge computing resources [28]. To further accelerate multi-node collaboration, Narayanan et al. introduced a pipeline mechanism that enables model splitting and parallel processing across multiple GPUs, thereby improving overall training throughput and reducing per-machine latency [29]. Wang et al. considered the pipeline mechanism in wireless networks and formulated the model multi-splitting problem to optimize distributed algorithms, thereby reducing communication overhead and improving overall inference efficiency [30]. However, model splitting introduces additional communication overhead, which becomes more pronounced in multi-segment scenarios. To address this, some studies have explored compression mechanisms to reduce the transmission of intermediate activations [31]. For example, Zheng et al. proposed a randomized Top-E sparsification method that selects the top E activation elements for transmission [32]. These works enable the deployment of LLMs by splitting the model into multiple segments across distributed nodes and improve communication efficiency by compressing intermediate activations exchanged
S1
a a
S2
Wait
b
b
Wait
c
S3
c d
SK S1 S2 S3 SK
Idle
d
a a b b c c d d
Delay Reduction
Fig. 2: Parallel computing process for model splitting. between model segments. Different from the existing works, the proposed scheme considers the collaborative inference pipeline parallelism for LEO satellite networks, and an adaptive compression strategy is tailored for multi-part partitioning scenarios, which effectively reduces communication overhead while preserving model inference accuracy. In addition, we formulate a joint optimization problem of layer partitioning and compression ratio, and develop a graph-based algorithm to efficiently solve it. III. P ROPOSED C OLLABORATIVE LLM I NFERENCE S CHEME A. System Model In this paper, a distributed LEO satellite network consisting of heterogeneous satellites is considered, in which on-orbit LLM inference for intelligent Earth observation tasks is collaboratively performed. As illustrated in Fig. 1, the network comprises a sensing satellite S0 equipped with optical sensors, and a set of computing satellites SC = {S1 , S2 , . . . , SK } interconnected via inter-satellite links. The sensing satellite S0 is responsible for capturing high-resolution remote sensing images and initiating the LLM inference process. These raw images are then delivered to the computing satellite chain for distributed processing. Each computing satellite Sk ∈ SC is equipped with onboard processors and memory resources for handling a sub-model of the LLM. Due to the limited computational capacity and memory of individual satellites, deploying an entire LLM on a single satellite for inference is infeasible. To address this, we adopt a collaborative LLM inference scheme in which the model is partitioned into multiple components and deployed across multiple satellites, with intermediate activations exchanged to enable end-to-end inference. However, the frequent intersatellite transmission of activations incurs substantial communication overhead. Therefore, we propose an activation compression scheme, which is detailed in the following two subsections. B. Collaborative LLM Inference Scheme A collaborative LLM inference scheme for satellite networks is proposed, consisting of model splitting, collaborative inference, and pipeline parallelism, as detailed below.
4
1) Model Splitting: In the proposed scheme, when a satellite is assigned a heavy computation workload or constrained memory resources, the target LLM can be split into multiple sub-models and deployed across several satellites. As illustrated in Fig. 1, the model is divided into K sub-models and each sub-model is assigned to a computing satellite in a chain of L satellites, where satellite Sk is allocated lk LLM layers. Due to the inherently sequential nature of collaborative inference, unbalanced layer assignments can lead to idle and waiting times. As illustrated in Fig. 2, consider two batches (yellow and green) with K = 4 satellites (S1 –SK ) hosting submodels a–d. In the first case, the total time for the yellow batch to execute a, transfer to b, and complete b exceeds the time for the green batch to finish a and transfer to b, causing the green batch to wait on S2 , which means that S1 is underloaded while S2 is overloaded. In the second case, computing c and transferring to d for the yellow batch is much faster than the arrival of the green batch at SK , leaving SK idle while S3 is overloaded. To address such imbalances, the model splitting point needs to be optimized and the splitting algorithm will be presented in Section IV. 2) Collaborative Inference: Based on the model splitting point, a complete LLM is divided into K sub-models, denoted as w = {w1 , w2 , · · · , wK }. The overall inference process is described as follows. We consider an inference task on a dataset D containing D images. ThePdataset is divided T into T mini-batches d1 , d2 , . . . , dT , with t=1 |dt | = D. For each mini-batch, the inference begins at the first sub-model w1 deployed on satellite S1 , expressed as st1 = f1 (dt , w1t ), where f1 (·) denotes the mapping function from the input to the intermediate output of the first sub-model, consisting of embedding layers, Transformer blocks, and output layers specific to the LLM. The intermediate activation st1 is then transmitted via the inter-satellite link to satellite S2 , serving as input to the next sub-model, which can be st2 = f2 (st1 , w2t ). The inference process continues through the remaining submodels, where each sub-model wk on satellite Sk takes as input the activation from the previous stage, and is formulated as stk = fk (stk−1 , wkt ), for k = 2, 3, . . . , K. The final output from the last sub-model is the predicted result and t t can be expressed as yK = fK (stK−1 , wK ). After all batches t are processed, the final results {yK | t = 1, 2, . . . , T } are aggregated at satellite SK and transmitted to the ground station via the satellite-to-ground link. 3) Pipeline Parallelism: In conventional systems, batch processing occurs sequentially due to tight coupling between computation and communication phases. Each computational stage remains idle until its required inputs are fully transmitted, creating deterministic execution dependencies that serialize pipeline workflows. For instance, the computation of f2 must wait until all st1 are received, and the transmission of st2 must wait until f2 has finished processing. As a result, the inference delay grows rapidly with increasing batch size. To address this, we propose a pipeline-based parallel execution scheme that allows satellites to compute and communicate concurrently. Specifically, while satellite S1 is transmitting st1 , it can immediately begin computing st+1 1 . Each satellite Sk can start computing st+1 while transmitting stk . This concurk
rent execution can be implemented using multiple threading techniques to enable parallel processing of computation and communication, ensuring non-blocking processing. Following the proposed scheme, all satellites can remain active at all times, significantly improving the resource efficiency and accelerating the inference process. C. Activation Compression Scheme An activation compression scheme is proposed that combines learnable sparsification for adaptively selecting informative activations with subsequent quantization and entropy coding to further reduce the transmission, as detailed below. 1) Gumbel Mask-based Sparsification: A lightweight and fully differentiable sparse compression module, primarily guided by Gumbel masking, is proposed to selectively retain the most informative activations while discarding less relevant ones. Given the input activation tensor X ∈ RN ×S×D , the goal is to generate a binary mask M ∈ {0, 1}N ×S×D that selects informative features. To preserve differentiability, the mask is approximated using a Gumbel-Sigmoid sampling mechanism. Each element of the mask is parameterized by a trainable logit αij and perturbed with Gumbel noise to simulate sampling. The noise term Gij is drawn from the standard Gumbel distribution using inverse transform sampling, computed as Gij = − log(− log(Uij )), where Uij ∼ Uniform(0, 1). This sampled noise is then added to αij before applying temperature scaling and a sigmoid function to obtain a continuous approximation of the binary mask. The perturbed logits are scaled by a temperature parameter τ > 0 and then passed through a sigmoid function, resulting in a continuous soft mask, and can be expressed as αij + Gij . (1) M̂ij = σ τ A smaller τ yields mask values that are closer to 0 or 1, effectively approximating discrete binary decisions, while a larger τ results in smoother, more probabilistic outputs between 0 and 1. To obtain discrete binary masks while maintaining gradient flow, the Straight-Through Estimator (STE) is applied. During the forward pass, the mask is binarized as Mij = I(M̂ij > 0.5),
(2)
and during the backward pass, the gradient is approximated as ∇Mij ≈ ∇M̂ij . The sparse activation is then computed as X̃ = M ⊙ X + (1 − M) ⊙ stopgrad(X),
(3)
where ⊙ denotes element-wise multiplication and stopgrad(·) preserves the forward value while blocking gradients from the deactivated features. To promote sparsity in the mask, a regularization term is introduced to constrain the expected number of activated elements. The sparsity loss is defined as S
Lsparse = λ
D
1 XX σ(αij ), SD i=1 j=1
(4)
5
where λ controls the sparsity level, and this term is jointly optimized with the task loss. To facilitate training, the temperature τ is gradually decreased during training using an annealing schedule: t τ (t) = max τmin , τ0 · 1 − , (5) T where t is the current epoch, T is the total number of epochs, and τ0 , τmin are initial and minimal temperatures. Gradually decreasing τ allows the mask to transition from soft exploration to hard selection as training progresses. 2) Quantization: Only the non-zero elements retained by the binary mask M are subject to quantization. Given a target bit-width b, the quantization step size is computed as xmax − xmin , (6) ∆= 2b−1 − 1 where xmin and xmax denote the minimum and maximum absolute values of the active elements in the current batch. Each non-zero element x̃ijk is first quantized by mapping it to an integer code using qijk = k j |x̃ijk |−xmin + 0.5 , and then dequantized back sign(x̃ijk ) ∆ to the real domain using x̂ijk = sign(x̃ijk ) (xmin + qijk ∆) . Since the quantization process involves non-differentiable rounding operations, the straight-through estimator (STE) is employed to preserve gradient flow, in which the gradient of the dequantized value is approximated by that of the original input. To mitigate the influence of outliers, the quantization range [xmin , xmax ] is dynamically estimated from each mini-batch. If no elements are active in a batch, quantization is skipped, and the sparse tensor is directly forwarded to the next stage. 3) Entropy-Guided coding: To evaluate compressibility, the expected bit length is approximated using the entropy of the quantized value distribution: X H(S) = − p(v) log2 p(v), (7) v∈V
where S denotes the set of quantized non-zero values, V is the set of unique values, and p(v) is the empirical frequency of symbol v. Based on this entropy estimate, the uncompressed bit length is calculated as Lraw = |S|b, where b is the fixed quantization bit-width. The corresponding estimated compressed bit length using entropy coding (e.g., Huffman coding) is given by Lhuff ≈ |S|H(S). The entropy value reflects the amount of redundancy in the quantized representation. A lower entropy indicates that fewer bits are required on average to encode the data, while a higher entropy corresponds to a more uniform distribution and thus lower compressibility. IV. P ROBLEM F ORMULATION A. Computation and Communication Model 1) Computing Delay: To quantify the computational delay incurred by each satellite during distributed inference, both the computational workloads and the memory capacities of the satellites are taken into account. Let Ck denote the total number of floating-point operations (FLOPs) required to execute the sub-model wk assigned to satellite Sk . Let fk represent
the computing capability of satellite Sk , measured in FLOPs per second. Accordingly, the computation delay of satellite Sk for a single mini-batch is given by Tkcomp = Cfkk , where Tkcomp denotes the time required to complete the inference of one batch through the sub-model wk on satellite Sk . 2) Communication Delay: In collaborative satellite inference, communication comprises three distinct phases: (i) input image upload from the sensing satellite to the initial computing satellite, (ii) intermediate activation transfer between successive computing satellites via inter-satellite links, and (iii) final result download from the last computing satellite to ground stations. The corresponding data sizes are denoted as Sinput , Skact , and Sout , respectively. For phase (i), the input image size is given by Sinput = N P , where N is the batch size and P is the number of pixels per image. For phase (ii), the intermediate activation size at stage k is modeled as Skact = N SD, where D is determined by the LLM architecture and S is the sequence length used in inference [33]. For phase (iii), the final output comprises classification logits or predicted labels over E categories, leading to an output size of Sout = N E. The communication delay at each phase depends on both the data volume and the available link bandwidth. Let rgs denote the satellite-to-ground data rate, and rsat denote the inter-satellite data rate. Accordingly, the delay for transmitting the input image from the ground station to the first satellite is T0comm = Sinput /rgs , the delay for transferring intermediate activations between satellites is Tkcomm = Skact /rsat , and the delay for sending the final output back to the ground is comm = Sout /rgs . TK B. LLM Inference Delay Model The inference process across K pipeline stages, each executed on a computing satellite, consists of two distinct phases: a startup phase and a steady-state phase: 1) Startup Delay: The startup delay refers to the time required for the first input sample to traverse all K stages sequentially, which is computed as the sum of computation and communication delays across all stages, as given by Tstartup =
K X
Tkcomp + Tkcomm .
(8)
k=1
2) Steady-State Delay: Once the startup is completed, each subsequent input can be processed in a steady-state stage. The delay for the steady-satege stage is determined by the slowest satellite, which becomes the bottleneck. Hence, the steadystate stage delay is given by Tsteady = maxk Tkeff . Based on the pipeline, computation and communication can overlap. For satellite k, the overlapping duration is comm Tkoverlap = min Tkcomp , Tk−1 , (9) which captures the maximal parallel time between computing and receiving data. Accordingly, the effective delay is Tkeff = Tkcomp + Tkcomm − Tkoverlap .
(10)
6
3) Total Inference Delay: For a batch of d input samples, the total end-to-end inference delay consists of three components. First, there is the initial transmission delay, denoted as T0comm , which represents the time required to send the data from the ground station to the first satellite. Second, the startup delay, Tstartup , accounts for the time needed to fill the pipeline, during which the first sample sequentially passes through all K stages. Third, after the pipeline is filled, the remaining B − 1 samples are processed in a pipelined manner, incurring a steady-state delay Tsteady that is determined by the slowest stage in the pipeline. Thus, the total inference delay is Ttotal = T0comm + Tstartup + (B − 1) max Tkeff . k
(11)
C. Accuracy and Memory Model
s.t.
Activation compression may cause approximation errors and degrade the LLM inference accuracy. Directly modeling the accuracy as a function of the full compression vector q = [q1 , . . . , qK−1 ] is intractable due to the highdimensional combinatorial space. Instead, the relationship between accuracy and compression is calibrated along the setting q1 = · · · = qK−1 = q. Specifically, an offline sweep over compression ratios q ∈ Q is performed and the corresponding end-to-end accuracy is measured on a calibration set, thereby obtaining empirical pairs (q, a(q)). A monotone regression d function Acc(q) is then fitted to characterize the relationship between compression and accuracy. In (13e), the per-stage accuracy function is defined using the same for all stages: d k ), Acck (qk ) ≜ Acc(q
∀k ∈ {1, . . . , K − 1}.
fk as the computational capacity of satellite k, then the corresponding computation delay is Tkcomp = Ckf(lk k ) . To reduce communication overhead between adjacent satellites, we introduce a new variable qk ∈ (0, 1], representing the compression ratio applied to the output feature map at satellite k. A smaller qk indicates higher compression. Assuming the uncompressed feature size is Sk , the communication delay Sk . from satellite k to k + 1 is given by Tkcomm = qkrsat 2) Optimization Problem: The final optimization problem is formulated as follows: K X Ck (lk ) qk Sk min + + (B − 1) max Tkeff (lk , qk ) k fk rsat {lk },{qk } k=1 (13a)
(12)
Accordingly, the constraint Acck (qk ) ≥ Accmin serves as a sufficient and conservative condition to ensure that compression on each inter-satellite transmission meets the required accuracy threshold. The onboard memory consumption is similarly modeled as a function of the assigned layers. Specifically, an offline profiling procedure is conducted to measure the memory usage under different layer allocations lk , and a fitted function Mk (lk ) is obtained for each satellite k. The resulting memory model is incorporated into the optimization via the constraint Mk (lk ) ≤ Mkmax , ∀k ∈ {1, . . . , K}, so that the layer assignment is constrained by the onboard memory budget of every satellite. D. Problem formulation An optimization problem is formulated to minimize the total inference delay for the proposed collaborative LLM inference scheme. The objective is to jointly optimize the model-splitting strategy and the inter-satellite activation compression ratios by assigning the L model layers to K computing satellites, subject to onboard memory and inference-accuracy constraints. 1) Optimization Variables: Let lk ∈ Z+ denote the number of layers assigned to satellite k, satisfying the total layer PK constraint l = L. The variable lk determines the k k=1 computational workload at satellite k. Denote Ck (lk ) as the total number of FLOPs required for the assigned layers, and
K X
lk = L,
lk ∈ Z+ ,
(13b)
k=1
Mk (lk ) ≤ Mkmax , qk ∈ [0, 1],
∀k ∈ {1, . . . , K},
∀k ∈ {1, . . . , K − 1},
Acck (qk ) ≥ Accmin ,
(13c) (13d)
∀k ∈ {1, . . . , K − 1}. (13e)
The constraint in (13b) ensures that the total number of layers allocated across all satellites equals the model depth L. Constraint (13c) ensures the memory requirement Mk (lk ) on each satellite does not exceed its maximum onboard memory Mkmax . Constraint (13d) constrains the compression ratios qk to lie in [0, 1], where qk = 1 indicates no compression and smaller values correspond to more aggressive compression. Constraint (13e) imposes per-stage accuracy constraints, where Acck (qk ) represents the accuracy function with respect to compression ratio at stage k, and Accmin is the acceptable lower bound. The steady-state delay at stage k, denoted by Tkeff (lk , qk ), is given by Ck (lk ) qk−1 Sk−1 Ck (lk ) qk Sk + − min , . Tkeff (lk , qk ) = fk rsat fk rsat (14) The above optimization problem is non-trivial to solve due to the following three challenges. Firstly, it is a mixedinteger nonlinear programming (MINLP) problem involving both discrete variables {lk } ∈ Z+ and continuous variables {qk } ∈ [0, 1], which are strongly coupled in the effective delay term maxk Tkeff (lk , qk ), making the joint optimization more complex. Secondly, the maximum term maxk Tkeff (lk , qk ) introduces non-smoothness, which prevents direct application of gradient-based or convex optimization techniques. Thirdly, the solution space expands exponentially with both the number of layers L and the number of computing stages K. The combinatorial nature of layer partitioning under the constraint PK l = L, combined with the continuous search space of k k=1 compression ratios {qk }, results in a high-dimensional search space and making exhaustive enumeration computationally prohibitive, especially for LLM. To address the non-smooth max operator in the objective function, an auxiliary variable θ is introduced to represent the maximum effective stage delay across all satellites. The transformation allows the objective to be linearized by minimizing θ, while the original max term is incorporated into
7
the constraints. The reformulated optimization problem is expressed as K X Ck (lk ) qk Sk min + + (B − 1)θ (15a) fk rsat {lk },{qk },θ k=1
s.t.
Tkeff (lk , qk ) ≤ θ,
∀k ∈ {1, . . . , K},
(13b), (13c), (13d), and (13e).
(15b)
To efficiently solve the layer partitioning problem defined in Eq. (15), the original problem is reformulated as a shortestpath problem over a directed acyclic graph (DAG). Each path in the graph corresponds to a valid assignment of model layers to satellites, and the objective is to find the path with the minimum total inference delay. 1) Graph Construction: A DAG G = (V, E) is constructed to represent the space of layer-to-satellite assignment decisions. The source node is (0, 0), indicating that no layers have been assigned yet, and the sink node is (L, K), corresponding to a complete layer allocation over K satellites. A directed edge e = ((l, k), (l′ , k + 1)) ∈ E exists if the (k + 1)-th satellite is assigned layers l + 1 through l′ , i.e., lk+1 = l′ − l, max is satisfied. and the memory constraint Mk+1 (l′ − l) ≤ Mk+1 In addition to layer allocation, each edge implicitly determines a compression ratio qk+1 ∈ [0, 1] for the output features transmitted from satellite k + 1 to k + 2, which affects both communication delay and accuracy. The optimal value of qk+1 on each edge is determined through a per-edge local optimization process, which will be detailed in the next subsection. 2) Edge Cost: Each edge e = ((l, k), (l′ , k + 1)) represents an assignment of lk+1 = l′ − l layers to satellite Sk+1 and carries two key attributes. First, the edge cost encodes the startup delay due to both computation and communication, which can be expressed as (16)
⋆ where qk+1 denotes the compression ratio determined for this edge. Second, each edge also stores the stage-specific effective delay under pipelined execution: eff Tk+1 (l′ − l) =
⋆ Sk+1 Ck+1 (l′ − l) qk+1 + fk+1 rsat Ck+1 (l′ − l) qk⋆ Sk − min , , fk+1 rsat
(17)
where qk⋆ is the compression ratio from the preceding stage. Unlike edge costs, these effective delays are tracked separately for each edge and used to estimate path-level bottlenecks. 3) Path Cost: The layer allocation corresponds to a path P from (0, 0) to (L, K). The total inference delay associated with P is given by X C(P) = C(e) + (B − 1)θ(P), (18) e∈P
min
P∈Paths(0→L)
C(P),
(19)
subject to feasibility constraints such as memory usage, accuracy requirements, and complete layer coverage.
A. Problem Reformulation
⋆ Sk+1 Ck+1 (l′ − l) qk+1 + , fk+1 rsat
P ∗ = arg
(15c)
V. P ROPOSED S OLUTION
C(e) =
where θ(P) = maxk Tkeff is the maximum effective delay among all stages in the path, corresponding to the steady-state bottleneck during pipelined execution. 4) Problem Reformulation: The original model-splitting optimization then reduces to a minimum-cost path search over the constructed DAG, which can be expressed as
B. Per-Edge Compression Optimization The compression ratio optimization is embedded as an inner-loop routine within the LLM splitting algorithm. 1) Solution Strategy: Given a current layer partitioning {lk } obtained from the graph-based reformulation, the compression ratios qk are next optimized to further reduce the total inference delay. The compression ratios directly influence the transmission time of intermediate features between adjacent satellites, and thereby affect both the stage-wise delay and the overall pipeline delay. Each stage’s effective delay includes a min operator coupling qk and qk−1 , which makes the optimization problem nonsmooth and prevents closed-form solutions. To address this, the compression optimization is reformulated as a constrained numerical problem: K X Ck (lk ) qk Sk + + (B − 1)θ (20a) min fk rsat {qk },θ k=1
s.t.
Tkeff (lk , qk ) ≤ θ, qk ∈ [0, 1],
∀k ∈ {1, . . . , K},
(20b)
∀k ∈ {1, . . . , K − 1},
(20c)
Acck (qk ) ≥ Accmin ,
∀k ∈ {1, . . . , K − 1}. (20d)
Due to the problem’s low dimensionality, practical solutions include grid-based search or constrained optimization solvers such as sequential quadratic programming (SQP). In particular, we adopt a full-grid enumeration method with fixed resolution to exhaustively evaluate all feasible {qk } combinations and determine the optimal one that minimizes the objective in (20a). The procedure is detailed in Alg. 1. The resulting optimal compression settings {qk∗ } are then used to refine layer assignment in the next iteration. The alternating optimization procedure between layer partitioning and compression is described in the next section. 2) Computational Complexity: The total number of compression configurations to evaluate is (N + 1)K−1 , where K is the number of computing satellites and N is the grid resolution for each compression ratio qk ∈ {0, N1 , . . . , 1}. For each candidate setting, the algorithm evaluates K stage-wise delay constraints and K−1 accuracy constraints, each assumed to take constant time. Therefore, the total computational complexity of Alg. 1 is O (N + 1)K−1 K . C. Cut Layer Selection Algorithm with Integrated Compression Given satellite communication conditions, task requirements, LLM size, and other factors, the split points are determined by the algorithm described below.
8
Algorithm 1: Discrete Search for Compression Ratios
Algorithm 2: DAG-Based Model Splitting Algorithm
Input: Layer assignment {lk }, accuracy functions Acck (·), grid resolution N 1 1: Initialize candidate set Q ← all {qk } with qk ∈ {0, N , . . . , 1} 2: Best cost C ∗ ← +∞, Best setting {qk∗ } ← ∅ 3: for all candidate {qk } ∈ Q do 4: Compute Tkeff (lk , qk ) and check constraint (20b) 5: Evaluate accuracy constraint (20d) 6: if all constraints satisfied then 7: Compute total delay cost Ctotal using (20a) 8: if Ctotal < C ∗ then 9: Update C ∗ ← Ctotal , {qk∗ } ← {qk } 10: end if 11: end if 12: end for 13: return Optimal compression ratios {qk∗ }
Input: Total layers L, satellites K, memory budget Mkmax Output: Optimal layer partition {lk } and compression ratios {qk }; 1: Initialize priority queue Q with start node (0, 0) and empty {lk }, {qk } 2: while Q is not empty do 3: Pop node (l, k) with lowest cost f = g + h; 4: if (l = L) and (k = K) then 5: return current path {lk } and corresponding {qk }; 6: end if 7: for each valid layer split l′ > l for satellite k + 1 do 8: if Memory constraint on layers l + 1 to l′ at Sk+1 violated then 9: continue 10: end if 11: Tentatively assign layers l + 1 to l′ to satellite Sk+1 ; 12: Solve compression subproblem (Eq. 20) to get optimal {qj }k+1 j=1 for current {lj }k+1 j=1 ; 13: Compute Tkeff (lk , qk ) for all previous stages; P q S Cj (lj ) 14: Update cost-to-come g ′ = k+1 + rj j ; j=1 f j
1) Solution Strategy: To efficiently solve the joint optimization of LLM splitting and compression configuration, an A*-based alternating optimization algorithm is designed that updates the optimal compression ratio at each stage during the path search. The essential components of the algorithm are defined as follows: • Startup cost: the accumulated computation and communication delay along the current path from the source node (0, 0) to v. It reflects the total startup latency before pipelined execution enters steady state: X g(v) = C(e), (21) e∈P0→v
•
where P0→v denotes the partial path ending at node v. Steady-state cost: the maximum stage-wise effective delay encountered along the partial path, used to estimate pipeline bottleneck: θ(v) = max T eff (e), e∈P0→v
•
(22)
where T eff (e) is defined as in Eq. (17). Heuristic estimate: a lower-bound estimate of the remaining delay to reach the terminal node (L, K). To ensure admissibility, we assume all remaining layers are assigned to the fastest available satellite: PL ci h(v) = i=l+1 , (23) fmax
where ci is the computation cost of layer i, and fmax = maxj>k fj is the maximum speed among remaining satellites. The total search cost is formulated as f (v) = g(v) + (B − 1)θ(v) + h(v),
(24)
which reflects the total inference delay for B pipelined inputs, combining startup cost, steady-state cost, and a heuristic estimate of future cost. During the search, for each valid layer split from (l, k) to (l′ , k + 1), the algorithm temporarily assigns layers l + 1 to l′ to the (k +1)-th satellite. A per-edge compression subproblem is solved (Eq. 20) to determine the optimal compression ratio eff qk+1 . The resulting Tk+1 is used to update g(v) and θ(v). If the new path yields a lower total cost f (v), it is inserted into
sat
15: Update max-stage delay θ = maxj≤k+1 Tjeff (lj , qj ); 16: Compute total cost f ′ = g ′ + (B − 1)θ + h; 17: Push node (l′ , k + 1) with updated {lj }, {qj }, g ′ , θ into Q; 18: end for 19: end while 20: return Infeasible
the priority queue. The search terminates once the goal node (L, K) is reached, returning the optimal partition {lk } and corresponding compression ratios {qk } that minimize the total pipeline inference delay. The complete procedure is outlined in Alg. 2. 2) Computational Complexity: Let L be the total number of model layers, K the number of satellites, and N the number of discrete grid values used for each compression ratio qk ∈ {0, N1 , . . . , 1}. The algorithm explores paths in L a layered graph with at most K−1 valid layer partitions. For each partition candidate, it solves a nested compression subproblem using grid search over (N + 1)K−1 configurations (as analyzed in Algorithm 1). Thus, the worst-case time com L (N + 1)K−1 K , where plexity of Algorithm 2 is O K−1 L K−1 represents the number of ways to divide L layers into K contiguous segments, (N + 1)K−1 is the number of candidate compression configurations, and K accounts for evaluating all stage-wise delays and constraints per candidate. VI. P ERFORMANCE E VALUATION A. Experimental Settings 1) Satellite Networks: In the simulation, a single orbital plane is considered from a Walker Delta constellation with parameters (1, 12/0, 53◦ ), where twelve satellites are evenly distributed in a circular LEO with an altitude of 500 km and an inclination of 53◦ . Each satellite moves along the same orbital trajectory, with an observation window of 10 minutes per cycle. The total simulation period spans 144 time slots, corresponding to a full 24-hour cycle, during which the orbital plane completes one full rotation around the Earth. The observation target and the ground station are located at (0◦ N, 0◦ E) and (−53◦ N, 180◦ W), respectively. The first satellite in the orbital plane is equipped with an imagery sensor and responsible for data collection. Within each 10-minute time slot, it captures approximately 300 high-resolution remote
9
sensing images, including resolutions of 1080p, 2K, 4K, 8K, and 16K, to simulate diverse data volume and quality demands in practical applications. The S2G communication utilizes the Ka-band, and the transmission rate is assumed to remain constant within each 15-second interval. The ISLs that offer a stable and high-throughput channel, assuming negligible relative motion between satellites on the same orbital path. Each computing-enabled satellite is equipped with onboard processing units and a memory capacity of 8 GB to support edge computing tasks, such as intermediate feature extraction, compression, or partial inference during the data relay and collaboration process. 2) Computation Configuration: The experimental evaluation is based on measurement data collected from a prototype satellite–ground collaborative inference testbed. The ground station is represented by a server equipped with an NVIDIA RTX 4070ti GPU. Satellite-side computation is emulated by four NVIDIA Jetson AGX Orin devices, each capable of delivering up to 275 TOPs of AI performance, which are widely used in practical aerospace systems, such as those deployed on SpaceX satellites [34]. The software stack is built on PyTorch for model inference, with FastAPI over HTTP enabling communication. A multi-threaded execution pipeline is implemented to decouple transmission and inference, allowing data processing and transmission to proceed in parallel across nodes. To simulate the heterogeneous computational capabilities of satellites in orbit, the Jetson AGX Orin devices are configured to operate under different power modes. Specifically, the power limits are set to 15W, 30W, and 50W, corresponding to satellites with heavy computational loads or energy constraints, moderate processing availability, and idle nodes with full capacity, respectively. 3) Communication Configuration: Both satellite-to-ground and inter-satellite communications are considered under the following communication settings. For the satellite-to-ground link, a carrier frequency of 40 GHz with a bandwidth of 1 GHz is considered. The satellite transmit power is set to 35 dBm, and the antenna gain is 37 dBi. The path loss exponent is configured as 2.5 to account for free-space attenuation. The communication availability is determined by a visibility angle threshold of 50◦ . For the inter-satellite free-space optical link, the laser wavelength is set to 1550 nm, with a transmit power of 10 dBW (10 W). The optical beam has a divergence angle of 50 µrad, and the receiver aperture diameter is 0.10 m. A total optical system loss of 6 dB is assumed, capturing pointing errors and hardware losses. Thermal noise is modeled using a Boltzmann constant of 1.38 × 10−23 J/K and a system temperature of 290 K. The available FSO communication bandwidth is set to 0.5 GHz. 4) Application Scenario: A typical satellite-based computer vision application is considered, in which real-time images captured by a remote-sensing satellite are processed collaboratively across a satellite constellation. In particular, an object detection task is considered, and the end-to-end delay from the start of image processing to the delivery of results to the ground is evaluated. To assess the performance of our collaborative inference scheme across different computational loads, four Vision Transformer (ViT) models of varying sizes
TABLE II: System Parameters in Simulation Parameter Number of satellites Batch size Ground server GPU Satellite devices Intersatellite link rate Satellite-ground link rate Downlink window
Value 5 64 NVIDIA RTX 4070ti 4× NVIDIA Jetson AGX Orin 0.5 Gbit/s 6 Gbit/s 10 minutes per satellite
TABLE III: Model and Dataset Settings Item ViT-B ViT-L ViT-H ViT-G EuroSAT RESISC45
Value / Description 0.086 Billion parameters, 2 GB memory 0.307 Billion parameters, 4 GB memory 0.632 Billion parameters, 7 GB memory 1.8 Billion parameters, 12 GB memory 27,000 RGB images, 10 scene classes 31,500 RGB images, 45 scene classes
are adopted: ViT-B (Base), ViT-L (Large), ViT-H (Huge), and ViT-G (Giant) [35], [36]. These models differ significantly in both parameter count and memory footprint. Specifically, ViT-B contains approximately 0.086 billion parameters and typically requires around 2 GB of memory during inference. ViT-L increases the parameter count to 0.307 billion and consumes about 4 GB of memory. ViT-H includes 0.632 billion parameters and requires approximately 7 GB of memory. ViTG, the largest model in our study, has 1.8 billion parameters and demands over 12 GB of memory. 5) Datasets: The EuroSAT dataset is a remote sensing scene classification benchmark based on Sentinel-2 satellite imagery [37]. It consists of 10 land use and land cover categories, such as agricultural, forest, and residential areas. The dataset contains a total of 27,000 RGB images with a resolution of 64×64 pixels, divided into 19,500 training images and 7,500 testing images. The RESISC45 dataset is another scene classification dataset comprising 45 categories, including airport, harbor, desert, stadium, and more [38]. It contains 31,500 RGB images with a resolution of 256×256 pixels, with each category represented by 700 images. The dataset is split into 25,200 training images and 6,300 testing images. 6) Benchmark Schemes: To evaluate the performance of our satellite-based collaborative inference framework, the following baseline strategies are considered: Ground-only: The sensing satellite captures raw images and transmits them to the ground station via K relay satellites. The transmission follows a pipelineparallel manner, where multiple inputs are continuously forwarded stage by stage. Raw images are downlinked for full-model inference on powerful terrestrial servers. • Single-satellite: The sensing satellite forwards the captured image to a designated computing satellite. The full model is deployed on this satellite to perform the complete inference locally. The final prediction result is then transmitted back to the ground station through relay satellites. •
7) Metrics: The effectiveness of the collaborative inference scheme is evaluated using the following metrics:
10
10 3
1.58x
2
5 1
0
0
1.46x 240p
240p
360p
360p
540p
540p
Resolution
720p