arXiv CS open access

User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

2026 · arxiv_cs

arXiv CS · Papers · 2026

distributed-systems internet networking protocols

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling Alfreds Lapkovskis , Ali Beikmohammadi , Sindri Magnússon , and Praveen Kumar Donta

arXiv:2608.11840v1 [cs.DC] 12 Aug 2026

Department of Computer Systems and Sciences, Stockholm University SE-106 91 Stockholm, Sweden {alfreds.lapkovskis, beikmohammadi, sindri.magnusson, praveen}@dsv.su.se

Abstract. Growing demand for artificial intelligence (AI) inference services requires scalable infrastructure, yet centralized serving costs rise with demand. We propose a collaborative distributed inference system combining dedicated infrastructure with resources contributed by service users. Dedicated resources provide baseline capacity for maintaining quality of service (QoS), while volunteered resources absorb increasing demand without proportional growth in centralized infrastructure. To capture stochastic and dynamic interactions among users, resources, tasks, and policies, we develop a high-dimensional generative Markov model with structured temporal factorization. The model supports simulation and provides a foundation for task scheduling and QoS-aware resource allocation optimization. We evaluate the system across user populations, resource capacities, and centralized and distributed scheduling policies. Simulations show that distributed scheduling becomes increasingly advantageous as the user population grows, improving request completion and P99 latency while substantially reducing dedicated resource consumption. These results demonstrate the feasibility of user-assisted collaborative inference for infrastructure-efficient autoscaling. Keywords: Distributed AI Inference · AI Inference Services · Generative Markov Model · Quality of Service · QoS-Aware Scheduling · Resource Autoscaling · Volunteer Computing

Introduction

Artificial intelligence (AI) inference has become a core service in modern computing systems. According to the AI Index Report 2026, corporate AI adoption has reached 88%, while global investments surged to $581.69 billion in 2025 [21]. As AI inference scales to meet this demand, cumulative inference costs increasingly outpace one-time training expenses, making operational efficiency a growing concern. These trends highlight the importance of the computing paradigms underlying AI inference. Central to these paradigms is cloud computing, the de facto standard for Internet applications [18], including AI inference. Its key advantages include on-demand resource provisioning, rapid elasticity, standardized access interfaces, utilization monitoring, and pay-as-you-go pricing [17,15].

A. Lapkovskis et al.

However, centralized clouds are also subject to significant challenges. The costs associated with the setup and management of cloud infrastructure are prohibitively high [19,18]. Furthermore, expenses grow proportionally with demand, making cloud computing increasingly costly at scale. This scaling also carries a substantial environmental burden. Training Grok 4 alone emitted more CO2 than an average car produces over its entire lifetime, and the annual inference cost of GPT-4o exceeds the drinking water demand of 12 million people [21]. More broadly, cloud data centers consume vast amounts of energy and freshwater, with AI workloads emerging as a primary driver [13], and these impacts are projected to worsen as AI demand continues to rise. Edge and fog computing partially address these limitations. By distributing computation closer to end users and data sources, they reduce latency and bandwidth consumption, and decrease reliance on centralized cloud resources [7]. The computing continuum extends this approach by integrating cloud, fog, and edge tiers into a unified, synergistic architecture [8]. Recent work has improved inference within these environments through heterogeneous on-device coexecution [26], task-aware DNN partitioning in mobile edge computing [9], and joint batching and transformer partitioning for distributed LLM inference [28]. Other studies address on-device inference techniques [24], disaggregated inference deployment across heterogeneous infrastructure [27], inference-data distribution for model monitoring [16], and cloud inference autoscaling [11]. These approaches improve inference within predefined deployment environments, but generally rely on dedicated infrastructure rather than resources that become available dynamically as users join the service. Moreover, computing-continuum systems still face architectural and economic barriers to widespread adoption [6,20]. Yet deploying dedicated edge and fog infrastructure is not the only path forward. A vast number of commodity devices owned by individuals and institutions worldwide, remain largely underutilized [4,19]. Harnessing these idle resources toward a shared computational objective is known as volunteer computing [15,18], a paradigm that offers access to cheaper, distributed compute without requiring energy-hungry data center infrastructure. However, existing systems provide limited support for stable QoS guarantees due to resource heterogeneity and intermittent availability [18]. Hence, this model has limited applicability to consumer-oriented AI inference applications. Yet the underlying concept remains compelling in the context of AI inference, where operational costs are a growing concern. Therefore, we propose a collaborative distributed inference system that partitions inference computations between dedicated resources, such as centralized servers, and volunteered resources, such as users’ devices (as shown in Fig. 1). Unlike centralized cloud inference or approaches distributed exclusively across dedicated computing continuum infrastructure, our system does not require dedicated computational capacity to scale in proportion to demand. In contrast to conventional volunteer computing, however, it retains dedicated infrastructure to provide a reliable baseline capacity and support QoS guarantees despite the intermittent availability and heterogeneity of volunteered resources. Given an

User-Assisted Collaborative Distributed Inference

Phase 1: Users request inference Computation Request 1 Request 2 Dedicated server Phase 2: Server schedules subtasks according to its policy Inference request

Subtasks for Request 1

ue eq

s ue

eq R

Subtasks for Request 2

Phase 3: Server aggregates results and returns responses to users

Subtask results

Online user devic es

vices

e de Offlin

Result for Request 1 Result for Request 2

Fig. 1: Overview of the proposed distributed inference system. The server receives requests, schedules subtasks across dedicated and volunteered resources according to the scheduling policy, and aggregates the results into responses. appropriate scheduler, the server can process all requests when few users are online; as the number of available users grows, the demand increases, but so does the pool of volunteered resources, allowing the system to progressively offload computation to user devices. Determining how to schedule tasks across these dynamic resources and how much dedicated capacity is necessary to satisfy QoS requirements requires a model that captures the complex interactions among users, resources, tasks, and policies. We therefore develop such a model to enable system analysis, scheduling optimization, and QoS-aware allocation of dedicated resources. Our contributions are threefold: 1. We propose a novel collaborative distributed AI inference system that combines the dedicated and volunteered user resources for automatic scaling while maintaining QoS and reducing the need for centralized resources. 2. We model the system as a generative Markov model whose temporal factorization captures sparse dependencies among structured state variables and their elements, enabling tractable simulation, inference, and policy learning. 3. We evaluate and compare various scheduling policies by simulating the proposed system under diverse configurations, varying the number of users, dedicated, and volunteered resources.

Proposed Work

We propose a collaborative distributed inference system that partitions inference computations between dedicated resources, such as centralized servers, and volunteered resources, such as users’ devices. Ensuring QoS in such a system requires not only provisioning additional dedicated resources but also optimizing computation scheduling strategies. In principle, QoS can be maintained either by increasing server capacity and executing more computations on dedicated infrastructure or by improving the scheduler to make more effective use of volunteered

A. Lapkovskis et al.

resources. However, volunteered resources are inherently heterogeneous, highly dynamic, and intermittently available. Scheduling decisions must therefore be adaptive and capable of anticipating future system states to prevent QoS degradation while efficiently utilizing the available resource pool. Optimizing such scheduling strategies directly on a live system is impractical: it would be costly, time-consuming, and require extensive trial-and-error exploration in a complex, dynamic environment. Consequently, a system model is needed to enable efficient evaluation and optimization of scheduling strategies. In the following section, we propose our model of the system. 2.1

System Model

We model the system using the generative Markov framework introduced in [12], where states st evolve over time according to the Markov property. The framework provides a unified representation of system components and allows their distributions to be modeled, modified, and extended independently. Its sparse factorization makes the high-dimensional model tractable for simulation, inference, and policy learning. We consider a system consisting of a dedicated server v0 , a set of users i ∈ I, and a set of nodes v ∈ V = {v0 } ∪ I, each providing resources r ∈ R. The system supports |K| types of inference tasks, where each task type k ∈ K consists of a set of subtasks p ∈ Pk . Users can both request inference tasks and contribute resources to execute subtasks. The system is controlled by two decision-making policies: a scheduler policy π and an executor policy ς. Together, these policies generate the joint action ut = ⟨uπt , uςt ⟩. We assume a fixed interval of 1 s between timesteps t. We define the system state st as a tuple of high-dimensional variables: st = ⟨ot , qt , at , xt , yt , ct , dt , ut ⟩,

(1)

where each variable captures a specific aspect of the system state. Each variable is indexed by the system entities to which it applies, with one dimension assigned to each entity type. Following the Markov property, variables may depend on other variables within the same or previous time step. These dependencies are generally sparse, since many entities are independent in various state variables. In the following sections, we describe the system variables and the implementations of the decision-making policies. 2.2

System Variables

Availability states (ot ). We represent user availability with the online indicator ot ∈ {0, 1}|I| and the state-duration variable dot ∈ N|I| , where ot,i = 1 indicates that user i ∈ I is online and dot,i denotes the number of time steps since the last online/offline transition. These variables evolve as dot ∼ P (· | dot−1 , ot−1 ) and ot ∼ P (· | ot−1 , dot ), factorized over users under independent availability processes. The duration dependence captures the fact that transition probabilities

User-Assisted Collaborative Distributed Inference

change with the time spent in the current state, while ot−1 selects the corresponding online or offline duration model. For each user, dot,i increments if the state persists and resets to 0 if it changes; ot,i then remains equal to ot−1,i when dot,i > 0 and switches to 1 − ot−1,i otherwise. We model availability durations with parametric survival distributions, following findings from [10], and sample transitions from their discrete-time hazards. We use a Weibull survival function S(t) ≜ exp(−(t/λ)k ) for online durations, with λ = 1133.1 and k = 0.5764, and a log-normal survival function S(t) ≜ 1 − Φ((log t − µ)/σ) for offline durations, with µ = log(5259), σ = 1.5, and standard normal CDF Φ(·). Given duration d − 1, the current state persists with probability S(d)/S(d − 1) and changes otherwise. These parameters yield median online and offline durations of approximately 10 minutes and 1.5 hours, respectively, and a steady-state user availability of approximately 10%. Inference requests (qt ). We represent inference requests with qt ∈ Q|I| , where Q = K ∪{k0 } and k0 denotes no active request, and with the request-state duration dqt ∈ N|I| , where dqt,i counts the time steps since the last request-state change of user i. The variables evolve as dqt ∼ P (· | dqt−1 , qt−1 , ot , yt−1 ) and qt ∼ P (· | qt−1 , ot , dqt ), factorized over users. Users can submit requests only while online; hence, if ot,i = 0, we set qt,i = k0 and dqt,i = 0. Each user has at most one active request, so requests alternate between k0 and some k ∈ K. The duration dqt,i increments while the request state persists and resets when a request starts, an active request completes according to yt−1 , or the user abandons it. In the implementation, we use a Pareto survival function S(t) ≜ 1 for t < κ and S(t) ≜ (κ/t)α for t ≥ κ, following heavy-tailed inter-request times observed in interactive systems [2,5]. We use this distribution both for user think time, when qt−1,i = k0 , and patience, when qt−1,i ̸= k0 and the request has not completed. Given duration d − 1, the current request state persists with probability S(d)/S(d − 1) and changes otherwise; when a new request starts, we sample its type k from Categorical(p1 , . . . , p|K| ). We set κ = 30 s and α = 1.5, yielding a minimum think time and patience of 30 s, a median of approximately 48 s, and equal steady-state fractions of time with and without an active request. Allocated resources (at ). We represent allocated resources with at ∈ |R|×|V | R≥0 , where at,r,v denotes the amount of resource r ∈ R allocated to the system on node v ∈ V . The variable evolves as at ∼ P (· | ãt−1 , ot ), where ãt−1 = [at−1 , . . . , at−H ] captures the autocorrelation of resource trajectories. We factorize the transition over resources and nodes, condition user resources on availability, and treat the server v0 as always online. Thus, we set at,r,i = 0 for offline users and sample allocations for online users and the server. Each at,r,v represents the total resource capacity allocated to the system, including both used and unused capacity. In the implementation, we use R={band in, band out, cpu, memory, storage} and fit one generative ARMA model [3] per resource. We collect workstation measurements every 3 s for 4 hours, resample them to 1 s, standardize each resource, and apply a Yeo–Johnson transformation [25]. For each r, we fit ARMA models with parameters p ∈ {1, 2, 5, 10} and q ∈ {0, 1, 2, 5, 10} by maximum likelihood,

A. Lapkovskis et al.

selecting ARMAA,r with the lowest AIC [1]. During simulation, we sample from ∗ ARMAA,r , invert the transformation, rescale using node-specific mean µA ∗ r,v and A standard deviation σr,v , and clip to [aor,v , aor,v ], where aor,v = 0 and aor,v denote the minimum and maximum permitted allocations. We assume homogeneous user nodes but allow different server parameters. Subtask readiness states (xt ). We represent subtask-data readiness with an auxiliary state x̂t ∈ X |V |×|K|×Pmax , a duration variable dxt ∈ N|V |×|K|×Pmax , and the actual readiness state xt ∈ X |V |×|K|×Pmax , where X = {empty, download, pause, ready}, Pmax = maxk∈K |Pk |, and p ∈ Pk indexes subtasks of task type k. These variables evolve as x̂t ∼ P (· | xt−1 , ot , uxt−1 ), dxt ∼ P (· | dxt−1 , x̂t , cxt ), and xt ∼ P (· | x̂t , dxt ), factorized over nodes, task types, and subtasks. The auxx,ς iliary state first applies scheduler and executor actions uxt−1 = ⟨ux,π t−1 , ut−1 ⟩: the scheduler can start or cancel downloads, while the executor can pause, resume, or abort them. We suppress transitions for offline users and treat the server v0 as always online; moreover, the server stores all subtask data a priori. The duration dxt,v,k,p matters only when x̂t,v,k,p = download. For all other states, we set it to 0. During an active download, dxt,v,k,p increments while the accumulated storage consumption cxt,storage,v,k,p remains below the required subtask-data size ck,p , and resets to 0 once cxt,storage,v,k,p = ck,p . The actual state xt copies x̂t , except that x̂t,v,k,p = download becomes xt,v,k,p = ready when dxt,v,k,p = 0, indicating download completion. In the implementation, paused downloads retain their accumulated storage, ready subtasks keep storage fixed at ck,p , and all non-storage download consumptions vanish outside the download state. Subtask execution states (yt ). We represent subtask execution with an auxiliary state ŷt ∈ Y |I|×|V |×|K|×Pmax , a duration variable dyt ∈ N|I|×|V |×|K|×Pmax , and the actual execution state yt ∈ Y |I|×|V |×|K|×Pmax , where Y = {idle, download, execution, upload, done}. The entry yt,i,v,k,p describes the execution state of subtask p ∈ Pk of user i’s request of type k on node v. These variables evolve as ŷt ∼ P (· | yt−1 , ot , uyt−1 ), dyt ∼ P (· | dyt−1 , ŷt , cyt ), and yt ∼ P (· | ŷt , dyt ), factorized over users, nodes, task types, and subtasks. The scheduler action uy,π t−1 assigns, cancels, or finalizes subtasks, while the executor action uy,ς t−1 can abort running subtasks. Assigning a subtask starts with download on user nodes and directly with execution on the server v0 ; offline user nodes reset non-completed subtasks to idle, whereas done subtasks can still be finalized because the server already has their outputs. The duration dyt,i,v,k,p tracks the current active phase y ∈ Ỹ = {download, execution, upload}. It increments while the phase continues and resets to 0 when the phase completes, is interrupted, or reaches done. The actual state yt copies ŷt , except that duration resets advance active phases as download → execution, execution → upload and upload → done on user nodes, and execution → done on the server. In the implementation, we model phase completion with discrete-time hazard models [23] fitted separately for each y ∈ Ỹ. We fit these models from measured executions of representative computer-vision subtasks under different bandwidth levels, batch sizes {1, 2, 5, 10, 20, 50}, and

User-Assisted Collaborative Distributed Inference

subtask types. The covariates include standardized resource consumptions cyt , batch size, log dyt,i,v,k,p , polynomial interactions up to degree q ∈ {1, 2, 3}, and one-hot subtask-type indicators; we select a common degree q by average AIC across phases. During simulation, subtasks with the same node, subtask type, and execution age form a batch, share one sampled completion event, and either advance by resetting dy to 0 or continue with dy + 1. Resource consumption (ct ). We define total resource consumption as |Rx |×|V|×|K|×Pmax ct = ⟨cxt , cyt ⟩, where cxt ∈ R≥0 captures subtask-data prepa|Ry |×|I|×|V|×|K|×P

max ration and cyt ∈ R≥0 captures request execution. We use y x R = R \ {band out} and R = R \ {storage}, with execution-stage subsets for downloading, computation, and uploading. The variables evolve jointly as (cxt , cyt ) ∼ P (· | c̃xt−1 , c̃yt−1 , at , x̂t , ŷt , dxt−1 , dyt−1 ), where the history windows c̃xt−1 and c̃yt−1 capture autocorrelation. We condition on allocated resources because they bound feasible consumption, on auxiliary states because inactive subtasks consume no resources, and on durations because subtasks with the same phase age may share resources through batching. Server and user consumptions are coupled: user transfers consume server bandwidth, while limited server bandwidth constrains user-side transfer rates. In the implementation, we fit generative ARMA models to measured consumption traces after resampling to 1 s, standardization, and Yeo–Johnson transformation. For cxt , we collect traces by downloading subtask data in 1 Kb chunks under representative inbound bandwidth levels obtained by applying Lloyd– Max quantization [14] to the inbound-bandwidth traces collected for allocatedresource modeling. We model storage as a delta and accumulate it over time. For cyt , we collect traces from representative subtasks under different bandwidth levels, batch sizes {1, 2, 5, 10, 20, 50}, and subtask types. For each resource and stage, we fit ARMA candidates with p ∈ {1, 2, 5, 10} and q ∈ {0, 1, 2, 5, 10}, selecting the model with the lowest AIC. During simulation, we sample nodelevel consumption, interpolate scaling statistics over the relevant bandwidth or batch-size levels using inverse-distance weighting [22], clip transfer consumption to the effective bandwidth shared with the server, proportionally downscale CPU-dependent throughput when CPU is overcommitted, and divide node-level consumption among concurrent transfers or same-age execution batches.

2.3

Decision-Making Policies

The system uses two types of policies. The server-side scheduler π issues readiness and execution actions, ux,π and uy,π t t , while the node-side executor ς issues constraint-enforcement actions, ux,ς and uy,ς t t . Their action domains are Ux,π = {none, download, cancel}, Uy,π = {none, execute, finish, cancel}, Ux,ς ={none, abort, pause, resume}, and Uy,ς = {none, abort}. Both policies condition on the current state excluding the actions being generated, i.e., uπt ∼ π(· | st \ ut ) and uςt ∼ ς(· | st \ ut ). Executor actions take precedence over scheduler actions. Executor policy. The executor policy ς enforces local feasibility by comparing total consumption from cxt and cyt with allocated resources at . Under

A. Lapkovskis et al.

non-storage overflow, it pauses all active subtask-data downloads on the affected node, since they share a single download channel and can later resume. Under storage overflow, it aborts downloads with the largest storage footprints until feasibility is restored. It also aborts executions whose required subtask data are no longer available or have been aborted. If overflow remains, it aborts executions in the order download → execution → upload; for execution, it groups subtasks by node, subtask, and execution age, and greedily aborts the heaviest normalized-load batches. Paused downloads resume once the node has no active executions and no non-storage overflow. Centralized scheduler policy. The centralized scheduler represents fully centralized inference baseline. Since all executions run on the server v0 , which stores all subtask data a priori, it schedules no user-node downloads. For each active request qt,i = k, it cancels obsolete executions, finishes requests whose required subtasks have reached done, and assigns missing subtasks p ∈ Pk only to v0 , subject to available server memory estimated by m b yk,p,n . Uniform scheduler policy. The uniform scheduler uses online volunteered nodes opportunistically. For each online node, the policy schedules downloading at most one missing subtask data whose storage requirement and conservative download-memory estimate m b x fit within the node’s free capacity; the server is excluded because it already stores all subtask data. The policy cancels obsolete executions, finishes completed requests, identifies requested but unplaced user– subtask pairs, and assigns them uniformly at random among feasible nodes. A node is feasible if it is online, has xt,v,k,p = ready, and has enough free memory for the marginal execution-memory estimate m b yk,p,n . We precompute y x m b and m b k,p,n from learned memory models using a µ + 2σ safety margin, interpolating m b yk,p,n across measured batch sizes. Memory estimates are needed because memory, unlike CPU and bandwidth, is a hard constraint. Affinity-based scheduler policy. The affinity-based scheduler uses the same download, cancellation, and finishing logic as the uniform policy, but assigns executions to improve batching. For each required subtask (k, p), it prioritizes eligible nodes by the number of active executions of the same subtask already placed on them, excluding offline nodes and nodes without ready subtask data. Using m b yk,p,n , it computes how many additional executions fit in each eligible node’s free memory and fills the resulting assignment slots with randomly shuffled users requiring (k, p). This greedily co-locates same-subtask executions to increase batch sizes while aiming to preserve memory feasibility. 2.4

Optimization Problems

Given the proposed system model, two optimization problems naturally arise. First, for a fixed executor policy ς, one can optimize the scheduler policy π to maximize QoS, e.g., by minimizing long-term average request latency: # " T 1X π r(st ) | s0 , u0:t−1 ∼ π , (2) max lim inf E π T →∞ T t=1

User-Assisted Collaborative Distributed Inference

where r(st ) ≜ −1⊤ δ(qt , dqt ) is a reward function, and δ(qt , dqt ) ∈ {0, 1}|I| denotes per-user latency increments. Second, for a fixed scheduler π and executor policy ς, one can optimize the allocation of dedicated server resources. In this case, the objective is to determine the amount of dedicated capacity required to satisfy a target QoS constraint under the given model and policies; e.g., one may require request latency to remain below 10 s for 99% of requests, corresponding to P99 latency. We define this problem as min ∥a† ∥w a†

s.t.

P dqt,i ≤ τk | qt+1,i = k0 , qt,i = k ≥ 1 − ϵk ,

(3) ∀k ∈ K,

where a† is a nominal, i.e., maximum, server resource capacity that parametrizes transition probabilities of at , and ∥·∥w is a weighted ℓ1 norm with per-resource cost weights w, τk are per-request-type latency thresholds, and ϵk ∈ (0, 1) are significance levels. Formal definitions of these problems are provided in [12]. It is noteworthy to mention that this paper does not focus on designing a scheduling algorithm or resource-allocation optimizer. Instead, our primary focus is to define the collaborative distributed inference system and develop a model of its dynamics. We use this model to simulate fixed policies under different configurations and compare collaborative distributed inference with centralized inference. Future work will use this model to address the optimization problems.

Evaluation

3.1

Experimental Setup

We evaluate the proposed system by simulating trajectories of the generative model, sampling st ∼ P (· | st−1 ). We parameterize ARMA resource-consumption and hazard-based duration models from measurements collected on a MacBook M4 Pro with 24 GB RAM and 512 GB storage. We consider four task types: light and heavy, each with 5 or 10 subtasks, sampled with probabilities {0.4, 0.3, 0.2, 0.1} to reflect that simpler tasks are typically requested more often. Subtasks are instantiated using established computer-vision models. Each simulation runs for 100,000 time steps, or approximately 28 hours. We vary user count {100, 1,000, 5,000, 10,000}, user capacity {1×, 2×} of base resource profile, server capacity {10×, 30×, 50×, 80×, 100×}, and scheduler policy. These dimensions test how the system behaves as demand and volunteered resources grow together, how offloading depends on user-device capacity, how much dedicated capacity is needed to sustain QoS, and how scheduling affects system behavior. We define the base resource profile as a moderate, non-intrusive contribution from a user device: 100 Mbit inbound/outbound bandwidth, 3 GB storage, 2.5 GB memory, and 200% CPU. The bandwidth represents neither a weak nor exceptional connection; the storage can hold several small subtasks or one large subtask with a few smaller ones. We set memory and CPU to rounded values

Number of queries

10 1.25

A. Lapkovskis et al. ×106

1.25

Centralized – Completed Centralized – Cancelled Uniform – Completed Uniform – Cancelled Affinity – Completed Affinity – Cancelled

1.00 0.75

×106

1.25

×106

1.25

×106

1.25

1.00

0.75

0.50

0.25

0.00

0.50

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(a) Server 10×

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

(b) Server 30×

×106

0.25 0.00

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

(d) Server 80× (e) Server 100×

Fig. 2: Completed vs. canceled requests under different scheduler policies and server capacities. Each subfigure corresponds to a different server capacity multiplier. The x-axis varies the user configuration, i.e., the number of users (|I|) and the per-user resource-capacity multiplier.

slightly above the largest measured average subtask consumption at batch size 2, making small batches of demanding subtasks feasible with high probability and larger lightweight batches more likely. These values define the maximum allocation thresholds aor,v , while realized capacities at still vary over time. We report request completions and cancellations, P99 latency as the primary QoS metric, dedicated resource consumption, and server/user subtask completion rates. We focus on P99 latency because other latency summaries, such as mean and P95 latency, showed similar trends and are omitted due to space constraints. For full reproducibility, we published our implementation code on GitHub.1 3.2

Results

Completed vs. canceled tasks. Fig. 2 compares completed and canceled requests. The centralized policy is unaffected by user capacity because all subtasks run on the server. Under the centralized policy, at 10× capacity, completions grow only through 1,000 users, then plateau and decline as cancellations surge. Higher capacities delay saturation, but gains diminish beyond 30×; at large user counts, cancellations still equal roughly one third to one half of completions. The uniform policy matches the centralized through 1,000 users and improves with 2× user capacity. At 5,000 users, it performs better at 10× and 30× server capacity and comparably above them. At 10,000, it yields more completions and fewer cancellations at every server capacity, including 100×. Thus, the growing volunteered resource pool offsets demand more effectively than centralized scaling, whose gains from 50× to 100× are incremental. Centralized execution may also form larger batches that save resources but delay individual subtasks, whereas uniform distribution creates smaller batches across more nodes. The affinity-based policy is competitive through 1,000 users and beats the centralized policy at 10× capacity, but otherwise generally yields the worst completion-to-cancellation balance. Its larger batches may extend execution and increase cancellation risk; node departures or executor aborts can also interrupt several batched subtasks simultaneously. Overall, distributed inference is effec1

https://anonymous.4open.science/r/AutoscalingArchitecture-C41F

P99 latency (s)

User-Assisted Collaborative Distributed Inference Centralized Uniform Affinity

500

400

300

200

100

500 400 300

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(a) Server 10×

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

(b) Server 30×

100 0

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

User configuration

10k 10k 1x 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

(d) Server 80× (e) Server 100×

Fig. 3: P99 request latency under different scheduler policies and server capacities. Each subfigure corresponds to a different server capacity. The x-axis shows user configurations. The curves represent average P99 latencies across request types. The faded regions represent the range of P99 latencies across request types, from the lowest to the highest.

tive and feasible, but batching efficiency must be balanced against latency and failure risk, motivating scheduler optimization using the proposed model. Task latencies. Fig. 3 reports P99 request latency. The centralized policy performs best or comparably through 1,000 users, and at 5,000 as server capacity grows. At 10,000, the uniform policy remains substantially better. Centralized latency varies more across request types under higher demand. Besides differences in task complexity, unequal request probabilities lead to frequent subtasks forming larger batches that run longer and reduce the resources available to other requests. More server capacity mitigates but does not eliminate this effect. The uniform policy matches the centralized policy through 1,000 users, outperforms it at 5,000 with up to 30× server capacity, and remains comparable above that. At 10,000, it consistently achieves much lower P99 latency. Its variation under constrained configurations likely arises because weaker users store fewer models, particularly for heavy tasks, forcing more execution onto the server. Increasing user and server capacities distributes more subtasks and largely removes this variation. At 10× server capacity, the affinity-based policy is slightly worse than the uniform policy but better than the centralized policy at large user counts. Beyond 30×, its latency changes little and generally remains above the uniform policy, likely because larger batches delay and couple multiple requests. Overall, both distributed policies show little improvement beyond 30×–50× server capacity, i.e., this range provides sufficient dedicated capacity to complement volunteered resources. Centralized scaling improves slowly and does not surpass the uniform policy at a large scale. Thus, distributed scheduling provides better tail latency and under lower server capacity than centralized scheduling. Dedicated resource consumption. Fig. 4 shows that the centralized policy consistently uses more dedicated resources. CPU consumption reaches every allocation limit, indicating persistent saturation. Since consumption is recorded before the executor resolves hard-constraint violations, mean memory demand reaches nearly twice the allocated 25 GB at 10× capacity and slightly exceeds 75 GB at 30×, causing the subtask aborts. At higher capacities, memory remains within its allocation and grows slowly, suggesting that demand approaches its

A. Lapkovskis et al.

CPU (%)

20000 15000 10000 5000

Centralized Uniform Affinity

20000

15000

10000

5000

100 100 1k 1x 2x 1x

1k 2x

5k 1x

100 100 1k 1x 2x 1x

5k 10k 10k 2x 1x 2x

(b) Server 30×

5000

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

100 100 1k 1x 2x 1x

User configuration

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

100 100 1k 1x 2x 1x

User configuration

150

100

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(f) Server 10×

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

5k 1x

5k 10k 10k 2x 1x 2x

(d) Server 80× (e) Server 100×

150

100 100 1k 1x 2x 1x

1k 2x

User configuration

150

Memory (GB)

Centralized Uniform Affinity

5k 1x

User configuration

(a) Server 10×

1k 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

100 100 1x 2x

1k 1x

1k 2x

5k 1x

5k 2x

10k 10k 1x 2x

User configuration

(g) Server 30×

(h) Server 50×

(i) Server 80×

(j) Server 100×

Fig. 4: Server resource consumption under different scheduler policies and server capacities. The first row reports CPU utilization, where 100% corresponds to full utilization of one CPU core, and the second row reports memory usage. The curves represent average resource consumptions over time. The faded regions represent the corresponding standard deviations.

required level while additional CPU accelerates processing, causing slight further increases in memory usage. Both distributed policies use substantially fewer dedicated resources and never exceed their allocations. Consumption changes little between 30× and 100×, indicating that approximately 30×–50× capacity sufficiently complements volunteered resources. The affinity-based policy is slightly more efficient than the uniform policy, likely because it considers node loads and exploits batching. Overall, distributed scheduling accommodates growing demand without proportional expansion of dedicated infrastructure. Subtask completion. Fig. 5 reports subtask completion rates on server and user nodes; the centralized policy is omitted from user-node plots. Under distributed policies, user-side rates are largely independent of server capacity. The uniform policy achieves approximately 0.75–0.95, improving with user capacity because stronger nodes better tolerate resource fluctuations. Affinity-based scheduling performs better at 100 and 1,000 users with 1× capacity, but degrades as user count grows, especially on weaker nodes. Larger batches may exceed expected resource demands, leading to the simultaneous abort of multiple subtasks due to node departures or resource-constraint violations. Centralized server-side rates at 10× capacity fall from 0.9–1.0 for up to 1,000 users to approximately 0.2 at 10,000. At 30×, they remain near 1.0 through 1,000 users but decline to approximately 0.6 at 10,000; only 100× capacity approaches consistently complete execution. Distributed policies reach near-perfect serverside rates with considerably less capacity, particularly for 2× users, because volunteered resources reduce server overcommitment. Some redundant work remains because rule-based schedulers cannot anticipate availability and resource fluctuations. Overall, distribution reduces latency and dedicated capacity needs, showing the value of scheduler optimization.

User-Assisted Collaborative Distributed Inference 1.0

1.0

0.8

0.6

0.4

0.2

0.0

Server completion rate

1.0

0.6

Centralized Uniform Affinity

0.4 0.2 0.0

100 100 1k 1k 5k 5k 10k 10k 1x 2x 1x 2x 1x 2x 1x 2x

User completion rate

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(a) Server 10×

100 100 1k 1x 2x 1x

(b) Server 30×

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

0.2

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

0.0

User configuration

0.9

0.8

0.7

0.6

User configuration

(f) Server 10×

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(g) Server 30×

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(h) Server 50×

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

(d) Server 80× (e) Server 100×

0.9

100 100 1k 1x 2x 1x

User configuration

0.9

100 100 1k 1k 5k 5k 10k 10k 1x 2x 1x 2x 1x 2x 1x 2x

0.6

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(i) Server 80×

100 100 1k 1x 2x 1x

1k 2x

5k 1x

5k 10k 10k 2x 1x 2x

User configuration

(j) Server 100×

Fig. 5: Subtask completion rates under different scheduler policies and server capacities. Completion rate denotes the fraction of scheduled subtasks completed successfully. The first and second rows show completion rates on the server and user nodes, respectively.

3.3

Limitations and Future Work

This paper focuses on the collaborative distributed inference architecture and its modeling framework, instead of a production-ready implementation. Accordingly, we assume homogeneous users with identical availability and resource distributions, which also allows us to omit explicit fairness mechanisms. A real deployment could condition user behavior and resources on temporal and userspecific context and incorporate contribution-aware request prioritization. We further use 1 s time steps from the server’s perspective; irregular observations can be aligned with this representation through interpolation. Although the conceptual execution state grows quadratically with the user count, our implementation uses limited slots to bound the tasks maintained per node and scale linearly. Alternative request-centric representations could further reduce this space cost, making model scalability and decentralization promising research directions. The framework can also naturally extend to multiple dedicated servers, DAG-based workflows, and autoregressive workloads such as LLM inference. Finally, optimizing the scheduler and dedicated resource allocation formulated in Section 2.4, as well as addressing deployment-level security and privacy, remains future work. These promising extensions build directly on the proposed model, whose modular factorization allows individual assumptions and components to be refined without redesigning the overall framework.

Conclusion

We proposed a collaborative distributed inference system that combines dedicated server infrastructure with resources contributed by users’ devices, enabling demand to scale without proportional growth in dedicated capacity. To capture the system’s stochastic and dynamic behavior, we developed a generative

A. Lapkovskis et al.

Markov model that supports simulation, analysis, and policy optimization. Our evaluation showed that distributed scheduling increasingly outperforms centralized inference as the user population grows, improving request completion and tail latency while substantially reducing dedicated resource consumption. These findings establish the feasibility of user-assisted inference and provide a foundation for future work on optimized scheduling, QoS-aware resource allocation, and extensions to richer workloads and deployment environments. Acknowledgments. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725 and also supported by MSCA’s TALENTS (101299722) project.

References 1. Anderson, D., Burnham, K.: Model selection and multi-model inference. Second. NY: Springer-Verlag 63(2020), 10 (2004) 2. Benevenuto, F., et al.: Characterizing user behavior in online social networks. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement. p. 49–62. IMC ’09, ACM, New York, NY, USA (2009). https://doi.org/10.1145/ 1644893.1644900 3. Box, G.E., et al.: Time series analysis: forecasting and control. John Wiley & Sons (2015) 4. Che, D., Hou, W.C.: A novel “credit union“ model of cloud computing. In: Cherifi, H., Zain, J.M., El-Qawasmeh, E. (eds.) Digital Information and Communication Technology and Its Applications. pp. 714–727. Springer, Berlin, Heidelberg (2011) 5. Crovella, M., Bestavros, A.: Self-similarity in world wide web traffic: evidence and possible causes. IEEE/ACM Transactions on Networking 5(6), 835–846 (1997). https://doi.org/10.1109/90.650143 6. Ding, A.Y., et al.: Roadmap for edge ai: a dagstuhl perspective 52(1), 28–33 (Mar 2022). https://doi.org/10.1145/3523230.3523235 7. Donta, P.K., et al.: Exploring the potential of distributed computing continuum systems. Computers 12(10) (2023). https://doi.org/10.3390/ computers12100198 8. Dustdar, S., et al.: On distributed computing continuum systems. IEEE Transactions on Knowledge and Data Engineering 35(4), 4092–4105 (2023). https: //doi.org/10.1109/TKDE.2022.3142856 9. Gao, Y., et al.: Optimizing multi-dnn inference on mobile devices through heterogeneous processor co-execution. IEEE Transactions on Mobile Computing 25(6), 7766–7781 (2026) 10. Javadi, B., et al.: Discovering statistical models of availability in large distributed systems: An empirical study of seti@home. IEEE Transactions on Parallel and Distributed Systems 22(11), 1896–1903 (2011). https://doi.org/10.1109/TPDS. 2011.50 11. Jin, Y., Yang, Z.: Scalability optimization in cloud-based ai inference services: Strategies for real-time load balancing and automated scaling. In: Proceedings of the 2025 4th International Conference on Big Data, Information and Computer Network. p. 266–270. BDICN ’25, ACM, New York, NY, USA (2025). https: //doi.org/10.1145/3727353.3727398

User-Assisted Collaborative Distributed Inference

12. Lapkovskis, A., et al.: Brief announcement: Generative markov model for distributed computing systems. arXiv preprint arXiv:2606.03061 (2026) 13. Li, P., et al.: Making ai less ’thirsty’. Commun. ACM 68(7), 54–61 (Jun 2025). https://doi.org/10.1145/3724499 14. Lloyd, S.: Least squares quantization in pcm. IEEE Transactions on Information Theory 28(2), 129–137 (1982). https://doi.org/10.1109/TIT.1982.1056489 15. Ma, P., et al.: Research allocation in mobile volunteer computing system: Taxonomy, challenges and future work. Future Generation Computer Systems 154, 251– 265 (2024). https://doi.org/https://doi.org/10.1016/j.future.2024.01.015 16. MARZBAN, M.F.A., et al.: Inference data distribution criteria for artificial intelligence or machine learning model monitoring (Jun 19 2025), uS Patent App. 18/544,782 17. Mell, P.M., Grance, T.: Sp 800-145. the nist definition of cloud computing. Tech. rep., National Institute of Standards & Technology, Gaithersburg, MD, USA (2011) 18. Mengistu, T.M., Che, D.: Survey and taxonomy of volunteer computing. ACM Comput. Surv. 52(3) (Jul 2019). https://doi.org/10.1145/3320073 19. Mengistu, T.M., et al.: cucloud: Volunteer computing as a service (vcaas) system. In: Luo, M., Zhang, L.J. (eds.) Cloud Computing – CLOUD 2018. pp. 251–264. Springer International Publishing, Cham (2018) 20. Meuser, T., et al.: Revisiting edge ai: Opportunities and challenges. IEEE Internet Computing 28(4), 49–59 (2024). https://doi.org/10.1109/MIC.2024.3383758 21. Sajadieh, S., et al.: The AI index 2026 annual report. Tech. rep., AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA (April 2026) 22. Shawky, M.M.: A comparative study of interpolation methods for the development of ore distribution maps. Discover Geoscience 3(1), 2 (2025) 23. Singer, J.D., Willett, J.B.: Applied longitudinal data analysis: Modeling change and event occurrence. Oxford university press (2003) 24. Wang, W., et al.: A survey of ai inference technologies for on-device systems. IEEE Internet of Things Journal 12(24), 51927–51950 (2025). https://doi.org/ 10.1109/JIOT.2025.3613691 25. Yeo, I., Johnson, R.A.: A new family of power transformations to improve normality or symmetry. Biometrika 87(4), 954–959 (12 2000) 26. Zhang, G., et al.: Task-aware collaborative inference and fine-grained dnn partitioning in mec networks. IEEE Transactions on Mobile Computing 25(6), 8911–8927 (2026). https://doi.org/10.1109/TMC.2025.3650680 27. Zhang, T., et al.: Dishelis: Optimizing deployment of disaggregated llms inference serving over heterogeneous environments via hierarchical max-flow. IEEE Transactions on Cognitive Communications and Networking 12, 5473–5488 (2026). https://doi.org/10.1109/TCCN.2026.3657037 28. Zheng, T., et al.: Joint optimization of dynamic batching and adaptive partitioning for distributed llms inference in mobile edge computing. IEEE Transactions on Mobile Computing 25(6), 8747–8763 (2026). https://doi.org/10.1109/TMC. 2026.3650838

User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

Analyze & explore related research in Conceptio

Related documents