InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network

I NFER S LICE : A R EFERENCE A RCHITECTURE FOR SLO- DRIVEN LLM I NFERENCE OVER A T ELCO W IDE -A REA N ETWORK

arXiv:2610.32638v1 [cs.DC] 23 Oct 2026

Juan Parra-Ullauri 1 Anthony Chiu 1 Hari Madhukumar 2 Zhaozhou Wu 2 Anderson Bravalheri 2 Alex Healing 1 Andy Corston-Petrie 1 Dimitra Simeonideou 2 Nektaria Efthymiou 1

A BSTRACT LLM inference is increasingly deployed as a core service for enterprise and interactive applications, but sovereignty, compliance, latency and resource-locality requirements can limit the suitability of centralised cloud deployments. Telco operators offer geographically distributed compute and network infrastructure for addressing these constraints. However, existing LLM serving frameworks are designed for tightly coupled data centres and provide limited support for coordinating geographically distributed and heterogeneous resources. We present InferSlice, a reference architecture for Service-Level Objective (SLO)-driven LLM inference over a telco infrastructure, integrating network slicing, GPU partitioning, elastic serving and observability. Led by BT in collaboration with the University of Bristol and NVIDIA, and realised on the JOINER UK national-scale wide-area network (WAN), InferSlice is evaluated on service differentiation across tiers, elastic scale-out, client-edge bandwidth as a bottleneck, and the reach of tensor, pipeline, data parallelism, and prefill/decode disaggregation over the WAN. Our results show that, on a general-purpose telco network, tiers yield distinct latency-throughput regimes, and weight streaming shortens scale-out time. Replication is placement-insensitive but memory-bound; pipeline parallelism tolerates near- and mid-cross-site hops, while tensor parallelism and disaggregation fail beyond a single subnet hop. We share practical insights from operating SLO-driven inference on a telco network, lessons learned and open questions, and hope to encourage further research on distributed and heterogeneous LLM inference systems. 1

I NTRODUCTION

Large Language Models (LLMs) have become a core service for applications ranging from conversational agents to enterprise decision systems and analytics (Huang et al., 2025). As their deployment becomes increasingly widespread, providing scalable, reliable, and low-latency inference has become a first-order systems challenge. LLM inference is predominantly served from centralised cloud infrastructure (Yu et al., 2022; Zhong et al., 2024), which can introduce structural limitations. First, sovereignty and compliance a : routing sensitive data through centralised infrastructure can raise data residency, regulatory, and operational control concerns. Second, latency: users or applications distant from available compute regions incur additional network delay before inference computation begins, degrading interactive service quality. Third, resource locality: centralised serving cannot always exploit compute capacity already available closer to demand, including within telecom infrastructure, even when

doing so could alleviate both of the constraints above.

1 BT Group 2 University of Bristol. Correspondence to: Juan Parra-Ullauri <juan.m.parraullauri@bt.com>.

Figure 1. (a) InferSlice deployed on JOINER PoPs with varied network and compute resources. SLO tiers are slices of network, GPU and AI configuration. (b) Tensor, Pipeline, Data Parallelism (TP, PP, DP) and Prefill/Decode Disaggregation (P/D-D) placed across PoPs.

This is a paper preprint. The final version may differ. The reported results and conclusions will not change. a The State of Sovereign AI Adoption in 2026, IDC InfoBrief.

Telco operators offer a compelling substrate for addressing these limitations. Their geographically distributed compute

InferSlice: SLO-driven LLM Inference over a Telco WAN

and network infrastructure, already deployed and operated under national regulatory and security frameworks, provides a foundation for serving AI workloads closer to where users and data originate (Taleb et al., 2017). Distributed inference across operator-controlled sites can provide shorter network paths, jurisdictional control, and greater operational control and predictability over the network and compute resources on which services depend. This opportunity extends the role of telecommunications infrastructure beyond connectivity toward a distributed platform for AI services. However, existing LLM serving frameworks are predominantly designed for tightly coupled data-centre environments (Yu et al., 2022; Zhong et al., 2024), where compute resources are connected by high-bandwidth, low-latency interconnects and can typically be managed as a closely coupled resource pool. Geographically distributed telco infrastructure introduces fundamentally different constraints, including heterogeneous accelerators, variable network latency and bandwidth, and resource availability that varies across sites. These constraints become particularly challenging when serving heterogeneous workloads with different service requirements. Interactive applications such as conversational assistants are sensitive to Time to First Token (TTFT) and Time per Output Token (TPOT), whereas batch workloads may prioritise sustained throughput and cost efficiency; other applications may require high availability or predictable tail latency, highlighting users and applications requiring different Service-Level Objective (SLO). Consequently, effective distributed serving requires more than replicating existing inference services across sites: inference configuration, resource allocation, and serving location must be selected according to application SLOs and the available compute and network resources. To address this gap, we present InferSlice, a reference architecture for SLO-driven distributed LLM inference over geographically distributed telco infrastructure. Led by British Telecom (BT) in collaboration with the University of Bristol and NVIDIA, InferSlice treats multiple telco compute sites as a shared inference substrate and provides the architectural mechanisms required to jointly manage application requirements, distributed compute resources, and network conditions. Rather than assuming a tightly coupled resource pool or a predetermined placement of inference functions, the architecture dynamically matches requests to serving sites and inference configurations according to each application’s SLO and the availability of compute and network resources. InferSlice is realised as a cloud-native platform that integrates distributed LLM serving, resource orchestration, network management, and end-to-end observability across multiple sites, extending service differentiation from network connectivity to AI inference itself.

We deploy InferSlice on the JOINER platform b , a UKWide Area Network (WAN) telco testbed spanning multiple cities in the UK, and evaluate its behaviour across both compute and network dimensions. Our experiments examine whether resource and scaling policies can enforce differentiated SLO outcomes across service tiers under increasing load, whether inference scaling mechanisms can reduce scale-out latency, and how wide-area network latency and bandwidth constrain distributed model parallelism and disaggregation across sites (Figure 1). All experiments are conducted over a shared WAN, without dedicated AI interconnects or RDMA between sites, reflecting real-world conditions in general-purpose telco infrastructure rather than specialised AI-cluster fabrics. Together, these evaluations characterise the feasibility and operating boundaries of SLOdriven LLM inference over geographically distributed telco infrastructure and inform the design principles presented in this work.

2

BACKGROUND AND R ELATED W ORK

2.1

Distributed LLM Inference Fundamentals

Serving LLMs at scale relies on distribution strategies that can be broadly categorised into three largely composable dimensions: model parallelism, replication, and functional disaggregation. While these dimensions are not fully independent, disaggregation in particular alters the internal structure of what would otherwise constitute a single replica. Each addresses a distinct scaling question: model parallelism scales a single model instance across accelerators; replication scales the number of independent model instances; and functional disaggregation scales distinct stages of the inference lifecycle independently of one another. 2.1.1

Model parallelism.

When a model does not fit, or does not perform adequately, on a single accelerator, its computation may be partitioned across devices. Tensor Parallelism (TP) partitions individual layers’ weight matrices across accelerators, requiring frequent collective communication, such as all-reduce, allgather, or reduce-scatter, between participating accelerators. Pipeline Parallelism (PP) instead partitions the model by layer across devices, exchanging intermediate activations between pipeline stages and trading reduced synchronisation frequency for pipeline bubble overhead. Expert parallelism (EP) shards mixture-of-experts layers across devices, routing tokens to the accelerators holding the relevant experts. Each of these strategies imposes synchronisation between participating devices at a frequency that typically confines b

JOINER (Joint Open Infrastructure for Networks Research), an international-scale open experimentation platform for future networks research and innovation.

InferSlice: SLO-driven LLM Inference over a Telco WAN

them to low-latency, high-bandwidth interconnect domains such as NVLink or InfiniBand; the binding constraint is synchronisation frequency rather than node or site boundaries per se. 2.1.2

Replication.

Often referred to as Data Parallelism (DP) in inference serving, replication deploys complete, independent copies of a model and distributes incoming requests across them, rather than partitioning a single model instance. Because replicas require no inference-time synchronisation with one another, replication tolerates substantially higher latency and lower-bandwidth interconnects than TP, PP, or EP, and is in principle viable across racks, sites, or wide-area network links. Replication therefore provides a natural mechanism for scaling capacity and elasticity: replicas can be added or removed independently as demand changes, while requests can be routed according to load, locality, or SLO requirements. This independence also provides fault isolation, as congestion or failure affecting one replica need not directly stall others. 2.1.3

Functional disaggregation.

A further scaling dimension separates distinct functions of the inference service into independently provisioned components. The most common example is prefill-decode disaggregation, where the compute-bound prefill phase, responsible for processing the input prompt, is separated from the memory-bandwidth-bound decode phase that generates output tokens autoregressively. These services are connected through a Key-Value (KV) cache transfer mechanism (Zhong et al., 2024). A related form of functional disaggregation externalises KV-cache management itself, allowing KV states to be offloaded, persisted, shared, or reused across inference workers. Such decoupling enables resources for different functions to be scaled according to their distinct computational characteristics, at the expense of additional coordination overhead and network dependencies whose feasibility depends on KV-cache size and available interconnect bandwidth. These dimensions compose: a deployment may apply TP within a node, PP across nodes within a site, replication across sites, and, orthogonally, disaggregate prefill from decode within each replica. Table 1 summarises the communication requirements imposed by each mechanism and whether the resulting cost is fixed by the serving configuration or varies with the workload. The present work explores model parallelism (TP and PP), replication (DP), and functional disaggregation (P/D-D) in a geographically distributed WAN.

2.2

SLO-Aware LLM Serving

As LLM inference has moved from research prototypes to production services, SLOs have become a primary means of expressing serving quality from the user’s perspective. Unlike traditional request response services, LLM requests are generated incrementally: a prompt is first processed to produce the initial output token (i.e., Prefill), after which subsequent tokens are generated autoregressively (i.e., Decode). This two phase structure gives rise to distinct latency characteristics and consequently to two commonly used serving metrics. TTFT measures the elapsed time from request arrival until the first output token is generated, and is strongly influenced by queueing, scheduling, and the compute intensive prefill phase. TPOT measures the latency between subsequent output tokens and is strongly influenced by the memory bandwidth and batch characteristics of the decode phase (Huang et al., 2025). An interactive application such as a chatbot is sensitive to both: a high TTFT is perceived as an unresponsive system, while a high TPOT reduces the rate at which generated content becomes available to the user. Other workloads may weight these objectives differently or instead impose end to end latency or throughput targets where token level responsiveness is less important, such as batch or offline processing. An SLO specifies a target on one or more such indicators, and SLO attainment, the proportion of requests meeting their assigned target, is related to but distinct from goodput, the rate of requests completed within their SLO per unit time (Wang et al., 2024): maximising throughput or minimising average latency does not necessarily maximise the number of requests that satisfy their individual SLOs, particularly once queueing is taken into account. This distinction is sharpened under heterogeneous SLOs, where a single deployment serves multiple applications or service classes concurrently, each with different TTFT, TPOT, or latency requirements, and a request with a relaxed target can absorb delay that would cause a stricter request to violate its own; recent SLO-aware serving systems therefore treat SLO requirements as explicit inputs to scheduling rather than treating all requests as interchangeable (Hong et al., 2025; Tang et al., 2025). For geographically distributed inference, this introduces a further dimension: where a request is served becomes part of the scheduling decision, since network latency, queueing conditions, and available capacity all vary by site. For InferSlice, SLO-awareness therefore provides the basis for treating a geographically distributed infrastructure as a shared resource pool, where placement is driven not only by proximity or utilisation but by which candidate site can satisfy a request’s SLO.

InferSlice: SLO-driven LLM Inference over a Telco WAN Table 1. Communication characteristics of distributed LLM inference mechanisms. Communication cost is either serving-dependent, fixed once the model and parallelism configuration are chosen, or workload-dependent, varying per request with prompt and output characteristics. Scaling mechanism Partition Communication requirement Serving vs. workload dependency

2.3

Tensor parallelism

Model (spatial)

Frequent fine-grained collectives, per layer per token

Mostly serving-dependent, as per-token volume is fixed by hidden dimension and TP degree; total volume grows with tokens generated

Pipeline parallelism

Model (spatial)

Activation transfer at stage boundaries, per engine step

Both, as activation size is servingdependent and burst frequency follows batch occupancy

Expert parallelism

Model (spatial)

Token routing and expert all-to-all, per layer per token

Both, as per-token volume is servingdependent and expert load is workloaddependent

Replication

None (full copy)

No inference-time sync between replicas, only load balancing

Neither, as there is no inter-replica traffic on the critical path

Prefill-to-decode disaggregation

Lifecycle (temporal)

Bulk KV-cache transfer per request, Both, as volume scales with prompt length plus handover coordination (workload) and per-token KV footprint (serving)

Telco Infrastructure as an AI Platform

Telecommunications networks provide a mature foundation for differentiated service delivery. Network slicing enables multiple logical services to share physical infrastructure while receiving differentiated resource allocations and performance objectives (Alam et al., 2025). However, these objectives have historically focused primarily on networklevel characteristics such as bandwidth, latency, reliability, and isolation, rather than the application-level compute requirements of AI services. Liu et al. extend this principle directly to LLM workloads through LLM-Slice, which creates dedicated wireless network slices for LLM services and binds communication resources to service requests (Liu et al., 2024). This demonstrates how service differentiation can be applied to the network delivery of LLM services, although the work focuses on communication resource control rather than the placement and orchestration of inference computation. The evolution toward Multi-access Edge Computing (MEC) further extended the role of telcos from connectivity toward distributed compute, placing resources closer to users and network access points (Taleb et al., 2017). While the broader MEC vision faced challenges including fragmented infrastructure and limited demand for generic edge workloads, the emergence of telco cloud and AI-RAN, wherein AI and RAN workloads can run concurrently on shared, GPU-accelerated infrastructure, is renewing interest in operator infrastructure as a platform for AI (Kundu et al., 2025). AI-RAN primarily targets the co-location of AI and RAN workloads at the radio edge, with its broader commercial opportunity depending on the emergence of latency-critical, edge-native applications such as physical AI, robotics, and autonomous systems.

LLM inference presents a complementary opportunity that does not require compute to be placed at the radio edge. Its service quality depends jointly on compute capacity, network conditions, queueing, and workload characteristics, while the geographical distribution and operational control of telco infrastructure can provide properties that are difficult to obtain from centralised cloud platforms: data and service sovereignty, stronger security and governance, lower latency through proximity to users, and greater predictability and reliability through controlled network and compute resources. This creates an opportunity to use existing telco compute sites, including central offices, regional data centres, and core network locations, as a distributed substrate for AI serving. In this setting, the service-differentiation principles long applied to network connectivity can be extended to AI services themselves, with application-level SLOs guiding the allocation and placement of distributed inference resources. 2.4

Related Work

Recent production systems have demonstrated increasingly sophisticated approaches to LLM serving under heterogeneous workloads and SLO requirements. Meta’s large-scale study of Llama inference explores millions of deployment configurations spanning heterogeneous accelerators, tensor, pipeline, expert, context, and data parallelism, as well as continuous batching and prefill-decode disaggregation, to maximise throughput while meeting latency SLOs (Cho et al., 2026). Similarly, Mitra et al. systematically evaluate disaggregated inference across hundreds of thousands of configurations, showing that its benefits depend strongly on workload characteristics, model size, dynamic rate match-

InferSlice: SLO-driven LLM Inference over a Telco WAN

ing, and elastic scaling (Mitra et al., 2026). Behdin et al. further demonstrate the scale and workload-specific optimisation required for industrial LLM serving, achieving substantial throughput improvements through the joint optimisation of models and serving infrastructure (Behdin et al., 2026). These works demonstrate the importance of jointly considering inference configuration, workload characteristics, and SLOs, but primarily target tightly coupled compute infrastructures. Meta’s study is particularly representative of this setting: although it considers diverse hardware and deployment configurations, the optimisation problem is formulated within the infrastructure available to a large-scale serving platform rather than across geographically distributed telco sites (Cho et al., 2026). More closely related to telecommunications, Yet et al. evaluate SLA-aware LLM and VLM inference across device, RAN-edge, and cloud tiers using a 5G Standalone AI-RAN testbed (Yet et al., 2026). Their experiments demonstrate the impact of execution location, model configuration, queueing, and network delay on SLA attainment, but use a fixed baseline policy and a hierarchical device-to-RAN-to-cloud topology rather than a distributed pool of independently provisioned serving sites. Wang et al. propose a wide-area distributed inference network that separates prefill at edge nodes from decode in a central cloud, with the two stages coordinated over a wide-area network (Wang et al., 2026). This work demonstrates the feasibility of LLM inference across wide-area telco infrastructure and is particularly relevant to the present setting, but its deployment follows a predetermined functional placement rather than dynamically selecting among multiple sites and inference configurations according to application-level SLOs. GORGO further considers network-aware LLM serving across geographically distributed inference replicas, jointly accounting for network latency, queueing, and KV-cache locality when routing requests to minimise TTFT and end-to-end latency (Toniolo et al., 2026). While this work demonstrates the importance of network conditions in cross-region inference decisions, it operates primarily at the request-routing layer and assumes an existing set of serving replicas, rather than jointly orchestrating compute resources, inference configurations, serving locations, and network resources across distributed sites. Petals demonstrates that very large LLMs can be executed across Internet-connected, heterogeneous, volunteer resources using fault-tolerant layer partitioning, routing, and dynamic load balancing across unreliable peers (Borzunov et al., 2023). While this work establishes the feasibility of geographically distributed inference over the public Internet, its objective is collaborative execution across unmanaged resources. In contrast, InferSlice considers operatorcontrolled telco infrastructure and investigates how networkaware service differentiation, resource slicing, and SLOdriven placement interact with multiple inference strategies

(TP, PP, DP and P/D-D) over a WAN. To the best of our knowledge, there is no existing reference architecture that brings these dimensions together by treating geographically distributed telco compute sites as a shared substrate for SLO-driven LLM inference, with serving location and inference configuration jointly selected according to application requirements, site-level resource availability, and network conditions. This motivates the reference architecture developed in this work.

3

S YSTEM D ESIGN

3.1

Physical Infrastructure Layer

For the development of InferSlice, we assume a telecomnative infrastructure as a set of geographically distributed edge computing sites, as discussed in Section 2.3. The physical infrastructure layer, represented at the bottom of Figure 2, consists of a set of compute hosts, each providing CPU, memory, storage, and hardware acceleration, both GPU and Network Interface Card (NIC). These hosts are interconnected by a WAN and together form edge computing Points of Presence (PoPs), which allow InferSlice to optimise the placement of the inference workload and reduce latency and network overhead (for example, serving a user from the PoP nearest to them). In the deployed system, the physical layer is modelled after the JOINER federated testbed where each PoP comprises Dell R660/R760 host machines with up to 64 physical CPU cores, 512 GB of RAM and an NVIDIA L40S GPU accelerator. Each host’s NIC is connected directly to the PoP’s edge router (Juniper ACX7100) via 10 Gbps SFP optics. Since the JOINER platform itself is multi-tenant, compute resources are exposed as Virtual Machines (VMs) to InferSlice. InferSlice’s storage is based on the general-purpose disks of the compute nodes, which the virtualisation layer pools into shared volumes. Further details of the testbed, including the host hardware each node carries, are given in the Annex A. 3.2

Virtualisation Layer

To enable scalability and configurability for modern AI workloads, a virtualisation layer is necessary. The resources of the testbed’s tenants are isolated from one another using JOINER’s virtualisation capabilities, implemented on top of VMware’s hypervisor and automation platform. Similarly, the physical network connectivity is abstracted as a Layer-2 Virtual Private Network (L2-VPN) underlay implemented by the testbed’s Software-Defined Wide Area Network (SD-WAN), with Maximum Transmission Unit (MTU) configured to 9000 to support the so-called jumbo frames and reduce the network overhead. As a result, the PoPs attach to a mesh network, exposed as a single “log-

InferSlice: SLO-driven LLM Inference over a Telco WAN

Client Request

OpenAI-Compatible API

Inference Layer: vLLM Scheduler

InferSlice Control Plane

KV Cache Manager (PagedAttention)

Prefill/Decode Disaggregation

Continuous Batching

Distributed Service Layer: Ray Planning & Optimisation

Autoscaler

Ray Head

HTTP router & Replica Router

Network & Compute Orchestration Run:ai Operator

LLM Parallelism Management

3 replicas 1 replicas

Resource Orchestration Layer: Run:ai, KubeRay Free Slice Standard Slice Compute Compute

Control Plane Geneve Tunnels

Distributed KV Caching

6 replicas

Object Store

Deployment Graph & Model Registry

KubeRay Operator

Traffic Forecasting

Ray Workers

0.5 GPU

1.5 GPU

3.0 GPU

4 CPU cores

8 CPU cores

16 CPU cores

Bandwidth Cap Policies

Metrics TTFT E2E Latency Queue Size/Time

Premium Slice Compute

KV Cache Hit rate Prefix/Decode stats Tokens/s

Network Management Layer: KubeOvn Free Slice Standard Slice Network Network

OVS Data Plane

Observability

Tensor/Pipeline/Data Parelillism

Bandwidth Cap Policies

Premium Slice Network

Network stats

Bandwidth Cap Policies

GPU & Compute stats Frontend

Virtualisation Layer: VMs, Kubernetes, Containers Semantic Routing

React / Grafana

Kubernetes Control Plane

Container Runtime

Storage and CN Interfaces

Master & Worker Nodes

GPU Device Management

Hypervisor

Prompt Caching

Data Sources Prometheus Scraper Run:ai API

Infrastructure Custom Scrapers Policy Enforcement Traffic Generator API Compute

GPU Pool

Slices

Tiers

Premium

Networking

Storage

Standard

Free

Figure 2. InferSlice reference architecture. Geographically distributed edge Points of Presence (PoPs) provide GPU-accelerated compute interconnected over a WAN, and are virtualised and containerised to give isolated, portable execution environments. Network slicing maps each service tier (Free, Standard, Premium) to its own logical network with differentiated QoS, while compute orchestration assigns each tier a guaranteed share of GPU and CPU capacity. A distributed serving layer expresses workloads as models and replicas and scales them within each tier’s resource envelope, and an inference optimisation layer manages model execution on the accelerator, including batching and KV-cache management. A control plane for planning, traffic forecasting, semantic routing, and prompt and KV-cache reuse configures these layers against per-tier SLOs, using cross-layer telemetry from the observability layer.

InferSlice: SLO-driven LLM Inference over a Telco WAN

ical” data-link. In addition to the base virtualisation, InferSlice further subdivides compute and network resources by employing a uniform containerised execution environment. Each inference service is packaged as a small group of container images, designed to be self-sufficient and to be deployed easily on any VM. Since containerisation bundles the required libraries into the image itself, and any auxiliary services (such as databases and proxy-servers) are deployed as supporting containers, this technique avoids dependency conflicts. Moreover, because containers share the host kernel, they are a lightweight form of virtualisation: workloads deploy faster, and scaling out is easily achievable by starting more copies of a container. The containerisation is achieved by implementing a Kubernetes cluster. We use Rancher’s lightweight Kubernetes distribution (RKE2) across all PoPs in the testbed. Containers run on the containerd runtime and GPU acceleration is exposed to the scheduler by the NVIDIA GPU Operator together with the NVIDIA Container Toolkit c . This means that each L40S GPU accelerator exposed to the platform can be shared among multiple containers. Persistent storage is provided by two complementary technologies built on top of the underlying node storage: Longhornd supplies block volumes for caches and application data, while model weights can optionally be served by on-demand streaming from MinIOe object storage. 3.3

Network Isolation and Slicing Layer

Network slicing logically partitions a shared physical network into multiple isolated networks (i.e., network slices), each of which can be associated with its own connectivity, resource allocation, and Quality of Service (QoS) policies (Wu et al., 2025a). Such isolated network slices enable service differentiation by allowing network and compute resources to be allocated and enforced independently across service tiers. This capability is particularly important for telecom operators deploying LLM inference over existing distributed infrastructure: heterogeneous inference services with different latency and throughput requirements (Wu et al., 2025b) must coexist across the same PoPs and over the shared WAN fabric, while retaining differentiated QoS performance. Rather than provisioning separate physical network infrastructure for each inference service with a specific QoS requirement, slicing allows an operator to expose differentiated service tiers over a common infrastructure with service-specific QoS differentiations in a dynamic, on-demand manner. In InferSlice, we adopt this operatororiented abstraction to couple network-level service differenc

https://docs.nvidia.com/datacenter/ cloud-native/container-toolkit/latest/index. html d https://longhorn.io/ e https://www.min.io/

+-----------------------------------------+ | Underlay Ethernet: physical node MACs | +-----------------------------------------+ | Underlay IP: tunnel endpoint IPs | +-----------------------------------------+ | UDP | +-----------------------------------------+ | Geneve: VNI + OVN metadata | +-----------------------------------------+ | Overlay Ethernet: logical workload MACs | +-----------------------------------------+ | Overlay IP: workload/slice IPs | +-----------------------------------------+ | TCP / UDP / SCTP / ... | +-----------------------------------------+

Figure 3. Simplified Geneve packet encapsulation in the KubeOVN overlay network.

tiation with the corresponding compute resources assigned to each inference tier. Network isolation is implemented using Kube-OVN (KubeOVN, 2026), an OVN-based Container Network Interface (CNI) plugin for Kubernetes. InferSlice maps each service tier—Free, Standard, and Premium—to a separate logical network (a subnet). This provides each tier with an independent network segment over which connectivity and QoS policies can be enforced. In our implementation, per-tier bandwidth limits and traffic policies are enforced at the OVS dataplane, allowing network resources to be differentiated without requiring physically separate network infrastructure. Connectivity between workloads belonging to the same logical network is provided by the Kube-OVN overlay network. When communicating workloads reside on different nodes, Kube-OVN encapsulates the logical network packets using Geneve tunnels and transports them over the physical underlay network (the JOINER infrastructure network). The Geneve encapsulation preserves the original Ethernet and IP headers of the logical network while adding the tunnel endpoint addresses required for communication between the corresponding Kubernetes nodes. The preservation of the inner network headers is particularly useful for QoS enforcement in our architecture. Since each isolated network slice is associated with a distinct IP address space (i.e., subnet), packets can be classified according to their inner source and destination addresses and mapped to the corresponding slice QoS policy. A simplified Geneve packet encapsulation in our architecture is illustrated in Fig. 3. This layer ensures that traffic from different service tiers is isolated at the network level, preventing cross-tenant interference and enabling per-tier QoS guarantees. The shared JOINER subnet represents a current contention point under high-load scenarios, which we discuss in Section 6.

InferSlice: SLO-driven LLM Inference over a Telco WAN

3.4

Compute Orchestration and Resource Scheduling Layer

The compute orchestration and resource scheduling layer manages how the shared accelerator pool is partitioned and allocated across service tiers and distributed inference workloads. Network isolation alone cannot guarantee service differentiation, since inference QoS ultimately depends on the compute capacity available to each workload. The layer therefore provides compute isolation and resource guarantees by assigning each tier a defined share of GPU and CPU capacity, together with quotas and scheduling priorities, while allowing resources to be shared across tiers. This is particularly important for heterogeneous service requirements: lower-priority workloads may benefit from fractional GPU allocation, whereas higher-priority tiers require sufficient reserved capacity to maintain their SLOs under contention. Together with the network management layer, the compute layer establishes a coordinated resource envelope in which GPU, CPU and network capacity are allocated according to the requirements of each service tier. In InferSlice, this functionality is implemented using NVIDIA Run:ai (Run:ai) and KubeRay (Ray Project), which provide complementary resource management and distributed workload orchestration capabilities within Kubernetes. Each service tier is mapped to a Run:ai project defining its GPU and CPU resource allocations, quotas and scheduling priorities, as shown in Fig. 2. Run:ai schedules workloads against these resource constraints and supports fractional GPU allocation, allowing multiple inference workloads to share the available accelerator pool while enforcing the resource boundaries of each tier. This enables, for example, the Free, Standard and Premium slices to use different GPU shares without requiring dedicated hardware for each tier. KubeRay manages the lifecycle of the distributed inference deployments that consume these resources. Declarative Ray-based deployment specifications are reconciled into the corresponding Kubernetes pods, which are subsequently scheduled within the compute resources assigned to their service tier. Replica counts are constrained by the available slice capacity, preventing scaling in one tier from consuming resources guaranteed to another. GPU and compute utilisation, scheduling state and workload status are exposed through the observability layer, providing the measurements required by the InferSlice control plane to monitor resource availability and, ultimately, support higher-level planning, optimisation and SLO-aware orchestration. 3.5

Distributed Serving and Replica Management Layer

The distributed serving and replica management layer acts as middleware between the inference workload and the vir-

tualised resources managed beneath it. Where the orchestration layer manages and schedules resources, this layer exposes them to the application through a distributed computing abstraction, so that the workload is expressed in application terms (e.g., models, replicas, and per-replica resource requirements) rather than in terms of nodes, devices, and pods. It translates these application-level requirements into resource requests the orchestration layer can satisfy and coordinates the resulting distributed components at runtime. This separation is needed because inference workloads must scale and recover independently of where their resources are physically located. Without such a layer, each workload would need to be tied to specific infrastructure, preventing the same resource pool from being shared across tiers and applications. The layer therefore provides a distributed runtime for the inference workload and manages the number and lifecycle of its replicas, while remaining agnostic to the underlying infrastructure. In InferSlice, this functionality is implemented using Ray (Moritz et al., 2018), deployed on Kubernetes through KubeRay as described in Section 3.4. Ray provides a distributed runtime in which a head node coordinates the cluster, worker nodes host the workload, and a shared object store supports data exchange between distributed components, as shown in Fig. 2. Each model deployment is described in a deployment graph and tracked in a model registry, together with the resources each of its replicas requires. These application-level specifications are mapped to resource requests submitted to the underlying Kubernetes and Run:ai layers, which place the corresponding pods within the compute slice of the tier. The Ray autoscaler adjusts the number of replicas of each deployment in response to serving demand, subject to minimum and maximum bounds and to the capacity of the tier’s compute slice, and issues the corresponding resource requests when replicas are added or removed. 3.6

Inference Optimisation Layer

The inference optimisation layer manages the lifecycle and execution of the model within each serving replica, from model loading and initialisation to GPU-resident execution and exposure through the inference API. Whereas the distributed serving layer manages replicas as application-level units, this layer determines how the model is instantiated and operated within each replica, including model loading, GPU memory allocation, request scheduling, batching, and KV-cache management. It therefore provides the abstraction between a model deployment and the underlying accelerator, hiding device-level execution details from the serving and orchestration layers. This separation is important because the same model can require different execution strategies depending on available GPU memory, workload characteristics, concurrency, and SLO requirements. The layer

InferSlice: SLO-driven LLM Inference over a Telco WAN

converts a model from a deployment artefact into a running inference service while exposing a consistent interface to the distributed serving layer. In InferSlice, this functionality is implemented using vLLM (Kwon et al., 2023) as the inference runtime. For each replica, vLLM loads the model weights, initialises the model on the allocated GPU resources, manages GPU memory during execution, and exposes an OpenAI-compatible inference interface to the serving layer. The runtime scheduler coordinates concurrent requests through continuous batching, while the KV-cache manager allocates and reuses attention states during autoregressive generation. These mechanisms allow multiple requests to share the resources of a replica while efficiently managing GPU memory and compute across concurrent requests. The layer also provides the mechanisms required for distributed inference, including tensor, pipeline, and data parallelism and prefill/decode disaggregation (Zhong et al., 2024), allowing the model to be partitioned or replicated according to the configuration specified in its deployment. 3.7

ploit reuse across requests and replicas, reducing redundant computation and the cost of transferring state across the network. Together, these components provide the control logic required to coordinate the underlying compute, network, serving, and inference layers. The control plane operates over the state exposed by the observability layer and translates this state into configuration changes applied to the underlying infrastructure. Resource utilisation, request rates, queueing behaviour, latency, throughput, network conditions, replica state, and KV-cache residency provide the feedback required to identify SLO violations and available capacity. The resulting decisions are then enforced through the compute orchestration, network slicing, distributed serving, and inference optimisation layers, closing the loop between observed system state and service configuration. This architecture separates policy and optimisation from resource enforcement, allowing the same control plane to reason over heterogeneous infrastructure and different inference configurations while leaving the underlying layers responsible for their respective execution mechanisms.

Control Plane 3.8

Observability Layer

The InferSlice control plane provides the decision-making layer that coordinates compute, network, serving, and inference resources to meet the SLOs of each service tier. Whereas the underlying layers execute resource allocation, replica management, and model inference, the control plane determines how these resources and execution mechanisms should be configured and where workloads should be served. It maintains a global view of workload demand, service requirements, resource availability, network conditions, and observed system performance, and uses this information to determine the serving configuration for each tier. This separation is essential in a distributed operator environment, where compute and network resources span multiple sites and the configuration that satisfies an SLO can change as demand and infrastructure conditions evolve. The control plane therefore provides the coordination mechanism through which local resource management is transformed into end-to-end, SLO-aware service orchestration.

The observability layer provides the measurements required to understand the state and performance of the distributed inference system across all infrastructure and service layers. Whereas the control plane determines how the system should be configured, the observability layer provides the telemetry required to make those decisions based on current and historical system state. It therefore collects and aggregates metrics across compute, network, serving, and inference components, providing a consistent view of resource utilisation, workload behaviour, and SLO attainment. This cross-layer visibility is particularly important in a distributed environment, where performance degradation may originate from compute contention, network congestion, replica saturation, or inference-level effects such as KV-cache pressure. The layer consequently provides the measurement and feedback mechanism that connects the execution layers with the control plane.

In InferSlice, the control plane comprises planning and optimisation, traffic forecasting, semantic routing, prompt caching, and distributed KV-cache management functions, as shown in Fig. 2. The planning and optimisation component determines the required serving configuration based on the SLOs and available resources, including replica counts, serving locations, resource allocations, and inference configurations. Traffic forecasting provides estimates of future demand to support proactive scaling and placement decisions, while semantic routing directs requests towards replicas according to application and service requirements. Prompt caching and distributed KV-cache management further ex-

In InferSlice, observability spans resource, network, serving, and inference telemetry, including GPU and CPU utilisation, allocated resources, network throughput and latency, request rates, queueing, TTFT, TPOT, token throughput, replica state, and KV-cache utilisation and hit rate, as shown in Fig. 2. Telemetry is gathered by a combination of standard and custom collectors. Prometheus scrapers collect metrics exposed by the Kubernetes, serving, and inference components, the Run:ai API provides GPU allocation and scheduling state for each tier, and custom scrapers collect measurements not available from standard exporters, such as network statistics and the request-level metrics reported by

InferSlice: SLO-driven LLM Inference over a Telco WAN

the traffic generator. Prometheus stores and aggregates these measurements as time series, and Grafana and a React frontend provide the interfaces through which they are queried and visualised. InferSlice’s React frontend is described in the Appendix Section B. The resulting telemetry supports both runtime monitoring and offline evaluation: current measurements allow resource pressure and SLO violations to be identified, while historical measurements support traffic forecasting, capacity planning, and analysis of the operating boundaries of different inference configurations. A common observation layer across geographically distributed resources allows InferSlice to analyse end-to-end service behaviour, rather than treating compute, network, and inference performance as independent systems.

4

E XPERIMENTAL S ETUP

4.1

Testbed and Workload

To assess SLO-driven inference, we deployed InferSlice across a representative subset of six PoPs from the JOINER federated testbed, selected to capture variation in inter-PoP distance. Table 2 lists the VMs at each site; six of them carry a single NVIDIA L40S accelerator, attached by PCI passthrough, with 48 GB of GPU memory and 864 GB/s of memory bandwidth. The VMs run Ubuntu 24.04 LTS on Intel Xeon Gold 6438Y+ hosts, provisioned with 16 vCPUs, 62 GiB of RAM and roughly 500 GB of local disk, and running the software stack summarised in Table 3. Each service tier is served by a set of vLLM replicas deployed as Ray services and orchestrated by KubeRay (Section 3.4), and accessed through vLLM’s standard OpenAI-compatible interface. In turn, each replica is configured to reserve up to 92% of the GPU memory granted to it by Run:ai, which is shared between the model and its KV cache. Model weights are staged in a distributed MinIO pool and streamed to replicas on demand. VMs without a GPU host the cluster’s control plane and auxiliary functions, such as monitoring, API servers, persistent storage and reverse proxying, as well as traffic generators for collecting experimental performance data. The traffic exchanged between distributed replicas (for example, KV-cache and collective transfers across PoP nodes) is carried over the standard TCP/IP network. Client requests are received through a standard OpenAIcompatible interface. To perform the initial functional validation and to emulate the workload, we use the vLLM benchmark CLI which sends independent, short-lived, stateless chat requests drawn from the ShareGPT dataset with a Poisson arrival process. The tool supports this dataset natively, and the Poisson process gives fine-grained control over the arrival pattern (vLLM, 2026). For the quantitative experiments, we set the arrival rate to 100 requests/s and

Table 2. PoP and computing capabilities used in the evaluation

PoP

VM

GPU

Bristol

VM1 VM2

— —

Southampton (Soton)

VM1 VM2

L40S L40S

Oxford

VM1 VM2

— L40S

London (UCL)

VM1 VM2

— L40S

Belfast (QUB)

VM1 VM2

— L40S

Dublin (TCD)

VM1

L40S

Table 3. Software stack supporting LLM inferece.

Component

Version

Kubernetes (RKE2) Container runtime (containerd) Kube-OVN NVIDIA GPU Operator NVIDIA driver CUDA NVIDIA Container Toolkit NCCL vLLM Ray KubeRay Longhorn MinIO server MinIO operator Run:ai

1.34.3 2.1.5 1.14.15 25.10.1 580.126.09 13.0 1.18.1 2.27.5 0.12.0 2.53.0 1.2.2 1.10.1 R.2025-04-08 7.1.1 2.24.11

employed a varying number of parallel ongoing requests between 2 and 32. The evaluation comprises three types of experiments: a service-tier experiment, which varies load and concurrency across the Free, Standard and Premium tiers (Section 4.3.1), elastic scale-out effects and optimisation (Section 4.3.2), and a distributed-inference experiment, in which the model is partitioned across PoP nodes (Section 4.3.3). We served each type with a different model. Llama-3.1-8B-Instruct is a dense 8B-parameter, instruction-tuned conversational model; since it fits entirely on a single L40S without need for partitioning, we used it for the service-tier experiments. GPT-OSS-20B, on the other hand, is a sparse mixture-of-experts model of 21B parameters (3.6B active) with a 128k context, designed for

InferSlice: SLO-driven LLM Inference over a Telco WAN

low-latency reasoning and agentic workloads; when its full 21B parameters are loaded into memory, it is a good candidate for the distributed-inference experiments.

paths less suitable for tightly coupled synchronous operations and favouring pipeline configurations with less frequent inter-stage communication.

Throughout the experiments, we recorded the following metrics:

Distant PoPs (≥ 15 ms, n = 32): Distant PoPs (e.g., pairs between the 10.203.42.0/24 and 10.203.43.0/24 subnets) exhibit ICMP latencies of approximately 15 ms– 22 ms, with a mean of 18.97 ms, and mean TCP throughput of 1.30 Gbps. These paths are unsuitable for synchronous parallelism strategies and motivate asynchronous or speculative execution models for cross-PoP inference disaggregation.

• TTFT, the elapsed time from request arrival to the generation of the first output token; • TPOT, the latency between consecutive generated output tokens; • aggregate token throughput, the total number of tokens generated per second across all requests; • goodput, the rate in requests per second of requests that meet their TTFT SLO (here TTFT < 500 ms; the distributed-inference experiments of Section 5.2.1 relax this to 2 s, where per-request cross-PoP costs make the stricter threshold uninformative); • success rate, the proportion of requests completed without error. 4.2

Network Substrate Characterisation

With Inferslice deployed over the JOINER network, we carried out a network characterisation experiment. The 80 cross-PoP directed pairs measured reveal a four-tier latency structure that reflects the geographic distribution of the JOINER testbed, as summarised in Table 4: Near-adjacent PoPs (< 5 ms, n = 8): Near-adjacent PoPs (e.g., pairs between the 10.203.41.0/24 and 10.203.43.0/24 subnets) exhibit ICMP latencies of approximately 4.06 ms–4.51 ms with TCP throughput of 6.47 Gbps–6.59 Gbps (mean 6.53 Gbps). These PoP pairs are plausible candidates for inter-PoP pipeline parallelism, as the network overhead is commensurate with intra-server NVLink latency at scale. Mid cross-PoPs (5 ms–10 ms, n = 32): Mid crossPoPs (e.g., pairs between the 10.203.43.0/24 and 10.203.45.0/24 subnets) exhibit a mean ICMP latency of approximately 7.57 ms and mean TCP throughput of 2.79 Gbps. These paths introduce moderate communication overhead but remain suitable candidates for pipeline parallelism with sufficiently coarse-grained inter-stage communication. Far cross-PoPs (10 ms–15 ms, n = 8): Far crossPoPs (e.g., pairs between the 10.203.44.0/24 and 10.203.45.0/24 subnets) exhibit a mean ICMP latency of approximately 10.90 ms and mean TCP throughput of 1.19 Gbps. The increased latency and reduced throughput impose greater communication overhead, making these

In addition to the cross-PoP connections, there are 10 directed intra-PoP pairs (i.e., connections within the same JOINER PoP site), which exhibit a mean ICMP latency of approximately 0.60 ms and mean TCP throughput of approximately 10.0 Gbps. This four-tier cross-PoP structure has direct implications for InferSlice’s placement policy: tensor parallelism should be confined to intra-PoP pairs; pipeline parallelism is viable up to approximately 10 ms; and expert parallelism for mixtureof-experts models, which tolerates higher latency due to conditional routing sparsity, is the preferred parallelism strategy for distant cross-PoP pairs. Section 6 returns to these implications in the context of the inference results below. 4.3

Experiments Definition

4.3.1

Service Differentiation

Telco LLM deployments must support a heterogeneous mix of AI applications with fundamentally different SLOs. Conversational agents and voice assistants require low and predictable latency, with interactive LLM serving commonly targeting P99 TTFT below 500 ms and bounded token generation latency to maintain acceptable user experience and conversational flow (MLCommons, 2025). In contrast, network operations assistants analysing multimodal telemetry may prioritise service availability, reliability, and predictable response times, while large-scale workloads such as content moderation, document processing, or synthetic data generation are primarily constrained by throughput and cost efficiency. These workload classes impose conflicting requirements that cannot be optimally satisfied by a single inference operating point. Consequently, resource allocation, request admission, autoscaling behaviour, and network resource assignment must be differentiated according to application SLOs. Figure 4 illustrates this concept through three representative LLM service slices, each enforcing distinct QoS objectives while sharing the same underlying infrastructure. To evaluate whether InferSlice can deliver differentiated inference services over shared telco infrastructure, we con-

InferSlice: SLO-driven LLM Inference over a Telco WAN Table 4. JOINER testbed network characterisation summary across 90 directed node pairs, comprising 80 cross-PoP and 10 intra-PoP pairs. TCP throughput values are unidirectional (TX).

Latency tier

Pairs

Mean ICMP (ms)

Mean TCP TX (Gbps)

Near cross-PoP (< 5 ms) Mid cross-PoP (5 ms–10 ms) Far cross-PoP (10 ms–15 ms) Distant cross-PoP (≥ 15 ms)

8 32 8 32

4.26 7.57 10.90 18.97

6.53 2.79 1.19 1.30

Intra-PoP (same subnet)

10

0.60

10.0

Conversational LLM Slice App: Consumer Interactive P99 TFTT: < 500ms T: 10-50 TPS QoS: Low latency High Reliability LLM Slice App: Mission Critical TFTT: 500ms-1s T: 50-200 TPS QoS: 99.9% Availability Batch Processing LLM Slice App: Content Processing TFTT: Best Effort T: 500-2000 TPS QoS: High throughput

S1

S2

S3

Physical Infrastructure

Figure 4. Three LLM service slices over shared telco infrastructure, each enforcing distinct latency, throughput, and availability policies without dedicated hardware per tenant.

ducted a service differentiation experiment on the JOINER platform. Three slices, Free, Standard and Premium, were instantiated over the same physical GPU pool and differ in the resources they are entitled to (Figure 2). At the compute layer, Run:ai assigns each slice a fractional GPU project of 0.5, 1.5 and 3 GPUs, with 4, 8 and 16 CPU cores respectively, so that prioritisation is enforced without dedicating hardware to a tenant. At the inference layer, KubeRay autoscales each slice up to a ceiling of 1, 3 and 6 replicas, each replica occupying 0.5 GPU. Kube-OVN provides slice-level virtual networking and traffic isolation. All slices serve the same Llama-3.1-8B-Instruct deployment, so any difference in behaviour is attributable to the resources and scaling headroom of the slice rather than to the model or hardware. Load is generated from the ShareGPT dataset by sweeping the client max concurrency from 2 to 32 against each slice. For each setting we record P99 TTFT and SLO-compliant goodput, defined as the rate of requests completing with a TTFT of at most 500 ms. The goal

is to determine at which offered load each slice stops meeting its SLO, and whether the slices degrade in the order implied by their allocations. A second experiment, reported in Appendix C.1, holds the resource allocation fixed and instead differentiates slices through their concurrency policy (target ongoing requests and max ongoing requests) under an increasing request rate. 4.3.2

Cold-start latency

Elastic scale-out only helps if new replicas become ready to serve quickly, and for multi-billion-parameter models the time spent acquiring weights can account for much of the delay between a scaling decision and the first served response. To reduce this delay, we load model weights through a parallel streaming reader instead of the default memorymapped path. This experiment evaluates that design choice by quantifying how much cold-start latency parallel streaming removes relative to the default loader, both when the service is deployed from scratch (cold) and when it scales back up from its minimum replica count (hot). To isolate the cost of weight acquisition during elastic scale-out, we benchmark the premium-tier RayService serving Llama-3.1-8B-Instruct (bf16) under two weightloading variants that differ only in the loader. The default configuration memory-maps safetensors weights from an HF cache persisted on a Longhorn PVC, whereas the runai-streamer variant loads the same cache through the Run:ai Streamer parallel filesystem reader (load format: runai streamer). Each variant is evaluated in a cold phase, in which the RayService is deleted and re-applied from scratch while load is already running, and in a hot phase, in which load is resumed after the service has scaled back down to its minimum replica count. A fixed load of 50 concurrent streaming chat completions, sampled from the conversation dataset used by the production traffic generator and capped at 256 output tokens per request, maintains per-replica ongoing requests above the autoscaler target and thereby forces the addition of five replicas under load. Container images are pre-pulled on all GPU nodes so that image retrieval does not contaminate cold timings.

InferSlice: SLO-driven LLM Inference over a Telco WAN

We record the cold deploy time from manifest apply to serve readiness and, for each worker pod, the time-to-firstserved-response, identified as the first increment of the pod’s ray vllm request success total counter above its phase baseline in the cluster Prometheus at 1 s scrape resolution. 4.3.3

Network bottleneck identification

Experiment N1: End client bandwidth limiting. The preceding network characterisation establishes the latency and throughput available across the JOINER substrate. We next ask when network capacity, rather than inference compute or request queueing, becomes the binding constraint on serving performance. We isolate the client-facing link and apply per-slice bandwidth caps while holding the serving configuration fixed. The client is located at a fixed site and the bandwidth cap is varied independently for the Free, Standard, and Premium slices. For each configuration, we measure P99 TTFT, P99 TPOT, token throughput, and container-level receive and transmit rates. Experiments are repeated at client concurrency levels of 10, 16, and 32. The 100 Gbps configuration serves as an effectively unconstrained baseline; lower caps are selected around the observed traffic demand to expose the transition from computeor queue-limited to network-limited operation. Experiment N2: Distributed model latency characterisation (TP/PP across PoP nodes). We characterise how model parallelism behaves when the shards of a single replica are placed across PoP nodes. We fix the model to GPT-OSS-20B and deploy 2-shard tensor-parallel (tp2) and pipeline-parallel (pp2) configurations across measured PoP pairs at ∼9, ∼18 and ∼33 ms Round-Trip Time (RTT). A co-located single-node deployment (tp1) is the baseline, so that any change in behaviour is attributable to shard placement alone. Each arm uses the production serving stack, orchestrated by Ray Serve with a vLLM backend, and streams weights from MinIO; only the parallelism strategy and worker placement differ. Because the parallelism under test sits below the OpenAI API layer, load is generated directly against the vLLM engine with vLLM bench serve. To separate per-request TTFT from client-side queueing, each arm’s offered rate is raised from a low queue-free reference until the median TTFT departs from the per-request baseline or the P99 develops a queue tail; the highest rate at which the response-time distribution shows no onset of queueing is reported as the configuration’s maximum queue-free offered load, and the saturated operating point is reported separately in the text. We report median and P99 TTFT, output throughput and goodput (requests/s meeting TTFT <2 s; the 500 ms service SLO of Section 4.1 is relaxed to 2 s for these WAN experiments, where per-request cross-PoP costs make the stricter threshold uninformative), and measure per-shard

transmit and receive rates from Prometheus byte counters to characterise the inter-shard communication pattern of each strategy. A configuration is deemed feasible if it remains SLO-compliant, which identifies the RTT range over which each strategy can be distributed. Four-shard configurations (pp4, tp2 pp2, tp4, tp4 ep4) span several inter-node links with different RTTs, so their results cannot be attributed to a single RTT value; these configurations do not initialise within a 40-minute deployment budget and are excluded from the comparison. Experiment N3: Distributed model bandwidth limiting With the latency behaviour of each strategy established in Experiment N2, we hold RTT fixed at each pair’s natural value and sweep the available inter-shard bandwidth downward to find where bandwidth becomes the binding constraint. The deployment follows Experiment N2, so only the bandwidth between shards changes between runs. Because measured PoP pairs couple latency and bandwidth, we isolate bandwidth by rate-shaping alone: an HTB rate cap is applied to each shard node’s overlay egress towards its peer shard (in both directions, using destination-/32 filters on the Geneve overlay), which leaves the path’s RTT untouched, and each cap is verified on the dataplane before its run. We sweep pp2 on three pairs, VM2@UCL–VM1@Southampton (∼9 ms), VM2@UCL–VM2@QUB (∼18 ms) and VM2@QUB– VM1@TCD (∼33 ms), and tp2 once on VM2@UCL– VM1@Southampton. Because tp2 sustains only a small fraction of that load, it is swept under a queue-free tenprompt load as a latency-bound reference. A descending ladder of caps is applied to every pair, with additional coarser caps on the 9 ms pair, an unconstrained reference repeated after every sweep to detect drift, and a 1 Gbps null control on every pair. Metrics follow Experiment N2, augmented with per-shard byte counters to verify on the wire that each cap binds. We define the bandwidth floor as the cap at which median TTFT doubles relative to the pair’s own unconstrained reference. Experiment N4: Latency characterisation in Functional Disaggregation (P/D-D across PoP nodes). We repeat the setup of Experiment N2, but focus on functional disaggregation, splitting the prefill and decode phases of inference across the measured PoP pairs. We fix the model to Llama-3.1-8B-Instruct, which fits on a single node, so that any behaviour change is attributable to the placement of the two phases alone. Each deployment uses one prefill worker and one decode worker, orchestrated by Ray Serve with a vLLM backend, with the KV-cache transferred between workers over NIXL. The workers are placed across measured PoP pairs spanning the mid and distant cross-PoP

InferSlice: SLO-driven LLM Inference over a Telco WAN

tiers of Table 4, and an aggregated, co-located deployment on a single node is the baseline. Because the cost of disaggregation depends on the volume of KV-cache moved per request, we use the ShareGPT workload of Section 4.1 at an offered request rate of 2.7 requests/s with 100 prompts, so that the prompt-length distribution, and hence the transfer size, reflects interactive chat. This rate is the highest queuefree load tested on every placement; the rate was raised in steps until the onset of queueing appeared in the P99 tail. As in Experiment N2, load is generated with vLLM bench serve, and we report median/P99 TTFT, TPOT, output throughput and goodput (requests/s meeting TTFT <2 s, the relaxed threshold of Experiment N2), augmented with the measured per-pod KV-cache transfer volume and per-shard transfer rates. A placement is deemed feasible if it remains SLO-compliant, which identifies the RTT range, at each pair’s natural bandwidth, over which prefill/decode disaggregation can be distributed for this workload class.

5

E VALUATION

5.1

Service differentiation across slices

We first evaluate whether InferSlice can expose distinct service tiers from a shared infrastructure by differentiating the resources allocated to each slice. The Free, Standard, and Premium slices are allocated a maximum of 1, 3, and 6 replicas respectively. We sweep the client max concurrency from 2 to 32 and measure P99 TTFT and SLO-compliant goodput, defined as the rate of requests whose TTFT meets the 500 ms SLO. A complementary experiment in Appendix C.1 instead holds the resource allocation aside and differentiates slices through their concurrency policy. At low load (up to concurrency 12) the three slices are indistinguishable (Figure 5). P99 TTFT remains well below the SLO and goodput grows linearly with load to about 2.5 req/s, because every slice has sufficient capacity for the offered load. The slices diverge once the load exceeds their capacity. Free, limited to a single replica, breaches the SLO at concurrency 16: P99 TTFT jumps from about 0.1 s to about 5 s and reaches about 13 s at concurrency 32. Its goodput peaks at 2.49 req/s (462 tok/s) and then collapses to below 0.1 req/s as requests queue behind the saturated replica. Standard, with up to three replicas, sustains the SLO for longer and breaches at concurrency 20, where P99 TTFT reaches about 2.8 s and rises to about 8 s at concurrency 32. Its goodput peaks at 3.18 req/s (840 tok/s) at the breach point and falls to about 0.1 req/s at concurrencies 28 and 32. Premium, with up to six replicas, keeps P99 TTFT below the SLO across the sweep, apart from a transient excursion at concurrency 16 (about 0.8 s) that we attribute to a replica cold start. Its goodput grows with load to 5.36 req/s (1,026 tok/s) at concurrency 32, where the sweep ends before saturation, so this value is a lower bound.

At their peaks, Premium therefore delivers about 2.2× the compliant goodput of Free and about 1.7× that of Standard, while Standard delivers about 1.3× that of Free. Beyond its capacity a slice does not degrade gracefully: queueing inflates TTFT for most requests and goodput falls sharply, whereas a slice with sufficient resources continues to scale. These breach points define the compute-limited operating regimes that Experiment N1 builds on (Section 4.3.3). 5.2

Cold-Start Reduction

During elastic scale-out, weight acquisition dominates the time required to deploy the pod. With the default loader, a cold deployment of the premium service takes approximately 950 s from manifest apply to serve readiness, and pods added under load serve their first response after a median of roughly 560 s, yielding an end-to-end scale-out time of approximately 1,500 s. Replacing the default loader with the runai-streamer parallel filesystem reader over the same PVC cache reduces the cold deploy time to approximately 330 s, a reduction of about 65%, and the median per-pod time-to-first-response to roughly 285 s, approximately halving the end-to-end scale-out time to approximately 610 s. The improvement cannot be attributed to storage-cache warmth: hot-phase medians match the cold ones in both variants (roughly 580 s versus 560 s with the default loader, and approximately 290 s versus 285 s with runai-streamer), indicating that the loader, not the cache state, bounds weight acquisition. Newly added pods in fact reach readiness within approximately two minutes in both variants, so the observed differences arise almost entirely during weight loading and first-token production. These measurements explain the cold-start transient observed in the service-differentiation sweep (Section 5.1) and suggest that weight streaming can substantially shorten scale-out for latency-sensitive slices. Figure 6 visualises these dynamics per pod in the cold and hot phases. 5.2.1

Network bottleneck identification

Eval Experiment N1: End client bandwidth limiting Figure 7 summarises the sweep across the three client concurrency levels, and the full per-tier results are given in Tables 9 to 11 in Appendix C.2.1. At moderate load (concurrency 10), all three tiers are largely insensitive to bandwidth while the available capacity is at or above approximately 1 Mbps. Reducing the cap to 1 Mbps leaves throughput (369–380 tok/s) and P99 TPOT (26–37 ms) essentially unchanged, and P99 TTFT remains below the 500 ms SLO for every tier. Reducing the cap to 0.5 Mbps produces a clear network bottleneck: the measured receive rate saturates at approximately 0.46 Mbps, throughput falls to 217–224 tok/s, P99 TTFT rises to 1.9–3.3 s, and P99 TPOT rises to 150– 276 ms. The transition is therefore bracketed between 0.5 and 1 Mbps at this load. Above it, inference and scheduling

InferSlice: SLO-driven LLM Inference over a Telco WAN Premium

Standard

Free

(b) SLO-compliant goodput SLO-compliant goodput (req/s)

(a) P99 TTFT

P99 TTFT (ms)

104

cold start

103 500 ms TTFT SLO

102

Free breach

6

Standard breach

≥ 5.36 req/s

5 4 3 3.18 req/s

2 1

2.49 req/s

0 2

4

6

8 10 12

16

20

24

28

32

2

4

6

8 10 12

16

20

24

28

32

Client max_concurrency

Figure 5. Service differentiation across the Free, Standard, and Premium slices (maximum of 1, 3, and 6 replicas) as client max concurrency increases. (a) P99 TTFT against the 500 ms SLO (log scale). (b) SLO-compliant goodput, the rate of requests whose TTFT meets the SLO. Open rings mark each slice’s peak and dotted lines mark the concurrency at which Free and Standard first breach the SLO. The Premium excursion at concurrency 16 is a replica cold start, and its peak is a lower bound because the sweep ends before saturation.

dominate latency; below it, network throughput becomes the binding resource and degrades both first-token delivery and subsequent token generation. The bandwidth required to stay in the unconstrained regime grows with load. As shown in Figure 8, the unconstrained Premium receive rate rises from 0.91 Mbps at concurrency 10 to 1.32 Mbps at 16 and 2.45 Mbps at 32, and Premium stays within its SLO under caps placed just above these values (1, 1.5, and 2.5 Mbps respectively). At the tightest margin (concurrency 32, 2.5 Mbps cap, roughly 2% above demand), TTFT and TPOT are preserved but throughput is about 7% lower than the unconstrained baseline (934 versus 1001 tok/s), indicating that the link is close to becoming binding. In contrast, the 0.5 Mbps cap is identical at every concurrency level, so it exposes the collapse but does not locate the floor: at all three loads the receive rate is pinned at 0.45–0.47 Mbps, throughput falls to 217–231 tok/s, and at concurrency 32 Premium P99 TTFT reaches 12.8 s and P99 TPOT 1.19 s (Figure 7). Figure 7 also separates network-limited from compute- or queue-limited behaviour. At concurrency 16 and 32, Standard and Free already violate the TTFT SLO with an effectively unconstrained 100 Gbps link (marginally for Standard at 16, by a wide margin otherwise), while their TPOT remains close to the Premium baseline. Their high TTFT therefore cannot be attributed primarily to the network and

instead reflects queueing or compute-side saturation, which the 0.5 Mbps cap then compounds. Bandwidth effects can appear before the 0.5 Mbps floor when demand approaches the cap: Standard at concurrency 32 with a 1.5 Mbps cap (demand 1.44 Mbps) shows P99 TPOT rising from 41 to 136 ms and its transmit rate roughly halving, even though its TTFT is already dominated by queueing. Taken together, these results distinguish the network bottleneck from the previously observed queue boundary: queue saturation predominantly increases TTFT, whereas severe bandwidth saturation reduces measured link and token throughput and increases both TTFT and TPOT by an order of magnitude.

Eval Experiment N2: Distributed model latency characterisation (TP/PP across PoP nodes) Table 5 reports end-to-end serving latency when model shards span PoP nodes, each row measured at its maximum queue-free offered load. At this load PP meets the 500 ms SLO on every measured path: median TTFT rises monotonically with link RTT (85 ms co-located, then 150, 221 and 364 ms at 9, 18 and 33 ms) and P99 stays below 0.5 s. However, goodput degrades with distance. The maximum queue-free rate falls from 4.40 requests/s co-located to 2.60, 1.55 and 0.85 requests/s, which is 59%, 35% and 19% of the colocated rate, and output throughput falls from 697 to 403, 239 and 135 tokens/s. PP therefore satisfies the latency SLO but sustains progressively less load as RTT grows, so its fea-

InferSlice: SLO-driven LLM Inference over a Telco WAN Table 5. Experiment N2: GPT-OSS-20B serving latency and inter-shard communication vs. inter-node RTT at each configuration’s maximum queue-free offered load (the Rate column; raised until median TTFT departs from the per-request baseline). Goodput = requests/s meeting TTFT <2 s. Per-shard TX and RX are Prometheus byte-counter rates in Mbps over the bench window, mean (peak): for the PP rows they are the sending and receiving shard, for the TP rows the per-shard values. All windows are post-warm-up steady state. Config

Pair

RTT Rate Med. TTFT P99 TTFT tokens/s Goodput Direction Per-shard TX / RX

tp1 (intra) single node 0 ms 4.40 pp2 VM2@UCL–VM1@Soton 9 ms 2.60 pp2 VM2@UCL–VM2@QUB 18 ms 1.55 pp2 VM2@QUB–VM1@TCD 33 ms 0.85 tp2 VM2@UCL–VM1@Soton 9 ms 0.085 tp2 VM2@UCL–VM2@QUB 18 ms 0.09 tp2 VM2@QUB–VM1@TCD 33 ms 0.035

85 ms 150 ms 221 ms 364 ms 1.67 s 2.9 s 4.8 s

replicas serving replicas serving

6 5 4 3 2 1 0

default runai-streamer

serve up 328 s

0

400

serve up 952 s

800

1200

1600

time since manifest apply (s) (b) hot phase

default runai-streamer 0

200

400

697 403 239 135 5.5 3.1 1.8

4.40 local only 1.0 (2.6) / 0.4 (1.6) 2.60 one-way 28.7 (98) / 29.5 (54) 1.55 one-way 14.8 (39) / 13.4 (24) 0.85 one-way 19.0 (44) / 17.6 (25) 0.02 all-reduce 30 (54) / 29 (54) 0.00 all-reduce 18 (98) / 18 (99) 0.00 all-reduce 11 (19) / 11 (19)

sibility at a given distance is determined by the capacity it retains rather than by latency alone. TP collapses even at the lowest loads tested. Its maximum queue-free rate is 0.035– 0.09 requests/s and its median TTFT of 1.7–4.8 s exceeds the 500 ms SLO on every pair. At the relaxed 2 s threshold, goodput is zero on the 18 and 33 ms pairs, and the 9 ms pair is the boundary case. The per-layer all-reduce on the critical path is consistent with this cost. EP is likewise infeasible: the enable expert parallel arms serve, but every request times out on expert all-to-all. Four-shard arms (pp4, tp2 pp2, tp4, tp4 ep4) fail to initialise within a 40-minute budget.

(a) cold phase 6 5 4 3 2 1 0

326 ms 221 ms 286 ms 466 ms 1.98 s 3.6 s 5.7 s

time since traffic start (s)

600

Figure 6. Scale-out dynamics of the premium-tier RayService under the default and runai-streamer loaders, with a fixed load of 50 concurrent streaming requests. Each step marks one pod’s first served response; diamonds mark serve-up. (a) Cold phase, time measured from manifest apply. (b) Hot phase, time measured from traffic start. Hot-phase behaviour matches the cold phase for both loaders, indicating that storage-cache warmth does not bound weight acquisition.

Eval Experiment N3: Distributed model bandwidth limiting Figure 9 shows that inter-shard bandwidth limits PP well before average traffic demand reaches the configured cap. Each configuration is swept at its maximum queue-free offered load (Table 5). At fixed RTT, median TTFT remains largely stable above a pair-specific threshold, then collapses abruptly as the cap is reduced further, accompanied by a drop in throughput. The threshold, defined as the cap at which median TTFT doubles relative to the unconstrained reference, lies between 100 and 200 Mbps for the 9 ms pair, 50 and 75 Mbps for the 18 ms pair, and 25 and 35 Mbps for the 33 ms pair. In contrast, TP exhibits no comparable cliff. Its TTFT remains broadly stable down to around 100 Mbps and then rises gradually, consistent with latency being dominated by the per-layer all-reduce chain. These measurements are reliable, since unconstrained references taken before and after every sweep agree and the 1 Gbps null control is indistinguishable from the unconstrained setting. The PP thresholds exceed the measured mean one-way traffic demand of 29, 15 and 19 Mbps on the 9, 18 and 33 ms pairs (Table 5). This gap reflects the bursty nature of pipeline activations, since performance depends not only on average bandwidth demand but also on whether each burst can drain before the next arrives. As the cap is reduced, burst-drain time consumes more of the inter-arrival interval, increasing TTFT despite low average link utilisation. The bandwidth floor therefore captures a burst-level constraint that mean

InferSlice: SLO-driven LLM Inference over a Telco WAN Premium

Concurrency 10

Standard

Free

Concurrency 16

Concurrency 32

P99 TTFT (ms)

104

103

500 ms TTFT SLO

P99 TPOT (ms)

103

102

100 Gbps

1 Mbps

0.5 Mbps

100 Gbps

1.5 (P,S) 1 (F) Mbps

0.5 Mbps

100 Gbps

2.5 (P) 1.5 (S) 1 (F) Mbps

0.5 Mbps

Client-edge bandwidth cap Figure 7. Experiment N1: Effect of client-edge bandwidth caps on P99 TTFT (top) and P99 TPOT (bottom) at client concurrency 10, 16, and 32. The dashed line marks the 500 ms TTFT SLO, and the shaded region marks the 0.5 Mbps cap. The middle cap is set near each tier’s unconstrained demand and differs by tier at concurrency 16 and 32. Full results are in Appendix C.2.1.

traffic demand alone does not reveal.

Premium unconstrained demand (Rx) Near-demand cap tested 0.5 Mbps cap (collapse)

Link rate (Mbps)

2.5 2.0 1.5 1.0 0.5 0.0 10

16

32

Client concurrency

Figure 8. Experiment N1: Premium unconstrained receive rate (measured at 100 Gbps) against client concurrency, with the neardemand cap tested at each load and the fixed 0.5 Mbps cap. The bandwidth needed to remain in the unconstrained regime scales with offered load.

Eval Experiment N4: Latency characterisation in Functional Disaggregation (P/D-D across PoP nodes). Table 6 reports serving latency for the disaggregated deployments relative to the aggregated co-located baseline of the same serving stack. All placements are driven at 2.7 requests/s, the maximum queue-free rate of the colocated baseline. Splitting prefill and decode across PoP nodes multiplies median TTFT from 0.11 s co-located to 4.2–4.4 s in every disaggregated arm, with P99 TTFT at approximately 19 s, and output throughput falls by roughly a quarter. The penalty is nearly invariant across pairs, since median TTFT stays within a narrow 4.2–4.4 s band from 9 to 21 ms of RTT. This indicates that the KV-cache transfer and the NIXL handover dominate the cost, while propagation delay plays little role. Median TPOT is essentially unchanged, so the queue-free per-token gain from disaggregation is negligible. The per-pod network counters on the KV-sending pod show that the transfer is not bandwidth-limited. Over the bench window the disaggregated pair transfers 1.5–3.6 GB of KVcache, or 15–35 MB per request. The stream is strictly oneway from prefill to decode, with the return path carrying only a few percent of the volume. It sustains 205–315 Mbps on average (peaks of 481–855 Mbps), which is roughly 10–

median TTFT (ms, log)

uncapped

105

104

103

102

pp2 9 ms pp2 18 ms pp2 33 ms tp2 9 ms

101

output throughput (tokens/s)

InferSlice: SLO-driven LLM Inference over a Telco WAN

400 350 300 250 200 150 100 50 0

pp2 9 ms pp2 18 ms pp2 33 ms tp2 9 ms

101

102 103 inter-shard bandwidth cap (Mbps, log)

102 103 inter-shard bandwidth cap (Mbps, log)

Figure 9. Experiment N3: median TTFT and output throughput vs. inter-shard bandwidth cap for pp2 on the VM2@UCL–VM1@Soton, VM2@UCL–VM2@QUB and VM2@QUB–VM1@TCD pairs and for tp2 on VM2@UCL–VM1@Soton, GPT-OSS-20B. Each configuration is swept at its maximum queue-free offered load (Table 5); the shaded region marks the effectively unconstrained setting.

25% of the 1.3–2.8 Gbps these paths can carry. The cost is therefore set by the handover itself and not by link capacity, and because it does not grow with RTT it is more consistent with a fixed per-request overhead of the KV connector than with a chain of RTT-bound transfer descriptors. Batching or pipelining the transfer would be needed to reduce this delay.

6

D ISCUSSION AND O PEN Q UESTIONS

6.1

Key Lessons

6.1.1

L1: Communication Patterns Define the Operating Boundaries of Distributed Inference

Table 7 summarises the resulting operating boundaries: which distributed inference techniques remain feasible as compute is spread across increasingly distant sites. We deem a technique feasible at a tier if it meets the 500 ms P99 TTFT SLO at a queue-free offered load that retains at least 50% of the co-located maximum queue-free rate. The first condition captures latency compliance and the second captures sustainable capacity. The reach of each technique is determined by its communication pattern rather than by available bandwidth alone. Tensor parallelism, which places an all-reduce on the critical path at every layer, does not remain feasible beyond a single subnet. Pipeline parallelism and prefill–decode disaggregation, which exchange activations and KV-cache state respectively, avoid synchronous all-reduce semantics and therefore tolerate wider-area placement in principle. Within our measurements, pipeline parallelism meets the latency SLO on every measured crossPoP pair (median TTFT 85–364 ms, P99 <0.5 s, Table 5), but the load it can sustain falls with RTT: the maximum queue-free rate is 2.60, 1.55 and 0.85 requests/s at 9, 18

and 33 ms, which is 59%, 35% and 19% of the co-located 4.40 requests/s. Under the criterion above, PP is therefore feasible at the Mid tier but not at the Far and Distant tiers, where it satisfies the latency SLO only at a fraction of the co-located capacity. The picture is also load-regime dependent: under saturation, client-side queueing inflates PP TTFT to 7.2–23.1 s on the same pairs. Configurations spanning more than one hop could not be initialised within our deployment budget. Prefill–decode disaggregation does not meet the SLO at any cross-PoP tier: its penalty is set by the KV-cache transfer and handover rather than by propagation delay, and median TTFT stays within 4.2–4.4 s across 9–21 ms. Data parallelism, which introduces no inter-node communication on the critical path, is insensitive to placement, but requires the model to fit within a single node’s GPU memory and therefore does not generalise to models exceeding that bound. 6.1.2

L2: Bottlenecks Shift with Load

The resource that limits SLO attainment is not fixed: it changes with offered load, and the three regimes we observed leave distinct signatures in the latency metrics. At low load, the three tiers are indistinguishable and inference and scheduling dominate latency (Section 5). As load grows, each slice reaches the capacity set by its replica ceiling and becomes queue-limited: TTFT rises by an order of magnitude while TPOT remains close to its unconstrained value (Tables 9 to 11). When the client-edge bandwidth cap is tightened, the slice becomes network-limited instead: measured link and token throughput clamp, and both TTFT and TPOT increase by an order of magnitude. The bandwidth needed to stay in the unconstrained regime also grows with

InferSlice: SLO-driven LLM Inference over a Telco WAN Table 6. Experiment N4: prefill/decode disaggregation across PoP pairs versus the aggregated co-located baseline (Llama-3.1-8B-Instruct, ShareGPT, 100 prompts). All placements operate at an offered load of 2.7 requests/s, corresponding to the maximum queue-free rate. All deployments use the production serving stack (vLLM 0.12). KV moved is the byte-counter volume on the KV-sending pod over the bench window. Placement (pair RTT) Med. TTFT (s) P99 TTFT (s) TPOT (ms) tokens/s KV moved (GB) peak/mean (Mbps) co-located (0 ms) VM2@UCL–VM1@Soton (9 ms) VM2@UCL–VM2@QUB (18 ms) VM1@Soton–VM2@QUB (21 ms)

0.11 4.24 4.37 4.44

0.17 19.0 19.1 19.6

load (0.91, 1.32 and 2.45 Mbps at concurrency 10, 16 and 32 for the Premium tier), so a cap that is sufficient at one load becomes the binding constraint at the next. A similar pattern appears across PoP nodes: PP meets the latency SLO at bounded load, but under saturation client-side queueing, not RTT, inflates its TTFT to 7.2–23.1 s. The practical implication is that no single utilisation metric is sufficient to identify the resource constraining SLO attainment. Instead, the control plane (Section 3.7) must distinguish queue saturation from network saturation by jointly interpreting TTFT, TPOT and link throughput, supported by the cross-layer observability described in Section 3.8. This distinction is essential for effective resource allocation: adding replicas can alleviate queueing but cannot resolve a bandwidth bottleneck. Resource scaling must therefore be guided by the identified bottleneck, rather than triggered by utilisation alone. 6.1.3

L3: Cold Start Limits Elastic SLO Enforcement

Elastic scale-out only protects an SLO if new replicas become ready before the queue builds. In our deployment, weight acquisition rather than scheduling bounds this delay: pods reach readiness within about two minutes under both loaders, so the remaining time is spent loading weights and producing the first token. Even with parallel weight streaming, which reduces end-to-end scale-out from approximately 1,500 s to 610 s, a replica added under load takes minutes to serve its first response, whereas the SLO breaches we measured develop within a single step of the concurrency sweep. The transient TTFT excursion of the Premium tier at concurrency 16 (Section 5) shows the same effect at small scale. Hot-phase scale-out matches the cold phase, so the delay is set by the loader and not by cache warmth, and keeping weights cached on the node does not remove it. Reactive autoscaling alone therefore cannot enforce latency SLOs under load growth, even when it is configured with generous replica ceilings. Capacity has to be provisioned ahead of demand, through traffic forecasting, warm standby replicas, or faster weight distribution, and the cold-start time of the largest model a tier serves sets the lead time that forecasting must achieve. Our measurements use an

25.2 24.3 25.1 25.2

406 297 308 297

– 1.54 3.55 1.95

– 567 / 205 855 / 315 481 / 208

8B model; the lead time will grow with model size, which makes this constraint more severe for the models that most need distributed serving. 6.2

Scope and Limitations

Our experiments use independent, short-lived, stateless chat requests drawn from ShareGPT, served on a homogeneous pool of NVIDIA L40S GPUs. Compute heterogeneity is emulated through Run:ai’s fractional GPU scheduler. We use Llama-3.1-8B-Instruct for service differentiation experiments and GPT-OSS-20B for distributed-inference experiments. The operating boundaries reported in Table 7 therefore characterise a specific workload class on a single accelerator generation, under the evaluated resource configurations. Agentic workflows introduce chained, stateful calls with growing contexts, shifting the optimisation objective towards task-level latency and making KV-cache locality an additional placement constraint. Longer-context workloads may consequently alter these operating boundaries, particularly for prefill/decode disaggregation, where KVcache transfer costs increase with the size of the context to be transferred. These scenarios, together with closed-loop control-plane adaptation to workload dynamics, remain outside the scope of the present evaluation. Service tiers were realised through statically configured per-slice policies, so the evaluation shows that coordinated policy enforcement yields distinct and predictable regimes, but not that the control plane can select configuration and serving location dynamically under changing load; the planning, forecasting and routing components of the architecture (Figure 2) were not evaluated in closed loop. Each distributed-inference technique was likewise evaluated in isolation, with network and compute dimensions varied one at a time (for example, rate-shaping inter-shard bandwidth at a fixed RTT), so interactions between techniques and between simultaneous network and compute pressure are not captured. Finally, model-parallel results cover at most two shards, since four-shard configurations failed to initialise within our time budget. These limitations motivate the open questions that follow.

InferSlice: SLO-driven LLM Inference over a Telco WAN Table 7. Feasibility of distributed inference techniques across JOINER cross-PoP latency tiers. A technique is feasible if it meets the 500 ms P99 TTFT SLO at a queue-free offered load that retains at least 50% of the co-located maximum queue-free rate. TP and PP evaluated with GPT-OSS-20B; DP and P/D-D evaluated with Llama-3.1-8B-Instruct. All deployments constrained by L40S GPU memory. ✓: feasible, ✗: unfeasible.

PoP tier Intra-PoP (same subnet) Near cross-PoP (<5 ms) Mid cross-PoP (5–10 ms) Far cross-PoP (10–15 ms) Distant cross-PoP (≥15 ms)

RTT (ms)

TCP TX (Gbps)

0.60 4.26 7.57 10.90 18.97

10.0 6.53 2.79 1.19 1.30

Distributed AI technique TP

PP

DP

P/D-D

✓ ✗ ✗ ✗ ✗

✓ ✓ ✓ ✗ ✗

✓ ✓ ✓ ✓ ✓

✓ ✗ ✗ ✗ ✗

TP: shards layers across nodes with per-layer all-reduce on the critical path; lowest memory per node, highest latency sensitivity. PP: partitions by layer group, exchanging only activations; meets the latency SLO at every measured RTT, but its sustainable queue-free rate falls to 59%, 35% and 19% of the co-located rate at 9, 18 and 33 ms. DP: replicates the model with no inter-node traffic; placement-insensitive, but the model must fit in a single node’s GPU memory. P/D-D: separates prefill and decode, transferring the KV-cache between them; unfeasible at every cross-PoP tier, with an RTT-insensitive penalty set by that transfer.

6.3 6.3.1

Open Questions Heterogeneity in compute and network

Operator infrastructure is inherently heterogeneous: sites differ in accelerator generation, memory capacity and available headroom, while WAN paths span more than an order of magnitude in latency and vary nearly as much in throughput (Table 4). Our results show that serving location is a first-order determinant of SLO attainment, making placement across heterogeneous sites a problem that existing serving systems have yet to address. We encourage the community to explore placement and routing policies that treat compute capability and network conditions jointly: when a nearby but less capable site should be preferred over a distant, better-provisioned one, how inference configurations and replica counts should be matched to mixed accelerators, and how heterogeneous replicas should interact with SLO-aware routing and autoscaling. Benchmarks that reflect heterogeneous, operator-controlled infrastructure rather than uniform clusters would make this research comparable and reproducible. 6.3.2

Composability

Our results establish distinct operating boundaries for distributed inference techniques over a WAN (Table 7): replication is insensitive to placement, pipeline parallelism is feasible up to mid-tier latencies, and tensor parallelism is confined to a subnet. These boundaries suggest a hierarchy that matches each technique to the network tier it tolerates: tensor parallelism within a node, pipeline parallelism across nearby nodes, replication across sites, and prefill/decode disaggregation within each replica (Section 2). However, the feasibility of individual techniques does not establish the feasibility of their composition. How far such compo-

sitions extend the feasible region, and how they interact, is a rich space for the community to explore. For example, replica-level autoscaling may amplify pipeline bubbles, while KV-cache transfer layered onto pipelined stages may introduce additional communication dependencies. A control plane that selects compositions according to site capabilities, network conditions and SLOs, accounting for the interactions between techniques rather than optimising each independently, is a natural next step. 6.3.3

Serving models that exceed single-node memory over the WAN

Our measurements identify communication on the critical path as the principal barrier to scaling out over the WAN. Replication is insensitive to placement but requires the model to fit on each serving node. Tensor parallelism is confined to a single subnet by per-layer all-reduce, while pipeline parallelism meets the latency SLO across sites but sustains less load as RTT increases, and is limited by the burst latency of activation transfers rather than by average bandwidth. Prefill/decode disaggregation encounters a different bottleneck: transferring 15–35 MB of KV-cache per request adds seconds to TTFT even though the transfer occupies only a small fraction of the available bandwidth, indicating that message-level latency, rather than throughput, constrains the handover. These findings point to concrete directions for inference systems research: overlapping communication with computation, compressing and shaping activation bursts, batching and pipelining KV-cache transfer descriptors, and relaxing synchrony through asynchronous or speculative execution. Progress here would allow operators to serve models that exceed the memory of any single site, which is currently out of reach over a WAN.

InferSlice: SLO-driven LLM Inference over a Telco WAN

6.3.4

Stateful and agentic workloads

Our experiments use stateless chat requests, whereas agentic applications chain stateful calls with growing contexts, shifting the service objective from per-request latency to task-level latency. In this setting, KV-cache locality becomes a placement constraint: routing a follow-up call to a site that already holds the relevant cache may be preferable to selecting the site with the most spare capacity. Our disaggregation results show that moving 15–35 MB of KVcache per request over the WAN adds seconds to TTFT (Section 5.2.1), so placement must account for cache residency and transfer cost as well as network conditions. Open questions include how SLOs should capture task completion time and tail latency across multi-step workflows, how routing should balance cache affinity against load and failure recovery, and how distributed KV-cache management should support reuse, migration and eviction across sites as contexts grow. Exploring evaluation methodologies and serving policies designed for stateful, agentic workloads over geographically distributed, operator-controlled infrastructure are key aspects for broader impact and adoption. 6.3.5

SLO-aware placement and control

The control plane of InferSlice includes planning and optimisation, traffic forecasting, semantic routing, prompt caching and distributed KV-cache components (Figure 2), whereas our evaluation exercises statically configured per-slice policies. Our results show that admission limits, replica ceilings and serving location each shape SLO attainment, and that the binding resource changes with load. The central open problem is closing the loop between observation, prediction and resource control, which requires forecasting demand accurately enough to scale ahead of cold-start delays, identifying the binding resource online, and jointly selecting serving locations, inference configurations and replica counts as compute and network conditions evolve, while preserving differentiated SLOs under load without over-provisioning. Energy, carbon and power availability add further objectives and constraints to this decision, as discussed next. We encourage the community to explore closed-loop, cross-layer control that integrates these decisions, since it is the point at which the other open questions come together. 6.3.6

Energy-aware AI grid

The geographic distribution of inference infrastructure also raises fundamental questions about the energy efficiency of AI serving. Rather than concentrating compute in purposebuilt, gigawatt-scale AI factories, operators could aggregate existing capacity across data centres, edge sites and other infrastructure into a geographically distributed AI grid, improving the utilisation of underused resources while placing workloads where compute capacity, network conditions and

energy availability are favourable. However, the energy implications cannot be inferred from accelerator utilisation alone, since distributed inference introduces network energy costs, replication increases the number of active accelerators, and the communication and synchronisation overheads characterised in this work can offset the gains from better utilisation. The same SLO can therefore be met at very different energy cost depending on the inference configuration and placement chosen, and powering down idle replicas must be weighed against the cold-start lead times that forecasting has to achieve. This motivates extending SLO-aware placement to jointly optimise compute, network and energy, including energy-aware routing, workload migration and consolidation, and the trade-off between local utilisation and WAN communication. Carbon-aware orchestration adds a further dimension, because sites draw on grids with different and time-varying carbon intensity, so flexible, throughput-oriented tiers could be deferred or shifted towards cleaner sites and periods, within the limits set by sovereignty, latency and the heterogeneity of sites discussed above. Finally, power supply must be treated as a first-class resource alongside compute and network capacity. Telco sites were provisioned for network equipment rather than dense accelerators, so available power, cooling and backup capacity may bound the GPU capacity each site can host and sustain, and sites with limited headroom may be less suitable than better-provisioned but more distant ones. Exploiting this requires power and energy telemetry to be exposed through the observability layer, so that the control plane can reason over latency, energy, carbon and power availability when selecting serving locations, configurations and replica counts. We encourage the community to explore such energy- and carbon-aware orchestration, and to report energy alongside SLO attainment in future evaluations.

7

C ONCLUSION

This paper presented InferSlice, a reference architecture for SLO-driven LLM inference over a telco WAN. InferSlice integrates network slicing, GPU partitioning, elastic serving and end-to-end observability, extending service differentiation from network connectivity to AI inference itself. Deployed across six PoPs of the JOINER testbed, on a shared network not dedicated to AI and without RDMA or specialised interconnects between sites, it shows that operatorcontrolled infrastructure can meet sovereignty, latency, and resource-locality requirements. Service tiers yield distinct latency-throughput regimes over the same GPU pool, with peak SLO-compliant goodput of 2.2× between the Premium and Free tiers, and parallel weight streaming reduces endto-end scale-out time from approximately 1,500 s to 610 s. Our experiments also show that the binding resource shifts

InferSlice: SLO-driven LLM Inference over a Telco WAN

with load, with queue saturation and bandwidth saturation leaving distinct signatures in TTFT and TPOT, so that crosslayer observability is a prerequisite for SLO enforcement.

cases, and challenges. IEEE Communications Surveys & Tutorials, 2025. doi: 10.1109/COMST.2025.3598406. arXiv:2405.03555.

Over the WAN, the reach of each distributed inference technique is set by its communication pattern rather than by bandwidth alone. Replication is insensitive to placement but bounded by single-node memory, pipeline parallelism meets the 500 ms SLO on every measured cross-site link while sustaining progressively less load as RTT grows, and tensor parallelism and prefill/decode disaggregation are not feasible beyond a single subnet. In the case of disaggregation, the penalty is set by the KV-cache handover rather than by propagation delay or available bandwidth. These boundaries suggest that distributed and heterogeneous inference systems should match each technique to the network tier it tolerates, and that serving models beyond the memory of a single site over a WAN remains an open problem. Our evaluation used statically configured policies, stateless chat workloads and a single accelerator generation, so closedloop control, stateful and agentic workloads, heterogeneous accelerators and energy-aware placement are part of our future direction. We hope that the lessons and open questions presented here encourage further research on SLO-driven inference over geographically distributed, operator-controlled infrastructure.

Behdin, K. et al. Scaling large language model inference for semantic job search. In Proceedings of the 2026 Conference on Machine Learning and Systems, MLSys, 2026.

ACKNOWLEDGEMENTS The authors thank the NVIDIA Run:ai team for their technical support throughout this work, including assistance with platform deployment, early access, and the provision of reference examples that informed the design presented in this paper. This work was partially supported by the UK’s Engineering and Physical Sciences Research Council (EPSRC) – JOINER project, EP/Y036514/1 – and the Smart Networks and Services Joint Undertaking – ECO-eNET project, Grant Agreement No. 101139133.

U SE OF G ENERATIVE AI D ECLARATION During the preparation of this work, the authors used Grammarly, Claude, and ChatGPT for grammar checking, and language and format editing. The authors subsequently reviewed and edited the content as necessary and take full responsibility for the content of this publication.

R EFERENCES Alam, K., Habibi, M. A., Tammen, M., Krummacker, D., Saad, W., Di Renzo, M., Melodia, T., Costa-Pérez, X., Debbah, M., Dutta, A., and Schotten, H. D. A comprehensive tutorial and survey of O-RAN: Exploring slicing-aware architecture, deployment options, use

Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., and Raffel, C. Distributed inference and fine-tuning of large language models over the internet. In Advances in Neural Information Processing Systems, volume 36, pp. 12312–12331, 2023. Cho, S. et al. Optimizing deployment configurations for large language model inference at scale. In Proceedings of the 2026 Conference on Machine Learning and Systems, MLSys, 2026. Hong, K., Li, X., Chen, L., Mao, Q., Dai, G., Ning, X., Yan, S., Liang, Y., and Wang, Y. SOLA: Optimizing SLO attainment for large language model serving with stateaware scheduling. In Proceedings of Machine Learning and Systems (MLSys 2025), volume 7, 2025. Huang, J., Huang, W., Xiong, Y., Li, E., Chen, X., Yu, X., and Zeng, L. SLO-aware scheduling for large language model inferences. arXiv preprint arXiv:2504.14966, 2025. Jisc. JANET network update. Presentation at the 37th Multi-Service Networks workshop (Coseners 2025), Cosener’s House, Abingdon, 3–4 July 2025, 2025. URL https://coseners.qmul. ac.uk/wp-content/uploads/2025/07/ 2025-07-03-Janet-update.pdf. Accessed 27 September 2026. JOINER. JOINER connects its first international node, in ireland. Web article, September 2025a. URL https://joiner.org.uk/news/ first-international-node-in-ireland/. Accessed 27 September 2026. JOINER. We officially launch. Web article, June 2025b. URL https://joiner.org.uk/news/ join-up-to-joiner/. Accessed 27 September 2026. JOINER. Architecture explained: Terminals, fabric, and the brain. Web article, July 2026a. URL https://joiner.org.uk/news/ joiners-architecture-explained/. Accessed 27 September 2026.

InferSlice: SLO-driven LLM Inference over a Telco WAN

JOINER. JOINER magazine. Brochure (PDF), 2026b. URL https://joiner.org.uk/wp-content/ uploads/2026/03/JOINER-Glossy-web.pdf. Accessed 24 September 2026.

Tang, Y., Lan, T., Huang, X., Lu, H., and Chen, W. SCORPIO: Serving the right requests at the right time for heterogeneous SLOs in LLM inference. arXiv preprint arXiv:2505.23022, 2025.

Kube-OVN. Kube-OVN Documentation. [Online]. Available: https://kubeovn.github.io/ docs/stable/, 2026. Accessed: Aug. 31, 2026.

Toniolo, A. R., Thorstenson, R., and Dinesh, A. GORGO: Online tuning for cross-region network-aware LLM serving, 2026.

Kundu, L., Lin, X., Gadiyar, R., Lacasse, J.-F., and Chowdhury, S. AI-RAN: Transforming RAN with AI-driven computing infrastructure. IEEE Communications Magazine, 2025. arXiv:2501.09007.

vLLM. Benchmark CLI for vllm. Website, 2026. URL https://docs.vllm.ai/en/stable/ benchmarking/cli/. Accessed 14 October 2026.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Liu, Y. et al. LLM-Slice: Network slicing for large language model services. arXiv preprint arXiv:2410.18499, 2024. URL https://arxiv.org/abs/2410.18499. Mitra, S. et al. Beyond the buzz: A systematic evaluation of prefill-decode disaggregation for large language model serving. In Proceedings of the 2026 Conference on Machine Learning and Systems, MLSys, 2026. MLCommons. Mlperf inference v5.1 benchmark. https://mlcommons.org/benchmarks/ inference-datacenter/, 2025. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. Ray: A distributed framework for emerging AI applications. In 13th USENIX Symposium on Operating Systems Design and Implementation, 2018. NDFF. What is ndff? Website, 2026. URL https:// www.ndff.ac.uk/what-is-ndff. Accessed 13 September 2026. Ray Project. KubeRay: A toolkit to run Ray applications on Kubernetes. https://github.com/ ray-project/kuberay. Accessed: October 2026. Run:ai. NVIDIA Run:ai: GPU orchestration and scheduling for AI workloads on Kubernetes. https://www. nvidia.com/en-gb/software/run-ai/. Accessed: October 2026. Taleb, T., Samdanis, K., Mada, B., Flinck, H., Dutta, S., and Sabella, D. On multi-access edge computing: A survey of the emerging 5G network edge cloud architecture and orchestration. IEEE Communications Surveys & Tutorials, 19(3):1657–1681, 2017. doi: 10.1109/COMST.2017. 2705720.

Wang, Y. et al. Cloud-edge collaborative wide-area distributed inference network for large models. Information and Communications Technology and Policy, 52(2):18– 23, 2026. Wang, Z., Li, S., Zhou, Y., Li, X., Gu, R., Cam-Tu, N., Tian, C., and Zhong, S. Revisiting SLO and goodput metrics in LLM serving. arXiv preprint arXiv:2410.14257, 2024. Wu, Z., Bravalheri, A., Ullauri, J. M. P., Nicolaescu, A.-C., Wu, Y., and Simeonidou, D. Drl for network slicing: Resource efficiency optimisation under reconfiguration avoidance. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 1–6. IEEE, 2025a. Wu, Z., Bravalheri, A., Ullauri, J. M. P., Wu, Y., and Simeonidou, D. Service-aware maximum likelihood-based network slicing for live low-latency streaming. In 2025 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6. IEEE, 2025b. Yet, M. et al. Sla-aware ai inference on RAN: A practical testbed for heterogeneous and disaggregated execution. In 2026 IEEE International Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 2026. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for transformerbased generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation, 2022. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024.

A PPENDIX A

JOINER: T ELCO WAN TESTBED

The Joint Open Infrastructure for Networks Research (JOINER) started as a research project in December 2023

InferSlice: SLO-driven LLM Inference over a Telco WAN Belfast

Glasgow

Edinburgh

Dublin

Manchester

Leeds

Birmingham

Nottingham

Powergate

Telehouse West

Asiera (formerly HEAnet)

North Wales Bristol South Wales 100 Gbps 200 Gbps 400 Gbps 800 Gbps

GÉANT via Slough Paris FABRIC/FAB

Harbour Exchange

Jisc optical transport PSBA regional access

Telehouse North

GÉANT via Docklands Amsterdam FABRIC/FAB

Figure 10. The JANET backbone and its international researchnetwork links (nominal link capacity 100 to 800 Gbit/s).

Beyond the objective of federating research facilities, JOINER offers unique capabilities to the affiliated institutions for accelerating research and development, including access to a hybrid cloud/edge with collocated computing resources at each node deployed in a distributed fashion and O-RAN compatible Radio Intelligent Controllers (RICs) and packet cores. In addition to these shared resources, research in the JOINER project also culminated in the creation and integration of the National Spectrum Facility, the AI Acceleration Facility and the Non-Terrestrial Networks in telecommunications (NTN) Emulation Facility. This combination of federated research capabilities, strong collaboration and production-scale national and international breadth positions JOINER to support the development of innovative technologies. By using JOINER, partners are able to test prototypes in a realistic large-scale environment, which helps to identify opportunities and issues that otherwise would remain hidden until the technology is deployed as part of a public-facing product. This is beneficial not only to the research community but also industry, operators, vendors and policymakers. A.1

with the goal of creating a nationwide experimentation platform focused on telecom innovation and future connectivity. Funded by the UK’s EPSRC as part of the Federated Telecoms Hubs, it started public operations in 2025 (JOINER, 2025b) focusing on interconnecting and fostering collaborations across participating institutions. Initially bringing together 13 universities and research laboratories across the UK, JOINER announced its first international node in Ireland still in 2025 (JOINER, 2025a). As of 2026, the list of institutions that are part of JOINER is: • Trinity College Dublin (TCD) • Queen’s University Belfast (QUB) • University of Glasgow • University of Edinburgh • Bangor University • University of Leeds • Cranfield University • University of Cambridge • Digital Catapult (DC) • University College London (UCL)

Architecture

JOINER infrastructure is implemented in the form of a distributed testbed, where each affiliated institution is assigned a terminal containing hardware to support the experimental capabilities, which generally comprises the following equipment (JOINER, 2026b): • Oscilloquartz OSA 5412 for precision time distribution (PTP/SyncE) and clock synchronisation. • high-performance Edgecore P4 switches based on the Intel Tofino chipset, complemented by AMD Alveo V80 accelerator cards for hardware offload and advanced network programmability. • Juniper ACX routers and EX switches, together with Fortinet firewalls for SD-WAN deployment and edge routing and securing. • Dell R760 and R660 servers with up to 64 physical cores, 512 GB+ RAM for edge computing, and Nvidia L40S GPUs for AI acceleration. • Dell ME5024 SAN arrays offering tiered storage (SSD/HDD) for high-throughput data access. • Some sites can host additional specialised hardware, including RFSoC FPGA evaluation kits, O-RAN compatible radios, and access to satellite links.

• Imperial College London • University of Oxford • University of Southampton • University of Bristol

Control and management functionalities, such as remote access, central authentication and authorisation and policy management, are aggregated via a central node called Brain, which also manages and automates the hybrid cloud. In

InferSlice: SLO-driven LLM Inference over a Telco WAN

Figure 11. The National Dark Fibre Facility and its external links. Access points include JOINER nodes in Bristol, Cambridge, Southampton and UCL, through which optical experiments can be carried out directly in the physical layer.

turn, the hybrid cloud is composed of VMware Telco Cloud Automation (TCA)f (managing the on-premises compute resources at all nodes) paired with complementary remote cloud services provided by Amazon Web Services (AWS)g , Microsoft Azureh and Google Cloud Platform (GCP)i . Additionally, JOINER is also able to automate physical servers via Metal as a Service (MAAS)j and deploy open-source virtualisation and container platforms, such as OpenStackk and OpenShiftl , on demand.

2025) and its access to other international research networks like the European GÉANTn and the USA-led FABo , through which it is also possible to connect to other institutions in Europe, North and South America and Asia. Regional access in Wales is provided via the Public Sector Broadband Aggregation (PSBA), which links sites across North and South Wales to JANET’s major interconnection points in the southwest and northwest of England. In Ireland, JANET also interconnects to the Irish NREN administrated by Asiera (formerly HEAnet).

A.2

Furthermore, JOINER is also connected to the National Dark Fibre Facility (NDFF) which supplies direct optical connectivity between some of the terminals (Bristol, Cambridge, Southampton and UCL) (NDFF, 2026) and enables optical experiments in both classical and quantum communications. Figure 11 details the physical layer, tracing the dark-fibre corridor across the southern part of England, along with an external link to BT’s premises in Adastral Park.

Connectivity

JOINER terminals provide access to data-link (layer 2 in the Open Systems Interconnection (OSI) model) and physical connectivity via a series of national and international partnerships with research and commercial networks. At its core, JOINER relies on JANETm , the UK’s National Research and Education Network (NREN), operated by Jisc, a not-for-profit organisation responsible for managing and operating education and research digital infrastructure. Figure 10 shows JANET’s major interconnection points (Jisc, f https://www.vmware.com/products/ cloud-infrastructure/telco-cloud-platform g https://aws.amazon.com/ h https://azure.microsoft.com/ i https://cloud.google.com j https://canonical.com/maas k https://www.openstack.org l https://www.redhat.com/en/technologies/ cloud-computing/openshift m https://www.jisc.ac.uk/janet

JOINER also supports NTN connectivity with links carried over Low Earth Orbit (LEO) and Geostationary Earth Orbit (GEO) services, and includes a nomadic terminal (assembled in a customised vehicle) that is backhauled via satellite and carries an on-board 5G radio access and core, with up to five hours of off-grid operation (JOINER, 2026a). On top of the physical underlay, JOINER provides multitenant network connectivity. Terminals interconnect at 10– n o

https://network.geant.org https://fabric-testbed.org/about/fab

InferSlice: SLO-driven LLM Inference over a Telco WAN

40 Gbit/s in a logical mesh (JOINER, 2026a), delivered as SD-WAN over L2-VPN. Different research projects and experiments can run simultaneously, and each one is able to deploy its own logical overlay and provision multiple isolated data-link layer networks mapped into Virtual Local Area Networks (VLANs) over the same shared infrastructure. Figure 12 consolidates the JANET and NDFF connectivity into the JOINER platform’s complete federated view, illustrating the overlay connectivity as a single mesh of dynamic VLAN-isolated links.

tiers. Speedometer-styled gauge charts display the request processing rate for each tier. Bottom-right: GPU slicing and allocation. This pane displays the occupancy of available GPUs in the cluster across different tiers. Fractional GPUs allocation is supported for each tier. Premium tier can occupy the largest share of GPU resource, followed by the standard tier and then the free tier.

B

C.1

I NFER S LICE DASHBOARD DESCRIPTION

For demonstration purposes, a user-friendly dashboard was developed to enable real-time interaction with the InferSlice setup. Users can interact with the dashboard and send predefined traffic patterns to the LLM endpoints, allowing them to observe the system’s real-time behaviour and performance. Autoscaling and resource allocation behaviour can be monitored according to the system’s current load, alongside real-time metrics such as latency, throughput, and network bandwidth. The dashboard focuses on comparing information across the premium, standard, and free tiers. Figure 13 shows the dashboard. Experiments are conducted across three tiers, each corresponding to a different network slice. The premium, standard, and free tiers are represented by purple, green, and blue, respectively. The dashboard presents real-time information across six panes. Top-left: Performance metrics by slice. This pane displays graphs of real-time inference performance metrics, including latency (ms), throughput (tokens/s), and network bandwidth (Mbps), over a fixed time window. Metrics from all tiers are shown on the same graphs to facilitate comparison. Bottom-left: Control and pattern selection. This pane allows users to interact with the dashboard and manually generate LLM traffic in real time. Users can specify the network bandwidth cap and select the traffic pattern to be generated. Three traffic patterns are available: Uniform Load, Bursty Traffic, and High Intensity. The behaviour of the InferSlice system is influenced by these user-defined parameters and is reflected throughout the dashboard. Topcentre: Running slices summary. This pane displays hardware resource utilisation by tier. The reported resources include CPU usage, memory usage, GPU allocation count, and GPU memory utilisation. The compute nodes associated with each tier are also displayed. Bottom-centre: Network topology. This pane presents the geographical locations of the compute nodes on a map of the UK, together with the network connections between them. Nodes and connections are colour-coded according to their respective tiers. Topright: Received prompts. This pane displays a sample of the real-time prompts currently being processed by the LLM. Prompts are categorised according to their corresponding

C

E XTENDED EXPERIMENTAL EVALUATION

Concurrency-policy differentiation

Section 5.1 differentiates the slices through the resources allocated to each. This experiment complements it by differentiating configurations through their concurrency policy, that is, how many requests each configuration admits for execution at once, as shown by the target and maximum concurrency settings in Table 8. The three configurations, C1, C2, and C3, produced clearly separated operating points in both throughput and latency. C1 sustained approximately 698–699 tokens/s across all tested loads, while maintaining mean TTFT close to 100 ms and P99 TTFT below 140 ms. C2 increased throughput to 883–902 tokens/s, representing a 26–29% improvement over C1, while maintaining P99 TTFT below the 500 ms interactive serving threshold at all tested request rates. C3 prioritised capacity, sustaining 2,117–2,130 tokens/s, approximately 2.4× higher than C2 and 3.0× higher than C1. Importantly, throughput remained stable once each configuration reached saturation, indicating that the available GPU resources were fully utilised and that additional load was absorbed through queuing rather than causing execution throughput degradation. Furthermore, no request failures were observed in any experiment, demonstrating stable operation across all configurations. A clear latency-throughput trade-off emerges across the three configurations. C1 consistently delivered the lowest latency, with mean TTFT remaining between 99 and 101 ms and P99 TTFT between 130 and 138 ms regardless of load. C2 accepted a moderate increase in latency, with mean TTFT ranging from 176 to 184 ms and P99 TTFT between 313 and 472 ms, while still satisfying interactive-service requirements. In contrast, C3 achieved the highest throughput but exhibited significant tail-latency degradation under increasing load. While P99 TTFT was only 308 ms at 8 RPS, it increased to 1,305 ms at 12 RPS and reached 2,001 ms at 32 RPS, about four times the interactive-service threshold. Notably, TPOT remained largely constant within each configuration, varying by less than 4% across the evaluated load range. This indicates that model execution performance remained stable and that the observed latency degradation originated primarily from request admission and queuing effects rather than from reduced decoding efficiency.

InferSlice: SLO-driven LLM Inference over a Telco WAN

Figure 12. The 14 JOINER sites as a federated overlay on top of JANET and NDFF, with the JOINER Brain colocated at Bristol and international connections to GÉANT, FABRIC/FAB and Asiera.

InferSlice: SLO-driven LLM Inference over a Telco WAN

Figure 13. InferSlice dashboard for real-time user interaction and monitoring. The dashboard provides interactive traffic generation, performance monitoring, resource allocation visualisation, and network topology information across premium, standard, and free service tiers. The interface enables users to observe the impact of traffic patterns and bandwidth constraints on autoscaling behaviour and resource distribution in real time.

InferSlice: SLO-driven LLM Inference over a Telco WAN

Together with the resource-based differentiation in Section 5.1, these results support the central hypothesis of InferSlice: service-level objectives can be translated into distinct and enforceable operating regimes over a common distributed infrastructure, and concurrency policy provides a second control for doing so. The observed behaviours reveal three practical service classes. C1 provides a latency-first regime suitable for highly interactive applications where responsiveness is the dominant requirement. C2 offers a balanced regime that increases resource utilisation while preserving bounded latency suitable for quality-critical workloads. C3 delivers a throughput-first regime that maximises infrastructure efficiency and is therefore suitable for asynchronous or batch-oriented inference workloads. More broadly, the results demonstrate that AI inference services can be differentiated in a manner analogous to telecom network services, allowing operators to offer multiple SLOdriven service tiers from the same pool of compute and network resources. This capability is fundamental for future telco AI platforms, where heterogeneous applications must coexist while receiving performance characteristics aligned with their specific operational requirements. C.2

Network Bottleneck Identification

C.2.1

Experiment N1 results: End client bandwidth limiting

At moderate load (concurrency 10), all three service tiers remain largely insensitive to bandwidth while the available link capacity exceeds approximately 1 Mbps (Table 9). Reducing the cap from 100,Gbps to 1 Mbps produces little change in throughput or TPOT: measured receive rates remain around 0.9–0.94 Mbps and throughput around 370– 380 tok/s, while P99 TTFT remains below the 500 ms SLO. A clear network bottleneck emerges when the cap is reduced to 0.5 Mbps. At this point, the measured receive rate saturates at approximately 0.46 Mbps across all tiers, token throughput falls to 216–224 tok/s, and P99 TTFT increases from hundreds of milliseconds to 1.9–3.3 s. TPOT simultaneously rises from approximately 26–38 ms to 150–276 ms. This sharp transition indicates a client-edge bandwidth floor between 0.5 and 1 Mbps for this workload: above the floor, inference and scheduling dominate latency; below it, network throughput becomes the binding resource and affects both first-token delivery and subsequent token generation. As concurrency increases, the bottleneck shifts, as shown in Tables 10 and 11. For Premium, the unconstrained traffic demand rises from approximately 0.91 Mbps at concurrency 10 to 1.32 Mbps at 16 and 2.45 Mbps at 32, and correspondingly the tier remains close to its unconstrained operating point under caps of 1, 1.5, and 2.5 Mbps, respectively. In every case, constraining the link to 0.5 Mbps forces the measured receive rate to approximately 0.45–0.47 Mbps

and collapses throughput to roughly 221–231 tok/s; at concurrency 32, Premium P99 TTFT reaches 12.8 s and P99 TPOT 1.19 s (Table 11). Standard and Free, however, already violate the TTFT SLO at higher concurrency even with an effectively unconstrained 100 Gbps link (Tables 10 and 11), while TPOT remains comparatively stable. Their high TTFT at these operating points therefore cannot be attributed primarily to network bandwidth and instead reflects queueing or compute-side saturation. Bandwidth restriction compounds this degradation once the 0.5 Mbps floor is reached. These results distinguish the network bottleneck from the previously observed queue boundary: queue saturation predominantly increases TTFT, whereas severe bandwidth saturation simultaneously reduces measured network throughput and token throughput and substantially increases both TTFT and TPOT.

InferSlice: SLO-driven LLM Inference over a Telco WAN

Table 8. Inference results across three InferSlice concurrency policies, demonstrating distinct latency–throughput operating regimes. Policy C1 enforces a strict concurrency limit suited to conversational, latency-sensitive workloads; Policy C2 enforces a conservative queuing policy suited to quality-critical workloads; Policy C3 maximises throughput for batch processing. All runs use Llama-3.1-8BInstruct, ShareGPT dataset, Poisson arrivals. Zero failed requests across all configurations.

Policy C1 target=1 max=8 concur=8

C2 target=2 max=16

C3 target=4 max=32

RPS

Mean TTFT (ms)

P99 TTFT (ms)

Mean TPOT (ms)

Total (tok/s)

P99 ≤ 500 ms

4 6 8 10 12

101.07 99.71 99.34 99.79 99.86

131.33 131.74 129.93 138.18 130.83

23.07 23.06 23.04 23.08 23.06

698 699 698 698 698

Yes Yes Yes Yes Yes

4 6 10 12 16

181 181 184 181 176

472 451 399 411 313

35.8 35.9 36.1 36.7 36.0

900 899 894 883 902

Yes Yes Yes Yes Yes

8 12 16 24 32

171 197 210 218 227

308 1305 1621 1761 2001

28.2 28.4 28.7 29.2 28.7

2117 2130 2121 2122 2122

Yes No No No No

Table 9. Client-edge bandwidth sensitivity at client concurrency 10. The no cap configuration represents the effectively unconstrained baseline. Bold values highlight the bandwidth transition region and resulting SLO violations.

P99 latency Tier

BW cap (Gbps)

Throughput SLO Measured link rate

Mean latency

Median latency

TTFT TPOT (ms) (ms)

(tok/s)

met

Rx (Mbps)

Tx (Mbps)

Premium No cap 277.06 37.99 Premium 1 × 10−3 239.16 37.32 Premium 5 × 10−4 3276.43 256.86

371.97 369.14 221.52

Yes Yes No

0.9100 0.9200 0.4600

0.1200 98.50 0.1200 96.09 0.0200 1237.59

24.65 76.81 24.79 76.45 53.71 898.96

24.60 100.0 24.70 100.0 41.38 100.0

Standard No cap 182.66 26.56 Standard 1 × 10−3 405.81 26.87 Standard 5 × 10−4 3338.20 276.36

377.97 379.55 216.53

Yes Yes No

0.9400 0.9100 0.4600

0.1200 101.76 0.1200 121.50 0.0200 1402.59

24.11 84.28 24.25 130.72 56.08 932.11

24.65 100.0 24.65 100.0 41.83 100.0

Free Free Free

371.59 377.12 224.29

Yes Yes No

0.9000 0.9300 0.4610

0.1200 0.1200 0.0289

24.23 121.71 24.18 121.54 50.85 872.36

24.93 100.0 24.91 100.0 42.71 100.0

No cap 381.11 26.39 1 × 10−3 377.27 26.35 5 × 10−4 1913.70 150.15

TTFT TPOT TTFT (ms) (ms) (ms)

110.47 112.89 948.83

TPOT (ms)

RPS

InferSlice: SLO-driven LLM Inference over a Telco WAN

Table 10. Client-edge bandwidth sensitivity at client concurrency 16. Premium remains within the TTFT SLO until the available bandwidth is reduced to 0.5 Mbps, whereas Standard and Free exhibit TTFT violations even under effectively unconstrained bandwidth, indicating queueing or compute-side saturation prior to network saturation. P99 latency Tier

Mean latency

Median latency

TTFT TPOT (ms) (ms)

TTFT TPOT (ms) (ms)

TTFT TPOT (ms) (ms)

(tok/s)

met

Rx (Mbps)

Tx (Mbps)

Premium No cap 418.16 39.49 Premium 1.5 × 10−3 321.93 39.50 Premium 5 × 10−4 4611.48 455.49

543.06 535.87 221.45

Yes Yes No

1.3200 1.3100 0.4500

0.1700 113.24 0.1700 109.36 0.0200 2090.22

25.91 115.29 26.26 115.67 85.43 1432.84

25.39 100.0 25.61 100.0 66.85 100.0

Standard No cap 508.64 33.68 Standard 1.5 × 10−3 517.36 35.39 Standard 5 × 10−4 3213.88 259.14

565.72 549.58 222.14

No No No

1.3800 1.3200 0.4500

0.1800 182.10 0.1700 173.26 0.0300 1579.19

25.46 179.52 25.49 179.17 80.19 1441.83

25.74 100.0 25.79 100.0 68.09 100.0

4547.57 27.17 4541.20 27.58 2948.25 248.18

437.21 438.43 224.22

No No No

1.0400 1.0500 0.4600

0.1400 1860.81 0.1400 1855.40 0.0300 1502.05

24.63 1675.30 24.67 1683.85 81.67 1382.30

25.20 100.0 25.23 100.0 67.99 100.0

Free Free Free

BW cap (Gbps)

Throughput SLO Measured link rate

No cap 1 × 10−3 5 × 10−4

RPS

Table 11. Client-edge bandwidth sensitivity at client concurrency 32. Premium sustains its unconstrained operating regime at approximately 2.5 Mbps, while severe bandwidth restriction to 0.5 Mbps causes substantial degradation in TTFT, TPOT, and token throughput. Standard and Free are already TTFT-limited under the unconstrained baseline, indicating a non-network bottleneck at this load. P99 latency Tier

BW cap (Gbps)

TTFT (ms)

Throughput SLO Measured link rate

Mean latency

Median latency

TTFT TPOT (ms) (ms)

TTFT TPOT (ms) (ms)

TPOT (ms)

(tok/s)

met

Rx (Mbps)

Tx (Mbps)

RPS

Premium No cap 379.14 44.51 Premium 2.5 × 10−3 416.14 47.14 Premium 5 × 10−4 12772.72 1187.84

1001.40 933.89 231.08

Yes Yes No

2.4500 2.3500 0.4700

0.3600 131.10 30.25 125.55 29.88 100.0 0.3400 132.70 30.67 124.12 29.91 100.0 0.0300 4564.35 178.11 4393.32 124.69 100.0

Standard No cap Standard 1.5 × 10−3 Standard 5 × 10−4

7876.05 7480.05 7291.51

40.72 136.18 537.05

617.85 604.78 223.35

No No No

1.4400 1.3400 0.4500

0.2000 4308.16 26.56 4211.11 27.21 100.0 0.0900 3400.20 40.59 3277.30 32.93 100.0 0.0300 3171.58 157.86 2789.21 135.70 100.0

Free Free Free

12716.79 12540.49 14120.28

31.98 26.91 553.79

446.12 446.78 231.38

No No No

1.0800 1.1000 0.4500

0.1400 8110.86 23.64 8070.19 24.17 100.0 0.1400 8107.47 23.55 8172.76 24.26 100.0 0.0300 5031.64 117.36 2843.13 125.58 100.0

No cap 1 × 10−3 5 × 10−4

InferSlice: SLO-driven LLM Inference over a Telco WAN

D

L IST OF A BBREVIATIONS

RIC Radio Intelligent Controller

AI Artificial Intelligence

RKE2 Rancher’s lightweight Kubernetes distribution

AWS Amazon Web Services

RTT Round-Trip Time

CNI Container Network Interface

SAN Storage Area Network

DC Digital Catapult

SD-WAN Software-Defined Wide Area Network

DP Data Parallelism

SLO Service-Level Objective

EPSRC Engineering and Physical Sciences Research Council

SSD Solid-State Drive SyncE Synchronous Ethernet

FPGA Field-Programmable Gate Array TCA Telco Cloud Automation GCP Google Cloud Platform TCD Trinity College Dublin GEO Geostationary Earth Orbit TP Tensor Parallelism HDD Hard Disk Drive TPOT Time per Output Token JOINER Joint Open Infrastructure for Networks Research TTFT Time to First Token KV Key-Value UCL University College London L2-VPN Layer-2 Virtual Private Network VLAN Virtual Local Area Network LEO Low Earth Orbit VM Virtual Machine LLM Large Language Model VPN Virtual Private Network MAAS Metal as a Service WAN Wide Area Network MEC Multi-access Edge Computing MTU Maximum Transmission Unit NDFF National Dark Fibre Facility NIC Network Interface Card NREN National Research and Education Network NTN Non-Terrestrial Networks in telecommunications OSI Open Systems Interconnection P4 Protocol-Independent Packet-Processing Programming Language PoP Point of Presence PP Pipeline Parallelism PSBA Public Sector Broadband Aggregation PTP Precision Time Protocol QoS Quality of Service QUB Queen’s University Belfast RFSoC Radio Frequency System-on-Chip

Related documents

• InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network#1448154
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference#189269
• How Developers Configure LLM Inference: A Large-Scale Study of Real-World Practices#1411920
• SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference#238580
• LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters#386838