TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure Yuhan ZhouP∗ , Yuchu Luo∗ , Hao NieS , Wangrunze LvB∗ , Yu ZhouS , Yibo ZhuS , Daxin JiangS , Chenren XuP P Peking University

S Stepfun

B Beijing University of Posts and Telecommunications

arXiv:2608.06007v1 [cs.DC] 6 Aug 2026

Abstract Modern LLM infrastructure increasingly manages tensors not only as computation data, but also as persistent states shared across distributed components. Existing systems optimize individual tensor management tasks, such as model weight loading, KV cache management, and checkpoint synchronization, by deeply integrating task-specific mechanisms with execution engines, networks, or storage backends. However, this specialization creates isolated silos that hinder the reuse and composition of tensor management strategies across evolving LLM workloads. In this paper, we identify tensor lifecycle management as a missing abstraction layer in LLM infrastructure and propose Tensor-as-a-Service (TaaS), which decouples tensor state management from computation logic. We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions, programmable lifecycle primitives, and a runtime that separates tensor management policies from execution mechanisms. This enables developers to write tensor management programs using TensorCast APIs while transparently leveraging distributed execution and data movement. We integrate TensorCast with vLLM and SGLang and evaluate it across diverse tensor lifecycle workloads, including model weight materialization, weight synchronization, KV cache management, and programmable request routing. Our results show that TensorCast achieves competitive performance with specialized tensor management systems while enabling new cross-component optimization policies. A programmable policy implemented with TensorCast improves median TTFT by up to 93.2% under highly concurrent multiturn agent workloads.

1

Task A Checkpointing

Task B Instance Auto-scaling

Task C KV Management

Training Node

Serving Node

Serving Node

Compute

Compute

Weights Lifecycle

Weights Lifecycle

KV Lifecycle

- identify - place - transform - materialize

- identify - place - transform - materialize

- identify - place - materialize - coordinate

Compute

!

Network & Storage Backend

Figure 1: Existing LLM infrastructure embeds tensor lifecycle management into task-specific stacks, causing repeated mechanisms and isolated optimization silos. Composable tensor management task programs Training Node

Serving Node

Compute

Compute

Task A

Task A + B

Serving Node

!

Compute Task A + B + C + …

Lifecycle Programming Interface identify / place / materialize / transform / coordinate / …

Distributed Lifecycle Runtime workflow execution / tracing / …

Distributed Tensor Pool weights / KV cache / ...

Tensor-as-a-Service Layer Network & Storage Backend

Figure 2: Decoupling tensor lifecycle management from computation. A unified Tensor-as-a-Service (TaaS) layer allows developers to compose tensor management task programs across workloads.

Introduction

Modern LLM infrastructure is rapidly scaling in both model size and deployment complexity, driven by models with billions to trillions of parameters (e.g., 2.8T parameters in Kimi K3 [1]) and stringent Service Level Objectives (SLOs) for millions of users. This evolution has pushed LLM systems beyond optimizing tensor computation alone: tensors increasingly become persistent states that must be managed across distributed components. For example, model weights

are distributed across elastic serving instances, KV caches are reused across nodes, and updated model weights are synchronized between training and inference pipelines. A large body of research has optimized individual tensor management tasks in LLM infrastructure. Representative examples include model weight loading for service auto-scaling [2–8], KV cache placement and transfer for prefill-decode

∗ This work was done while Yuhan Zhou, Yuchu Luo, and Wangrunze Lv were affiliated with Stepfun as employees or interns.

1

disaggregation [9, 10] and prefix cache sharing [11–15], as well as model checkpoint storage and resharding for largescale training [16, 17]. These systems achieve significant performance improvements by tightly integrating tensor management mechanisms with execution engines [18–20], network infrastructures (e.g., scale-up versus scale-out networks [21]), or storage backends. However, this task-specific optimization paradigm creates a fundamental abstraction gap in LLM infrastructure. Although different workloads manipulate tensors through similar lifecycle operations such as identifying, placing, moving, transforming, and materializing tensor states, existing systems expose these operations only through workloadspecific interfaces tightly coupled with execution engines, network layers, or storage backends, creating isolated vertical silos (Fig. 1). Such tight coupling makes the underlying tensor management mechanisms difficult to extract, reuse, or compose across workloads. This lack of a unified tensor lifecycle abstraction leads to two fundamental limitations. First, developers repeatedly reimplement similar tensor lifecycle mechanisms for each workload, increasing system complexity and slowing the evolution of LLM infrastructure. For example, adapting a serving scheduler from a load-aware policy to a cache-affinity-aware policy may require coordinated modifications across request routers, KV cache systems, and execution engines. Second, the lack of composability prevents LLM systems from exploiting optimization opportunities that span multiple components. A distributed LLM service may need to simultaneously balance instance load, preserve KV cache locality, and migrate request states across instances. However, existing siloed solutions make such cross-component optimization difficult to express. In this paper, we argue that tensor management should become an independent abstraction layer rather than remain embedded inside individual execution systems. As tensor management requirements continue to evolve and often require composing multiple lifecycle operations, such a layer must provide programmable control over tensor states throughout their lifecycle. We propose Tensor-as-a-Service (TaaS), a new abstraction layer that decouples tensor lifecycle management from the computation that produces or consumes tensors (Fig. 2). Guided by this vision, we design and build TensorCast, a distributed tensor management layer for LLM infrastructure. TensorCast realizes TaaS through three key design principles: i) Tensor-native abstraction. TensorCast represents tensors as first-class system objects with explicit identity, ownership, and lifecycle semantics, enabling developers to manage tensors independently from their physical placement and representation; ii) Programmable lifecycle management. TensorCast exposes composable primitives for tensor movement, transformation, and materialization, allowing

developers to define workload-specific policies without reimplementing low-level mechanisms; iii) Policy-mechanism separation. TensorCast separates tensor management policies from execution mechanisms: developers can write ordinary programs using TensorCast APIs to compose and control tensor lifecycles, while the TensorCast runtime transparently executes these operations across distributed cluster resources without requiring modifications to the underlying execution engines. Unlike traditional distributed object stores that treat data as opaque blobs [22–24], or computation frameworks that schedule user-defined tasks [25, 26], TensorCast provides a data-centric programming model where developers control tensor lifecycle policies while the system manages the execution mechanisms, including placement, movement, transformation, and materialization. We integrate TensorCast with vLLM [18] and SGLang [19] LLM inference frameworks, and evaluate it across diverse tensor lifecycle scenarios. Our results demonstrate that TensorCast achieves competitive performance with specialized tensor management systems in model weight materialization (§6.1), weight synchronization (§6.2), and KV cache management (§6.3), while enabling new cross-component optimization policies through programmable lifecycle composition. In particular, a programmable request router implemented with TensorCast reduces median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads by composing tensor lifecycle operations to jointly balance instance load and KV cache locality without modifying existing serving components (§6.4). TensorCast is open-sourced at https://github.com/tensorcast-ai/tensorcast. Contributions. • We identify tensor lifecycle management as a missing abstraction layer in modern LLM infrastructure and characterize the common lifecycle primitives shared across diverse tensor management workloads. • We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions and programmable lifecycle primitives to separate management policies from execution mechanisms. • We implement TensorCast and evaluate it across model weight materialization and synchronization, KV cache management, and programmable request routing. The results demonstrate both the efficiency and programmability of the proposed tensor lifecycle abstraction. This work does not raise any ethical issues.

2

2.2

2 Background and Motivation 2.1 Tensor Management Tasks in LLM Infrastructure

Observed Tensor Lifecycle Primitives

Although these workloads differ in application semantics, they expose a common feature: tensor states are repeatedly manipulated through a small set of common lifecycle primitives that are shared across different application logic. Tensors require independent management of identity, placement, transformation, and coordination beyond the computation that produces or consumes them, as summarized in Tab. 1. • Identify and Own. Tensors first need stable identity, metadata, and ownership semantics before they can be shared across engines, processes, or time; • Place and Move. Tensor states must be placed, replicated, migrated, or prefetched across nodes and instances according to workload policies; • Materialize. A logical tensor state must eventually be materialized into a compute-ready representation inside an execution engine; • Transform. Tensor states often require slicing, viewing, layout conversion, or resharding when crossing parallelism configurations or engine boundaries; • Compose and Coordinate. Real workloads combine multiple lifecycle steps, such as publishing, moving, transforming, and materializing tensors, and therefore require a policy-level mechanism to coordinate these operations across distribute components. These observations reveal that tensor management is no longer tied to individual workloads or execution engines. Instead, LLM infrastructure requires a common abstraction for managing tensor states across their lifecycles, from ownership to movement, transformation, and coordination.

LLM training and serving workloads increasingly treat tensors not only as intermediate computation results, but also as persistent states that must be created, transferred, stored, and transformed across system components. Representative examples include model weights shared across serving instances, KV caches reused across requests, and checkpoint tensors synchronized across training and inference workers. Consequently, efficient management of these tensor states becomes a fundamental challenge in LLM infrastructure. We next discuss three representative scenarios. Service Auto-Scaling To accommodate fluctuating request loads, LLM service providers dynamically scale serving instances, enabling a serverless Model-as-a-Service (MaaS) paradigm [27, 28]. Unlike traditional model deployment where weights are loaded once during initialization, elastic LLM serving requires repeatedly distributing large model weights (often exceeding 100 GB) to newly launched instances. Therefore, model weights become shared tensor states whose placement and materialization directly affect scaling latency. Existing solutions optimize this process by streamlining loading from storage backends [2–4] or retrieving weights from active serving instances [5–8]. KV Cache Management. Modern Transformer-based LLM serving systems [29, 30] maintain KV caches to avoid recomputing attention states during autoregressive generation. These KV tensors, generated throughout the prefill and decode phases, preserve the context required for future token generation. As context lengths grow and multi-turn interactions become prevalent, KV caches become large, fragmented, and reusable tensor states rather than transient execution buffers. Consequently, substantial research focuses on KV cache placement and transfer [9, 10, 15, 19, 31] for prefilldecode disaggregation [32–35], as well as KV reuse strategies for prefix caching [11, 14, 36–39].

2.3

Limitations of Specialized Solutions

Recently, numerous dedicated solutions have emerged to optimize individual tensor management tasks within LLM infrastructure, bringing significant performance improvements, as discussed in §2.1. However, despite sharing these common lifecycle primitives, existing systems expose them only through task-specific interfaces. Consequently, the same underlying tensor operations are repeatedly reimplemented and deeply coupled with different execution engines, network layers, and storage backends. This fragmentation leads to the following limitations.

Dynamic Checkpointing and Resharding. During LLM training and post-training, model tensors frequently move between different system components and parallelism configurations. Checkpointing periodically materializes model states for fault tolerance and evaluation, while RL post-training requires synchronizing newly updated weights from training actors to rollout workers. Furthermore, heterogeneous GPU resources and changing parallelism strategies require resharding checkpoint tensors into different layouts [16]. Consequently, several specialized systems have emerged to optimize model checkpointing, synchronization, and resharding processes [16, 17, 40–42].

Lifecycle primitives are coupled with task-specific mechanisms. Specialized solutions often integrate tensor lifecycle operations directly with workload-specific optimization logic. For example, Mooncake [9] adopts a min-TTFT strategy to schedule requests on prefill instances, which requires estimated prefill durations from the execution engine and predicted KV transfer times from the network. While such tight integration enables effective optimization 3

Workload/Task

Tensor State

MaaS serving

Model weights

Lifecycle Challenge

Common Lifecycle Primitives

Distribute immutable weights to new instances. KV cache management KV cache blocks Share reusable context across requests/instances. Synchronize updated weights from trainRL post-training Model weights ers to rollout workers. Agentic reasoning Reasoning states Branch, offload, and restore states across instances.

identify, place, transform, materialize identify, place, materialize identify, transform, materialize identify, materialize, coordinate

Table 1: Common tensor lifecycle abstractions across representative LLM infrastructure workloads. for KV-centric serving, the underlying tensor placement and movement mechanisms cannot be easily reused by other workloads. Consequently, tensor management tasks such as model weight distribution and KV cache migration are implemented as independent “vertical silos” that are deeply integrated with execution engines, network layers, and storage backends (Fig. 1). This task-specific design creates substantial engineering overhead as new tensor lifecycle requirements emerge. Furthermore, because optimization policies are frequently hard-coded within individual systems, adapting them requires modifying low-level mechanisms rather than composing existing primitives. For instance, transitioning a PD-disaggregated serving scheduler from min-TTFT [9] to a cache-affinity and load-balancing aware strategy [10] requires intrusive changes across the serving stack. This tight coupling restricts code reuse and limits the extensibility of tensor management strategies.

and Tree-of-Thought (ToT) [49] introduce intermediate execution states, such as reusable KV caches and branching context representations, that may need to be reused, duplicated, or migrated across instances. As such workloads become increasingly integrated into LLM applications and RL post-training rollouts [50–53], supporting them requires a programmable tensor management abstraction rather than another task-specific optimization stack.

2.4

Requirements for Tensor-as-a-Service

The limitations above stem from the absence of a unified abstraction for managing tensor lifecycles across LLM workloads. Conceptually, modern LLM systems consist of tensorproducing and tensor-consuming computation, while tensors increasingly become externally manageable states that require independent placement, movement, transformation, and reuse. However, existing systems embed these lifecycle mechanisms inside individual optimization stacks, limiting their composability and extensibility. Therefore, we propose decoupling tensor management from execution logic into an independent service layer that provides a state-management interface to execution engines as well as network and storage backends (Fig. 2). This Tensor-as-a-Service (TaaS) paradigm must satisfy the following requirements:

Lack of programmable composition across tensor lifecycles. Beyond optimizing individual tensor management tasks, modern LLM services increasingly require composing multiple lifecycle primitives under workload-specific policies. For example, a distributed LLM serving system may need to simultaneously balance instance load, preserve KV cache locality, and migrate request states across instances [43]. Such optimization requires coordinating tensor placement, migration, and materialization with application-level decisions. However, existing solutions expose these capabilities through fixed, task-specific interfaces. KV cache systems focus on cache storage, reuse, and transfer [9, 12, 44–46], while request schedulers optimize routing decisions [10]. Combining these mechanisms requires intrusive integration across multiple components, preventing developers from flexibly implementing new tensor management policies. This limitation has also been identified from the perspective of LLM execution engine design, where supporting emerging applications requires more programmable control over the inference process [47]. This limitation becomes increasingly important for emerging workloads with dynamic execution patterns. Reasoning approaches such as Chain-of-Thought (CoT) [48]

Tensor-native abstraction. The TaaS layer should natively understand tensor semantics and lifecycle, including different tensor states such as model weights and KV caches. Unlike traditional distributed object storage systems such as Redis [22], Vineyard [23], and Ray-Plasma [24], which treat data as opaque blobs, a tensor management layer should maintain tensor identity, ownership, placement, and transformation semantics. It should provide first-class handles for users to access and manipulate tensors while abstracting the underlying creation, transfer, storage, and transformation mechanisms through a high-performance implementation. Programmable lifecycle management. To support potentially infinite combinations of tensor management policies, 4

connect & exec. operations LLM App Caller

TensorCast Worker

result & signals

Application Caller. Application callers implement workloadlevel tensor management policies. They may physically reside within a TensorCast cluster but remain logically external to the tensor management runtime, accessing TensorCast as a service for LLM applications. Typical examples include request routers, auto-scaling managers, and orchestrators for multi-instance inference (e.g., ToT reasoning). These callers explicitly coordinate tensor operations with application-level computation (e.g., prefill and decode), enabling fine-grained control over tensor lifecycles without modifying the underlying execution engines. We later demonstrate an application caller through a programmable KV cache rebalancing policy in §3.3.

LLM Instance Caller Cluster

Figure 3: TensorCast’s programming model. the TaaS layer should provide programmable lifecycle primitives rather than hard-code optimization strategies. Different from compute-centric frameworks such as Spark [26] and Ray [25], where users dispatch computation tasks, TaaS users orchestrate tensor states by composing lifecycle operations and defining workload-specific policies. This programmability enables system extensibility while maintaining a reusable tensor management substrate.

Instance Caller. Instance callers provide the mechanism boundary between TensorCast and execution engines. They operate logically inside the TensorCast cluster and typically integrate directly with inference or training instances to expose engine-resident tensor states. For example, to support KV cache sharing, a minimal SGLang integration caller implements the HiCache [12] interfaces while connecting to a TensorCast worker, allowing the entire TensorCast cluster to serve as the KV backend. Unlike existing task-specific integrations such as Mooncake [9] and LMCache [11], which typically couple tensor movement mechanisms with optimization policies, TensorCast’s instance integration fixes only the mechanism required to access engine-resident tensors. The management policy remains programmable through callers, allowing different tensor management strategies to be implemented without modifying the execution engines.

Cluster-scale execution. As a unified service layer, TaaS must efficiently manage tensor states across cluster-scale resources with heterogeneous memory, network, and storage hierarchies. Achieving this requires distributed metadata management, fault tolerance, and a runtime environment with minimal overhead on execution paths.

3

TensorCast Programming Framework

To realize the TaaS paradigm described in §2.4, TensorCast exposes tensor lifecycle primitives through a programmable framework that separates tensor management policies from execution mechanisms. Specifically, TensorCast maps the lifecycle abstractions identified in §2.2 into four programming abstractions: Artifact for tensor identity and ownership, Operation for tensor lifecycle manipulation, Plan for composing distributed workflows, and Signal for policy feedback. This section presents the programming model (§3.1), the lifecycle APIs (§3.2), and an illustrative example (§3.3).

3.1

3.2

Lifecycle Programming Abstractions

TensorCast provides four programming abstractions that realize the tensor lifecycle primitives described in §2.2. Artifact represents tensor states and ownership; Operation provides lifecycle manipulation primitives; Plan composes distributed tensor workflows; and Signal exposes runtime information required for policy decisions. Together, these abstractions separate tensor management policies from execution mechanisms while providing a unified interface for diverse LLM workloads. Tab. 2 summarizes the essential APIs.

Programming Model

TensorCast separates tensor management policies from execution mechanisms through a caller-worker programming model. Callers define workload- specific tensor management strategies by composing lifecycle operations, while TensorCast workers provide the distributed runtime that stores tensors and executes these operations. As Fig. 3 illustrates, users write tensor management programs on callers; each caller connects to a TensorCast worker to obtain a Runtime handle. Workers maintain tensor states, execute lifecycle operations in a distributed manner, and return execution results together with internal state signals to callers for error handling, tracing, and policy decisions. Based on their roles, callers fall into two categories:

Artifact. It is the first-class representation of tensor states in TensorCast. It decouples tensor identity, ownership, and lifecycle from the underlying physical representation, allowing callers to manipulate tensors without explicitly managing their storage locations or device layouts. From the caller’s perspective, an Artifact is a logical handle to a tensor stored in the unified tensor pool. The actual tensor content is maintained by TensorCast workers as distributed replicas and is materialized only when callers request a concrete tensor representation through tensor(). This separation enables TensorCast to independently optimize tensor 5

Category

Class & Function

Description

Artifact

register(tensor, art_id) -> Artifact put(tensor, art_id) -> Artifact artifact(art_id) -> Artifact Artifact.tensor_meta() -> TensorMeta Artifact.tensor_dict() -> Tensor

Register a caller-leased tensor to TensorCast. Put a system-owned tensor to TensorCast. Get an Artifact handle from TensorCast given an ID. Get the tensor metadata of an Artifact. Materialize a tensor from an Artifact. Blocking. Derive a new Artifact from a view of an existing one.

Artifact.view(slice, name) -> Artifact

Plan.run(concurrency) -> PlanResult

Create a new call context. Create an executable plan with a context. Build a new plan step on a TensorCast worker. Build a new plan step on an instance. Execute the plan with specified concurrency. Blocking.

Worker Operation

prefetch(artifact, device) -> PlanStep prefetch_many(artifacts, device) -> PlanStep pin(artifact, device) -> PlanStep

Transfer an Artifact replica to the worker on device. Batched prefetch call. Pin an Artifact replica on the worker’s device.

Instance Operation

publish(req_id, ttl) -> PlanStep manifest(req_id) -> PlanStep hydrate(req_id, artifacts) -> PlanStep evict(req_id) -> PlanStep transform_into(artifact, spec, target) -> PlanStep

Flush a request’s KV cache to TensorCast. Get the set of Artifacts of a request’s KV cache. Load a request’s KV caches into the instance. Evict a request’s KV caches on the instance. Transform the artifact tensor according to spec rule and fill into the instance-owned target buffer.

Signal

connect(worker_addr) -> Runtime list_instances() -> list[Instance] list_workers() -> list[Worker]

Connect to a worker and get a runtime. List the statuses of all instances. List the statuses of all TensorCast workers.

Plan

context(id, ddl, key) -> CallContext plan(ctx) -> Plan Plan.on_worker(worker) -> PlanStepBuilder Plan.on_instance(inst) -> PlanStepBuilder

Table 2: Overview of essential TensorCast programming APIs. placement, replication, and movement while preserving a stable interface for callers. Artifacts are immutable and maintain a canonical identity, including tensor metadata, format, and content information, to deterministically identify tensor states. Callers introduce tensors into TensorCast under two ownership models: i) System-Owned. TensorCast assumes responsibility for the tensor lifecycle, which is suitable for durable global tensors such as model weights; ii) Caller-Leased. Callers retain ownership while granting TensorCast temporary management authority through a lease. This model supports transient but shareable tensors, such as KV caches during migration.

retrying. Based on the execution location, operations are categorized into two types: i) Worker Operation. These operations operate entirely within the TaaS layer without interacting with computational logic. They manipulate tensor states inside TensorCast, such as processing tensors locally on a worker (e.g., pin()) or transferring replicas between workers (e.g., prefetch()); ii) Instance Operation. These operations interact with computation engines and manipulate engine-resident tensor states through instance callers. Representative examples include publishing a request’s KV cache into the TensorCast cluster as a set of Artifacts through publish() and loading them back through hydrate(), as well as transforming model weights and filling rank-specific VRAM parameter buffers through transform_into(). The instance adaptor residing within the execution instance implements these operations.

Operation. Unlike general-purpose distributed frameworks that execute arbitrary user code (e.g., Ray [25]), TensorCast exposes a finite but extensible set of tensor lifecycle operations. This design provides a stable optimization boundary: callers define management policies by composing operations, while TensorCast controls their distributed execution. An operation carries a CallContext and behaves as a lazy handle that blocks only when callers explicitly await its result. The context contains a tracing ID, a deadline for maximum execution duration, and an idempotency key for reentrant

Plan. It provides the composition abstraction for coordinating multiple lifecycle operations across distributed workers and instances. Each operation in a plan corresponds to a PlanStep, and TensorCast maintains step dependencies through a Directed Acyclic Graph (DAG). Callers initiate execution through run(), after which TensorCast executes 6

operations according to the dependency graph and specified concurrency level. All operations within a plan share the same execution context. Consequently, TensorCast traces and retries a plan as an idempotent workflow using a unified ID and idempotency key. Plan allows callers to orchestrate complex tensor workflows without manually managing operation ordering, synchronization, or failure handling. TensorCast deliberately avoids distributed transaction semantics. A plan provides dependency ordering, tracing, deadline propagation, and idempotent retry, but does not provide ACID atomicity or rollback. Partially executed plans — such as a published KV cache or a prefetched replica — may remain and are reused, retried, or cleaned up according to operation 1 semantics. This design matches LLM tensor lifecycles, which 2 predominantly create immutable replicas, materialize ten- 3 sor views, or move recomputable states without requiring 4 5 general-purpose distributed transactions. 6 Signal. It closes the policy loop by exposing runtime sta- 7 tuses that callers need to make tensor management decisions. 8 9 Instead of embedding placement or scheduling policies in- 10 side TensorCast, the system provides observations that 11 enable callers to implement adaptive management strategies. 12 Prominent signals include list_workers(), which returns 13 the status of individual workers such as memory and net14 work pressure, and list_instances(), which returns the 15 status of individual instances such as load level and the corre- 16 sponding worker. By combining these runtime signals with 17 the Plan abstraction, callers can formulate and execute diverse tensor management policies while reusing the same 18 19 underlying lifecycle mechanisms. 20

3.3

Example: Programmable KV Cache Rebalancing

List. 1 demonstrates how an application caller can implement a workload-specific tensor management policy by composing TensorCast lifecycle abstractions. The example implements a programmable KV cache rebalancing policy for distributed LLM serving, which jointly considers instance load and KV cache locality without modifying the underlying inference engine or KV cache backend. The caller first uses Signal APIs to observe the runtime states of serving instances and applies an external policy to select a request, together with the KV cache of its entire context, for migration between instances (lines 9–10). The caller then constructs a Plan with a CallContext to compose three lifecycle operations: (1) an instance operation publish() flushes the KV cache from the source instance into the TensorCast cluster and registers the corresponding Artifacts; (2) a worker operation prefetch_many() optionally transfers the artifacts to the worker connected to the destination

21 22 23 24

instance; and (3) an instance operation hydrate() materializes the KV cache inside the destination inference engine. The prefetch step is optional because hydrate() can automatically fetch required artifacts once they are registered in TensorCast. This example highlights the programmability enabled by TensorCast: callers control tensor management policies, while TensorCast provides reusable mechanisms for tensor storage, movement, and materialization. The same programming model can be extended to other workloadspecific tensor management strategies. We evaluate this lightweight caller-driven rebalancing policy in §6.4. import tensorcast as tc # Initialize the caller's TensorCast runtime rt = tc.connect(gateway_addr) def rebalance(): # Choose to rebalance a request from inst1 # to inst2 decided by a rebalancing policy instances = rt.signals().list_instances() inst1, inst2, req_id = decide(instances) # Perform KV cache migration with a Plan ctx = tc.CallContext(id="tracing_id", deadline_ms=5000, idempotency_key="idem_key") plan = tc.Plan(ctx) # Flush request's KV cache as artifacts # from inst1 into the cluster flush_res = plan.on_instance(inst1).publish( req_id) # Optionally prewarm all KV cache # on the worker CPU DRAM connected by inst2 plan.on_worker(inst2.worker).prefetch_many( flush_res.artifact_result, device="cpu") # Load request's KV cache into inst2 plan.on_instance(inst2).hydrate(req_id, flush_res.artifact_result) # Execute the composed lifecycle workflow plan.run(concurrency=1)

Listing 1: LLM router with request-level KV cache rebalancing with TensorCast APIs. Error-handling codes are emitted for simplicity.

4 System Architecture 4.1 System Components The TensorCast runtime realizes the caller-worker programming model described in §3.1. As shown in Fig. 4, a TensorCast cluster consists of worker nodes, instance nodes, and a lightweight global store (GS). Callers specify tensor management policies through TensorCast APIs, while workers execute lifecycle operations over distributed tensor states. Instance nodes provide the mechanism boundary between 7

Rebalancer App

Caller Runtime

Plan Executor

tensor transfer flow ❺

Worker #3 ❷

Role: basic + home KV

Plan Executor

Plan Executor

KV

!

Plan Executor ❼

Instance Adaptor Inference Engine

❹

❽

Worker List

Caller Runtime

Instance List

Instance Adaptor

Artifact Locations Shard Leases

Instance. Instance nodes implement the mechanism boundary between TensorCast and computation engines. A typical instance node runs an LLM execution engine (e.g., an inference or training framework) and contains a TensorCast caller runtime that connects to a gateway worker. TensorCast integrates with the execution engine through an Instance Adaptor, which implements the minimal interfaces required to export, import, or transform engine-resident tensor states. This separation ensures that instance integration fixes only the underlying tensor management mechanisms, while workload-specific policies remain programmable through callers. Consequently, new tensor management strategies can evolve without repeatedly modifying execution engines or backend implementations.

Worker #N Role: basic + gateway

❻

❸ Caller Runtime

deployment, and monitoring of heterogeneous worker resources (e.g., assigning high-performance nodes as KV cache shard homes) and allows TensorCast to introduce new lifecycle capabilities through additional worker roles.

GS-worker heartbeat operation dispatch flow

❶

Worker #2 Role: basic + gateway

caller-worker connection

Worker #1 Role: basic + gateway

Tensor Ops. Plan

Inference Engine

KV

Backend DB

KV

Instance A

Global Store

Instance B

Figure 4: TensorCast system architecture. All components belong to the same TensorCast cluster except for the Req. Router App process. Steps 1 – 8 illustrate the workflow of executing the KV migration task shown in List. 1 (line 12–23). See detailed explanation in §4.5.

Global Store. TensorCast uses a Global Store (GS) as a lightweight control-plane metadata service rather than a component in the tensor data path. The GS maintains cluster metadata, including worker and instance status, the replica locations of low-cardinality tensors (§4.2), and shard ownership information. To avoid introducing a centralized bottleneck, workers cache metadata locally and refresh it through update exchanges with the GS. Therefore, most tensor management workflows bypass the GS and execute directly among workers and instances. The GS additionally serves as the entry point for cluster observability, including workload monitoring and runtime metrics.

TensorCast and computation engines, allowing TensorCast to manage engine-resident tensors without embedding workload-specific policies into the execution layer. Worker. A worker is the fundamental execution unit of TensorCast, responsible for maintaining tensor states and executing lifecycle operations. Rather than deploying separate services for storage, workflow execution, and metadata management, TensorCast composes worker functionalities through different roles. Currently, three roles exist: i) Basic. A worker with the basic role contributes its memory and disk resources to the unified tensor pool (§4.2). It also maintains a local plan executor for processing worker-local plan steps (e.g., artifact prefetch()) dispatched from other workers, but does not directly receive plans from callers. All TensorCast workers operate with the basic role at minimum. ii) Gateway. Gateway workers provide the entry point for callers to submit lifecycle workflows. They maintain caller connections, resolve plan-step dependencies, and dispatch operations to corresponding workers or instances for execution. In practice, gateway workers can share a common routable address behind a load balancer to provide scalable access to external callers. iii) Shard Home. For high-cardinality tensors such as KV caches, TensorCast partitions tensor states into multiple shards (§4.2). Workers with the shard home role maintain the ownership and consistency invariants of specific shards. This composable role architecture separates different lifecycle responsibilities while preserving a unified worker abstraction. It simplifies configuration,

4.2

Unified Tensor Pool

The unified tensor pool realizes the Artifact abstraction by separating logical tensor identity from physical tensor residency. Although callers interact with stable Artifact handles, TensorCast dynamically manages concrete tensor replicas across workers, devices, and storage tiers. Since different tensor states exhibit different lifecycle characteristics, TensorCast adopts different management policies for low-cardinality and high-cardinality tensors. Low-Cardinality Tensors. Some tensors in LLM workloads are immutable, durable, and accessed relatively infrequently. Representative examples include static model weights, which typically have a small number of tensor identities but large data sizes. We classify such tensors as low-cardinality tensors. Since their metadata changes infrequently, TensorCast maintains their replica locations centrally through the GS. During operations involving low-cardinality tensors, workers query the GS to locate available replicas. For example, an inference instance may load a Qwen3.5 model from local storage and introduce the weights into the tensor pool through 8

put(weights, "Qwen/Qwen3.5-35B-A3B"). TensorCast creates an Artifact replica on a worker, or directly generates

4.3

Distributed Plan Execution

TensorCast executes the Plan abstraction in a distributed manner, allowing callers to compose tensor lifecycle workflows without introducing a centralized execution bottleneck. Rather than maintaining a global operation scheduler, TensorCast distributes plan execution across workers, where each worker locally executes lifecycle operations assigned to it. A single operation is represented as a one-step plan, providing a unified execution path for all tensor workflows.

a CUDA IPC handle when the worker and instance reside on the same host. Subsequent instances initializing the same model can obtain the corresponding Artifact and materialize the weights through tensor(), fetching replicas from the original worker or nearby workers populated through prior prefetch() operations. In practice, a newly launched inference instance is typically deployed together with a co-host or nearby TensorCast gateway worker, allowing model weights to be shared with CUDA IPC or transferred through GPU Direct RDMA. Because low-cardinality tensors contain few distinct metadata entries and incur infrequent metadata accesses, centralized metadata management through the GS does not become a scalability bottleneck.

Worker-Local Plan Execution. Callers submit plans to a connected gateway worker and do not manage distributed execution logic locally. The gateway worker coordinates plan execution by resolving step dependencies and dispatching individual operations to corresponding workers or instances. This design keeps caller programs lightweight while maintaining a clean boundary between TensorCast system and the caller SDK. Each worker maintains a local plan executor that supports three execution responsibilities: i) Execute worker-local operations, such as pinning tensor replica residency or transferring replicas between workers through prefetch(); ii) Invoke instance adaptors to execute operations that interact with engine-resident tensor states, such as publishing or hydrating KV caches; iii) Maintain plan execution progress, including dependency resolution, operation dispatch, retry handling, and result propagation. Gateway workers provide this coordination functionality for callersubmitted plans. At the control plane, workers discover each other through worker metadata cached from the GS and communicate through RPCs. At the data plane, tensor contents bypass the GS and are transferred directly between workers and instances through high-performance communication paths, such as P2P RDMA when available.

High-Cardinality Tensors. In contrast, other tensor states exhibit high cardinality: they contain many fragmented objects, are frequently created and reused, and require dynamic placement decisions. KV caches are a representative example. Inference engines continuously generate and consume KV cache blocks or pages, making them significantly more dynamic than model weights. Maintaining the locations of all such tensors directly in the GS would introduce excessive metadata traffic. To address this challenge, TensorCast partitions high-cardinality tensors into shards and assigns each shard to a worker with the shard home role. The shard home worker maintains the ownership and consistency invariants of its shard. To prevent split-brain ownership during worker failures or network partitions, TensorCast associates each shard with a shard lease containing the shard home worker, expiration time, and a monotonically increasing fencing token. Assume 𝑁 shards exist. For a given shard_id ∈ {0, 1, . . . , 𝑁 − 1}, workers with the shard home role compute their ranks using Highest Random Weight [54] hashing: rank = HRW(shard_id, worker_id). The top-𝑘 workers for each shard (we use 𝑘 = 3 in our implementation to guarantee lease acquisition and reduce contention) compete to acquire the corresponding shard lease through the GS. The GS grants the lease to the first arriving worker and remains responsible only for lease metadata, while workers cache lease records locally. Shard home workers periodically renew leases through heartbeat messages. During worker or network failures, an expired lease triggers a new ownership acquisition process. After a new shard home worker acquires the lease, the fencing token increases, invalidating operations from stale workers because all workers reject high-cardinality tensor operations carrying outdated fencing tokens. This lease-based design distributes tensor metadata management across workers and limits GS operations to shard number 𝑁 , rather than per-tensor queries, allowing TensorCast to scale to high-frequency tensor state queries.

4.4

Lifecycle Consistency and Fault Tolerance

TensorCast adopts fault tolerance mechanisms that preserve tensor lifecycle semantics under failures. Different tensor states follow different recovery strategies: durable low-cardinality tensors rely on persistent replicas, while high-cardinality tensors rely on distributed shard ownership and lease-based consistency. Worker Failover. Workers maintain cached execution metadata, while persistent metadata remains recoverable from the GS. Following a worker failure, recovered workers reconstruct transient caches, such as worker and instance lists, and reacquire shard leases from the GS. Low-cardinality tensor replicas remain available if their physical replicas survive or reside on persistent storage. In contrast, high-cardinality tensors whose shard leases expire are considered unavailable until they are reconstructed or republished. 9

Global Store Failover. The GS maintains persistent metadata required for cluster coordination, including worker status, replica information, and shard ownership. This metadata is committed to a backend database for durability. In production deployments, standard replication mechanisms such as state machine replication (e.g., Paxos [55], Raft [56]) or chain replication [57, 58] can provide GS availability.

4.5

DuckDB [59], maintaining control-plane metadata including worker status, artifact indices, replica locations, and shard leases. Workers do not access the global store during tensor data transfer. Instead, they cache required metadata locally and execute operations through the distributed plan runtime. Worker liveness and metadata consistency are maintained through heartbeat and reconciliation RPCs. Tensor Materialization and Transfer. To realize Artifact materialization efficiently, workers implement a streaming pipeline that resolves tensor metadata, allocates target memory, transfers tensor contents from local storage or remote workers, verifies integrity, and exports device-specific handles to callers. GPU-resident tensors on the same host are exposed through CUDA IPC to avoid unnecessary data copies. For inter-worker tensor movement, TensorCast uses RDMA when possible and falls back to a multi-connection TCP transport based on userspace mTCP [60]. GPU transfers use pinned streaming buffers and asynchronous CUDA streams to overlap data movement with execution.

Putting Everything Together

We use the programmable KV cache rebalancing example from List. 1 (lines 12–23) to illustrate how TensorCast realizes a complete tensor lifecycle workflow. As Fig. 4 illustrates, the rebalancer application, instance A, and instance B connect to workers 1, 2, and 𝑁 , respectively. When the application caller invokes plan.run(), the plan is submitted to worker 1, which acts as the gateway worker responsible for coordinating execution ( 1 ). To execute the publish() operation on instance A, worker 1 dispatches the corresponding plan step to worker 2, which is connected to instance A ( 2 ). The plan executor on worker 2 invokes the instance adaptor inside instance A to export the engine-resident KV cache ( 3 ). The instance adaptor retrieves the KV cache from the inference engine, determines the corresponding shard, locates the shard home worker 3 using the locally cached shard lease, and transfers the KV cache to worker 3 ( 4 ). After the KV cache is registered as Artifacts in TensorCast, worker 1 dispatches the optional prefetch_many() operation to worker 𝑁 ( 5 ). Worker 𝑁 uses its local lease information to locate the shard home worker 3 and transfers the required KV replicas locally ( 6 ). Finally, worker 𝑁 invokes the instance adaptor inside instance B ( 7 ) to execute hydrate(), which materializes the KV cache back into the destination inference engine ( 8 ). The entire data path is distributed across workers and instances without GS, and the GS remains responsible only for control-plane metadata such as shard ownership and leases.

Programming Interface. The Python SDK provides the programming interface used by application callers and instance callers. It exposes TensorCast abstractions through a PyTorch-compatible interface and communicates with local workers through protobuf-generated gRPC stubs. The SDK manages caller-side runtime state, including session context, deadlines, retries, metadata caching, and the lifecycle of returned shared-memory handles. The SDK deliberately does not communicate directly with the global store, preserving the separation between application programs and the TensorCast control plane.

6

TensorCast Showcases & Evaluation

We implement TensorCast as a distributed C++ runtime with a Python SDK. The runtime consists of approximately 160K lines of C++ code, while the SDK contains approximately 95K lines of Python code. We highlight key implementation details of tensor materialization, data transfer, and LLM framework integration below. The implementation is publicly available at https://github.com/tensorcastai/tensorcast.

We evaluate TensorCast to answer two questions: i) whether a unified tensor lifecycle layer can provide efficient mechanisms comparable to specialized tensor management systems; and ii) whether its programmable abstractions enable new tensor management policies through composition of lifecycle primitives. We integrate TensorCast with mainstream inference engines, vLLM [18] and SGLang [19], and evaluate four representative scenarios: model weight materialization (§6.1), model weight synchronization (§6.2), KV cache management (§6.3), and programmable request routing (§6.4). The first three scenarios evaluate the efficiency of TensorCast’s reusable tensor lifecycle mechanisms, while the last demonstrates the programmability enabled by composing these mechanisms into a workload-specific policy.

Runtime Implementation. The TensorCast runtime is implemented as a distributed C++ service. Workers communicate through gRPC and maintain tensor replicas across CPU memory, GPU memory, and local storage. The global store is implemented as a Python gRPC service backed by

We evaluate TensorCast’s ability to materialize large lowcardinality tensor states during elastic LLM serving by integrating it into vLLM as a model weight loading module.

5

Implementation

6.1

10

Model Weight Materialization

3

10

2

10

1

10

0

Default InstantTensor

1.3k 211

39.5

20.7

TensorCast cold TensorCast warm 11.5

11.0 10.7

0.345

4.31 3.35

10

3

10

2

Default InstantTensor

1.3k

SSD Storage Type

TensorCast cold TensorCast warm

246 45.3

0.367

0.345

JFS

Time (s)

Time (s)

10

72.2 24.2

JFS

tmpfs

(a) Weight load time.

42.8 35.0

24.3

42.4 35.3

SSD Storage Type

26.9 23.7

tmpfs

(b) End-to-end ready time.

10

4

10

3

10

2

10

1

10

0

Default InstantTensor 265 72.0 76.8

5.7k 985 148

TensorCast cold TensorCast warm 83.1

13.5 18.2 0.696

0.587

0.498 JFS

Time (s)

Time (s)

Figure 5: Instance launch time for Qwen3-30B-A3B model.

SSD Storage Type

10

4

10

3

10

2

Default InstantTensor

5.7k 1.0k

(a) Weight load time.

299

181

116

106 112 26.1

25.0 JFS

tmpfs

TensorCast cold TensorCast warm

SSD Storage Type

46.9 52.2

24.9

tmpfs

(b) End-to-end ready time.

Figure 6: Instance launch time for Qwen3-235B-A22B model.

Update Time (s)

TensorCast

15 10

2.00x

1.81x

1.61x

5 0

1

2 TP Size

(a) Qwen3-14B.

4

Update Time (s)

Default

20

40 35 30 25 20 15 10 5 0

Default

TensorCast

tensor materialization to occur concurrently with instance launch or during a pre-flight stage.

1.14x 1.66x

Results. We report both the model weight loading time during instance launch and the end-to-end instance-ready time to evaluate how TensorCast accelerates instance initialization. As shown in Fig. 5 and Fig. 6, TensorCast consistently outperforms all baselines across settings. In particular, under JFS, a common case where MaaS providers host model weights, for the Qwen3-30B-A3B model, TensorCast-cold reduces model weight loading time by 60.7× and 10.2× compared to Default and InstantTensor, respectively (Fig. 5a). Consequently, it reduces end-to-end launch-ready time by 28.5× and 5.4×, respectively (Fig. 5b). These gains stem from two factors: i) TensorCast concurrently reads slice views for each rank while also issuing parallel read streams for each rank’s assigned slice; and ii) tensor transfers to VRAM are pipelined to hide H2D copy overhead. For TensorCastwarm, vLLM consumes pre-materialized tensors in less than 1 second, as TensorCast uses CUDA IPC to enable zerocopy access. As a result, end-to-end launch time is only dominated by runtime initialization overhead, yielding 228.6× and 40.7× speedups over Default and InstantTensor, respectively, for the Qwen3-235B-A22B model (Fig. 6b). These results demonstrate that TensorCast can efficiently materialize large immutable tensor states while preserving a general tensor management abstraction.

2.63x 1

2 TP Size

4

(b) Qwen3-32B.

Figure 7: Model weight synchronization time. Setup. We use a single node with 64 CPUs, 500 GB DRAM, and 8 × H800 GPUs to launch a vLLM instance (TP=8). This setup emulates a MaaS auto-scaling scenario, where new instances are launched as quickly as possible. We load model weights in safetensors format from three storage types: i) a high-performance distributed file system (JuiceFS [61]); ii) node-local SSD; and iii) node-local DRAM (tmpfs). Given a model path on each storage type, the vLLM instance materializes weights through four configurations: i) Default, vLLM’s native weight loader; ii) InstantTensor [62], a fast weight loader with pipelined and zero-copy I/O; iii) TensorCast cold, where a TensorCast worker reads rank-specific weight slices from storage and materializes them into VRAM, after which the vLLM process consumes the tensors through the tensor_dict API; and iv) TensorCast warm, where weight slices are pre-materialized locally through the prefetch API and vLLM directly consumes the materialized tensors. This design exploits TensorCast’s separation between tensor lifecycle management and execution initialization, allowing 11

40 20 0

2

4 Node Number

8

(a) 16k prompt w/ RDMA.

100 200 2

Mooncake

TensorCast

4 8 Node Number

(b) 16k prompt w/o RDMA.

TensorCast

75 50 25 0

2

4 Node Number

8

(c) 35k prompt w/ RDMA.

Mooncake TTFT Reduction (%)

60

Mooncake 0

TTFT Reduction (%)

TensorCast TTFT Reduction (%)

TTFT Reduction (%)

Mooncake

50 25 0 25 50

2

TensorCast

4 Node Number

8

(d) 35k prompt w/o RDMA.

Figure 8: Qwen3-32B TTFT redcution ratio with TensorCast and Mooncake KV storage backend.

10 0

2

Node Number

4

(a) 16k prompt w/ RDMA.

Mooncake

TensorCast

50 100 150 2

Node Number

4

(b) 16k prompt w/o RDMA.

40

Mooncake

TensorCast

30 20 10 0

2

Node Number

4

(c) 35k prompt w/ RDMA.

TTFT Reduction (%)

20

0

TTFT Reduction (%)

TensorCast TTFT Reduction (%)

TTFT Reduction (%)

Mooncake

20 10 0 10 20 30

Mooncake

2

TensorCast

Node Number

4

(d) 35k prompt w/o RDMA.

Figure 9: Qwen3-235B-A22B TTFT reduction ratio with TensorCast and Mooncake KV backend storage.

6.2

compared with tensor transfer. In contrast, for the larger Qwen3-32B model, TensorCast benefits from concurrently transferring multiple tensor views, better utilizing available network bandwidth and achieving higher acceleration as TP size increases. These results demonstrate that TensorCast can efficiently manage versioned tensor states across distributed execution instances while allowing different consumers to materialize the tensor representations required by their parallelism configurations.

Model Weight Synchronization

We evaluate TensorCast’s ability to synchronize versioned tensor states across distributed LLM instances by integrating it into SGLang as a new model weight updating module. Setup. The experimental setup consists of a two-node cluster, where each node runs a TensorCast worker with 64 CPUs, 500 GB DRAM, and 4 × H800 GPUs. We use the WeightPublisher helper provided by TensorCast, which wraps the put() API to publish different versions of model weights into a TensorCast worker on one node. A serving SGLang instance on another node synchronizes model weights by retrieving tensors from the TensorCast cluster. This setup represents the training-serving synchronization pattern commonly seen in RL post-training, where a trainer publishes updated model weights and rollout workers periodically synchronize their parameters. During synchronization, each tensor-parallel (TP) rank retrieves only its corresponding tensor slice through the view() API, demonstrating TensorCast’s ability to materialize different tensor layouts from a shared tensor state. To ensure a fair comparison against the Default baseline, which retrieves full model weights from JFS through a file path, we disable RDMA for TensorCast.

6.3

High-Cardinality Tensor Management

We evaluate TensorCast’s ability to manage high-cardinality tensor states by integrating it into SGLang’s HiCache module [12] as a KV cache storage backend. Each KV page is represented as a high-cardinality Artifact, with KV pages distributed across TensorCast workers. Setup. The experimental setup consists of an 𝑁 -node cluster, where each node is equipped with 64 CPU, 1TB DRAM, 8 × H800 GPUs, and 200 Gbps RDMA links. We deploy one TensorCast worker and one SGLang instance on each node. To evaluate KV reuse, we first send prompts from the LongBench dataset [63] to a single instance, and subsequently resend the same prompts to all other instances in the cluster simultaneously. This workload forces 𝑁 − 1 instances to concurrently retrieve KV pages from the TensorCast cluster, guaranteeing a 100% cache hit rate and stress-testing highcardinality tensor retrieval. We group prompts into short and long categories, with approximately 16k and 35k tokens on average, respectively. We compare TensorCast against the

Results. As shown in Fig. 7, TensorCast accelerates model weight synchronization by 1.14× to 2.63× compared with the default updater. The acceleration trends vary with TP size and model scale. For the Qwen3-14B model, increasing TP size introduces additional tensor slicing overhead on the TensorCast worker, which becomes the dominant cost 12

Preset

𝜎

Median

Mean

P5

P95

Represents

fast 2.1 0.6 medium 3.0 0.8 slow 4.1 1.0

8.2 s 20.1 s 60.3 s

9.8 s 27.7 s 99.5 s

3.0 s 5.4 s 11.6 s

22.0 s 75.0 s 311 s

Tight coding-agent loop; short tool replies dominate Typical SWE agent with mixed view, edit, and tool execution Long-tool-dominated workflows; research/planning agents

𝜇

Table 3: The three inter-turn delay presets used by the LogNorm sampling within a session. state-of-the-art Mooncake [9] integration in SGLang under identical settings. Mooncake represents a highly optimized KV-centric system, allowing us to evaluate whether TensorCast’s general tensor lifecycle abstraction introduces performance overhead compared with a specialized KV management design.

Setup. Based on the widely used cache-aware routing policy, we implement a lightweight rebalancing policy that periodically observes instance states and migrates selected requests, together with their KV caches, from the most-loaded instance to the least-loaded instance using the Plan abstraction shown in List. 1 (see detailed rebalancing policy in §A). This combined policy aims to preserve KV cache locality while avoiding load imbalance across instances, without losing context during migration. The rebalancing policy is independent of TensorCast and can be customized without modifying the SGLang scheduler, KV cache backend, or TensorCast runtime. We compare the TensorCast router against three baseline routers implemented in the SGLang model gateway [65]: i) a pure load-aware router using the widely used power-of-two-choices policy [66]; ii) a load-aware router with Mooncake as the backend for sharing KV caches across instances; and iii) a pure cache-aware router that routes requests to instances with the longest local prefix match. The cluster consists of four nodes, each equipped with 2 × H20 GPUs. Each node runs one Qwen3-32B instance with TP=2.

Results. We measure the average TTFT reduction of the subsequent 𝑁 − 1 instances relative to the first instance to evaluate the effectiveness of KV reuse. We also evaluate RDMA-disabled settings, since KV pages are increasingly transferred across clusters or datacenters without RDMA interconnects [64]. For the Qwen3-32B model with TP=2, as shown in Fig. 8, with RDMA enabled (Fig. 8a and Fig. 8c), TensorCast achieves performance comparable to Mooncake, with TTFT reductions ranging from 60% to 87.5% (i.e., 2.5× to 8× speedup). Performance improves as the number of nodes 𝑁 increases because TensorCast distributes highcardinality tensor states across shard homes rather than centralizing metadata management through the global store (§4.2). With RDMA disabled (Fig. 8b and Fig. 8d), TensorCast significantly outperforms Mooncake while maintaining positive TTFT reductions. This improvement comes from TensorCast’s userspace mTCP implementation [60] with multipath transmission, which reduces kernel copy overhead and aggregates higher effective bandwidth compared with in-kernel TCP. For the Qwen3-235B model with TP=8, as shown in Fig. 9, TensorCast again achieves comparable performance to Mooncake under RDMA and better performance without RDMA. Compared with TP=2, TP=8 introduces 4× more concurrent KV page retrieval requests, increasing contention and reducing TTFT improvements for both systems. These results demonstrate that TensorCast can efficiently manage highly dynamic, high-cardinality tensor states while preserving the performance of specialized KV cache systems.

6.4

Workloads. We generate dynamic multi-turn workloads from real agent rollouts on the SWE-Gym task pool, collected by running the OpenHands agent on approximately 2,400 real-world Python repository tasks [67]. We treat each rollout trajectory as a multi-turn request session and maintain concurrent sessions as workload traffic to the cluster. The interval between consecutive requests within a session is sampled from a LogNormal distribution, since this rightskewed distribution reflects human-action intervals [68, 69]. We use three LogNormal parameter presets–fast, medium, and slow–with parameters (𝜇, 𝜎) as described in Tab. 3. Results. We measure the median TTFT of all requests across all instances and report the results in Fig. 10. As the number of concurrent user sessions increases, the median TTFT increases rapidly for all methods. However, the rebalancing policy implemented by composing TensorCast lifecycle APIs achieves the lowest median TTFT across all workloads. Specifically, under the heaviest loads, 128 sessions for fast and 256 sessions for medium and slow, the TensorCast router reduces median TTFT by 71.4%, 93.2%, and 70.4% for fast, medium, and slow, respectively, compared with the loadaware router using Mooncake. We also report the average cache hit rate in Fig. 11. As the number of concurrent user

Programmable Request Router

We evaluate TensorCast’s programmability by implementing a workload-specific request routing policy that composes multiple tensor lifecycle operations. We integrate an instance adaptor into SGLang, enabling callers to execute the instance operations listed in Tab. 2 on SGLang instances. 13

TTFT P50 (ms)

10

4

10

3

Load-aware

4

Load-aware + Mooncake

8 16 32 64 128 Session Number (a) Agent fast

Cache-aware

10

4

10

4

10

3

10

3

4

8 16 32 64 128 256 Session Number (b) Agent medium

4

TensorCast

8 16 32 64 128 256 Session Number (c) Agent slow

Cache Hit Rate (%)

Figure 10: Request TTFT against different router policies with three levels of agentic coding workloads.

100 90 80 70 60 50 4

Load-aware

Load-aware + Mooncake Cache-aware TensorCast 100 100 90 90 80 80 70 70 60 60 50 8 16 32 64 128 4 8 16 32 64 128 256 50 4 8 16 32 64 128 256 Session Number Session Number Session Number (a) Agent fast (b) Agent medium (c) Agent slow

Figure 11: Request cache hit rate against different router policies with three levels of agentic coding workloads. sessions increases, the cache hit rate decreases for all methods due to limited local cache capacity on each instance and Mooncake overload. In contrast, the TensorCast router exhibits the smallest hit-rate drop by jointly balancing KV cache placement and instance load while preserving requests from the same session on cache-hitting instances. This behavior effectively reduces request TTFT. These results validate the programmability of TensorCast: developers can compose tensor lifecycle primitives to implement new optimization policies that coordinate request scheduling and tensor placement, without modifying existing serving components.

7

TensorCast with training frameworks like Megatron-LM [70] and DeepSpeed [71] for future work. Integration cost and abstraction overhead. Decoupling tensor management does not remove engine-specific integration: TensorCast confines the export, import, or transformation of engine-resident tensor states to an instance adaptor, so customized programs can evolve in callers without repeatedly modifying engines, transports, or storage backends. We do not claim that a general abstraction is inherently faster than every specialized implementation. §6.1–§6.3 evaluate reusable mechanisms and show that TensorCast achieves comparable or better performance, while §6.4 shows optimization enabled by composition.

Discussion 8

Applicability beyond inference workloads. While our integration and evaluations in §6 focus on inference workloads, TensorCast can also express tensor lifecycles in training workloads: At the abstraction level, checkpoints can be mapped to versioned, system-owned Artifacts, with view() and transform_into() expressing rank-specific resharding; gradients, optimizer states, and long-lived activations can be exposed as caller-leased tensors. We leave the integration of

Other Related Work

In addition to the studies for efficient tensor management discussed in §2.1, many other studies have investigated optimizations for LLM workloads. Efficient LLM inference. An increasing body of literature focuses on enhancing the efficiency of Transformer-based LLM inference. These optimizations encompass parallelism [72–74], request scheduling [75–77], memory management 14

[44, 78, 79], quantization [80, 81], and specialized GPU kernels [82, 83]. Serving as a complementary approach, TensorCast offers flexible, large-scale tensor management and integrates seamlessly with these existing techniques.

References [1] Kimi k3: Open frontier intelligence. https://www.kimi.com/blog/kimik3. (Accessed on 07/23/2026). [2] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-Latency serverless inference for large language models. In USENIX OSDI, 2024. [3] Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Baoquan Zhang, Shining Wan, Gengyuan Dan, et al. Deepflow: Serverless large language model serving at scale. arXiv e-prints, pages arXiv–2501, 2025. [4] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping. arXiv preprint arXiv:2306.03622, 2023. [5] Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, et al. {\lambda} scale: Enabling fast scaling for serverless large language model inference. arXiv preprint arXiv:2502.09922, 2025. [6] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. BlitzScale: Fast and Live Large Model Autoscaling with O (1) Host Caching. In USENIX OSDI, 2025. [7] Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. Hydraserve: Minimizing cold start latency for serverless llm serving in public clouds. arXiv preprint arXiv:2502.15524, 2025. [8] Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork. https://lmsys.org/blog/2025-12-10-rfork/. (Accessed on 01/28/2026). [9] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In USENIX FAST, 2025. [10] Anonymous. Dualmap: Enabling both cache affinity and load balancing for distributed LLM serving. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. under review. [11] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al. Lmcache: An efficient kv cache layer for enterprise-scale llm inference. arXiv preprint arXiv:2510.09665, 2025. [12] SGLang HiCache. https://lmsys.org/blog/2025-09-10-sglang-hicache/. (Accessed on 01/19/2026). [13] Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. In ACM EuroSys, 2025. [14] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In ACM EuroSys, 2025. [15] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-Efficient large language model serving for multi-turn conversations with CachedAttention. In USENIX ATC, 2024. [16] Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, et al. {ByteCheckpoint}: A unified checkpointing system for large foundation model development. In USENIX NSDI, 2025. [17] Moonshot checkpoint engine. https://github.com/MoonshotAI/chec kpoint-engine. (Accessed on 01/19/2026). [18] vLLM: The High-Throughput and Memory-Efficient inference and serving engine for LLMs. https://vllm.ai/. (Accessed on 01/19/2026).

Programmable LLM systems. Numerous studies explore the programming of LLM behaviors. One line of research targets frameworks for prompt engineering and agent orchestration, including LangChain [84], LangFlow [85], AutoGen [86], and DSPy [87]. These tools simplify agentic workflows to accelerate LLM application development. Another direction provides fine-grained control over the LLM inference process–encompassing tokenization, attention, KV cache management, and decoding–through systems like microserving LLM [88] and PIE [47]. However, existing programmable LLM systems operate primarily at the execution layer of LLM workloads. In contrast, TensorCast introduces programmable management of the data lifecycle (i.e., tensors), making it an orthogonal and complementary approach to current frameworks.

9

Conclusion Remarks

As LLM systems continue to evolve toward increasingly dynamic and stateful workloads, efficient tensor lifecycle management becomes a fundamental infrastructure challenge beyond individual optimization techniques. TensorCast introduces the missing tensor management layer by decoupling tensor states from the computation that produces and consumes them, enabling tensors to be managed as programmable, first-class system resources. Through lifecycle primitives and a distributed runtime, TensorCast achieves the efficiency of specialized tensor management mechanisms while allowing developers to compose new workload-specific policies. Our results demonstrate that future LLM infrastructure should not only optimize computation over tensors, but also provide programmable control over the lifecycle of the tensor states that drive these computations.

15

[19] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024. [20] Tensorrt-llm. https://nvidia.github.io/TensorRT-LLM/. (Accessed on 01/19/2026). [21] NVIDIA DGX SuperPOD: Next Generation Scalable Infrastructure for AI Leadership. https://docs.nvidia.com/dgxsuperpod/reference-architecture/scalable-infrastructureh200/latest/_downloads/bbd08041e98eb913619944ead1f92373/RA11336-001-DSPH200-ReferenceArch.pdf#page=8.10. (Accessed on 01/19/2026). [22] Redis – Your App is about to get faster. https://redis.io/. (Accessed on 01/22/2026). [23] Wenyuan Yu, Tao He, Lei Wang, Ke Meng, Ye Cao, Diwen Zhu, Sanhong Li, and Jingren Zhou. Vineyard: Optimizing data sharing in dataintensive analytics. Proc. ACM Manag. Data, 1(2), jun 2023. [24] Ray Plasma. https://github.com/ray-project/plasma. (Accessed on 01/22/2026). [25] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging ai applications. In USENIX OSDI 18, 2018. [26] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica. Resilient distributed datasets: A Fault-Tolerant abstraction for In-Memory cluster computing. In USENIX NSDI, 2012. [27] Garrett McGrath and Paul R Brenner. Serverless computing: Design, implementation, and performance. In 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), pages 405–410. IEEE, 2017. [28] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. Infless: a native serverless system for low-latency, high-throughput inference. In ACM ASPLOS, 2022. [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [30] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [31] Zhisheng Hu, Jiacheng Shen, and Ming-Chang Yang. Flexkv: Flexible index offloading for memory-disaggregated key-value store. arXiv preprint arXiv:2512.16148, 2025. [32] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In USENIX OSDI, 2024. [33] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118– 132. IEEE, 2024. [34] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024. [35] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. D\’ej\avu: Kv-cache streaming for fast, faulttolerant generative llm serving. arXiv preprint arXiv:2403.01876, 2024.

[36] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325–338, 2024. [37] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. In ACM SIGCOMM, 2024. [38] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems, 44(1):1–27, 2025. [39] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving. arXiv preprint arXiv:2403.19708, 52:20–38, 2024. [40] Megatron Team. Dist checkpointing package. https: //docs.nvidia.com/megatron-core/developer-guide/latest/apiguide/dist_checkpointing.html. (Accessed on 01/26/2026). [41] PyTorch Team. Getting started with Distributed Check- point (DCP). https://docs.pytorch.org/tutorials/recipes/distributed_checkpo int_recipe.html. (Accessed on 01/26/2026). [42] Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, and Minjia Zhang. Universal checkpointing: Efficient and flexible checkpointing for large scale distributed training. arXiv preprint arXiv:2406.18820, 2024. [43] Dingyan Zhang, Jinbo Han, Kaixi Zhang, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Jingren Zhou, and Rong Chen. Simple is better: Multiplication may be all you need for llm request scheduling. In USENIX OSDI, 2026. [44] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In ACM SOSP, 2023. [45] vllm – Automatic Prefix Caching. https://docs.vllm.ai/en/stable/desig n/prefix_caching/. (Accessed on 01/19/2026). [46] Fast and Expressive LLM Inference with RadixAttention and SGLang. https://lmsys.org/blog/2024-01-17-sglang/. (Accessed on 01/29/2026). [47] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging llm applications. In ACM SOSP, 2025. [48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. [49] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023. [50] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [51] Ruoyu Qin, Weiran He, Weixiao Huang, Yangkun Zhang, Yikai Zhao, Bo Pang, Xinran Xu, Yingdi Shan, Yongwei Wu, and Mingxing Zhang. Seer: Online context learning for fast synchronous llm reinforcement learning. arXiv preprint arXiv:2511.14617, 2025. [52] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint 16

arXiv:2501.12599, 2025. [53] Haoyuan Wu, Xueyi Chen, Rui Ming, Jilong Gao, Shoubo Hu, Zhuolun He, and Bei Yu. Totrl: Unlock llm tree-of-thoughts reasoning potential through puzzles solving. arXiv preprint arXiv:2505.12717, 2025. [54] David G Thaler and Chinya V Ravishankar. Using name-based mappings to increase hit rates. IEEE/ACM Transactions on networking, 6(1):1–14, 2002. [55] Leslie Lamport. Paxos made simple. ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), pages 51–58, 2001. [56] Diego Ongaro and John Ousterhout. In search of an understandable consensus algorithm. In USENIX ATC, pages 305–319, 2014. [57] Robbert Van Renesse and Fred B Schneider. Chain replication for supporting high throughput and availability. In USENIX OSDI, volume 4, 2004. [58] Jeff Terrace and Michael J Freedman. Object storage on craq: Highthroughput chain replication for read-mostly workloads. In USENIX ATC, 2009. [59] DuckDB – Run analytics where your data lives. https://duckdb.org/. (Accessed on 06/24/2026). [60] EunYoung Jeong, Shinae Wood, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. mtcp: a highly scalable user-level tcp stack for multicore systems. In USENIX NSDI, 2014. [61] JuiceFS. https://github.com/juicedata/juicefs. (Accessed on 06/24/2026). [62] InstentTensor. https://github.com/scitix/InstantTensor. (Accessed on 06/24/2026). [63] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 3119–3137, 2024. [64] Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, and Mingxing Zhang. Prefill-as-a-service: Kvcache of next-generation models could go cross-datacenter. arXiv preprint arXiv:2604.15039, 2026. [65] Sglang model gateway. https://docs.sglang.io/docs/advanced_feature s/sgl_model_gateway. (Accessed on 07/12/2026). [66] Michael Mitzenmacher. The power of two choices in randomized load balancing. IEEE transactions on parallel and distributed systems, 12(10):1094–1104, 2002. [67] Swe-gym/openhands-sampled-trajectories dataset. https://huggingf ace.co/datasets/SWE-Gym/OpenHands-Sampled-Trajectories. (Accessed on 07/12/2026). [68] Lawrence Brown, Noah Gans, Avishai Mandelbaum, Anat Sakov, Haipeng Shen, Sergey Zeltyn, and Linda Zhao. Statistical analysis of a telephone call center: A queueing-science perspective. Journal of the American statistical association, 100(469):36–50, 2005. [69] R Dean Malmgren, Daniel B Stouffer, Adilson E Motter, and Luís AN Amaral. A poissonian explanation for heavy tails in e-mail communication. Proceedings of the National Academy of Sciences, 105(47):18153– 18158, 2008. [70] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multibillion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [71] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,

pages 3505–3506, 2020. [72] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and {Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 559–578, 2022. [73] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15. IEEE, 2022. [74] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In ACM SOSP, 2024. [75] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023. [76] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In USENIX OSDI, 2022. [77] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In USENIX OSDI, 2024. [78] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management. In USENIX OSDI, 2024. [79] Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. Jenga: Effective memory management for serving llm with heterogeneity. In ACM SOSP, 2025. [80] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023. [81] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, WeiChen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100, 2024. [82] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with ioawareness. Advances in neural information processing systems, 35:16344–16359, 2022. [83] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005, 2025. [84] LangChain: Build context-aware reasoning ap- plications. https://gith ub.com/langchain-ai/langchain. (Accessed on 02/15/2026). [85] Langflow: Low-code AI builder for agentic and RAG applications. https://www.langflow.org/. (Accessed on 02/15/2026). [86] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 3(4), 2023. [87] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative 17

[88] Hongyi Jin, Ruihang Lai, Charlie F Ruan, Yingcheng Wang, Todd C Mowry, Xupeng Miao, Zhihao Jia, and Tianqi Chen. A System for Microserving of LLMs. arXiv preprint arXiv:2412.12488, 2024.

language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations, 2024.

18

A

Algorithm 1: Router Request Rebalancing Policy.

As shown in Alg. 1, the programmable router in §6.4 runs a simple rebalancing policy periodically. At each tick, it first updates an exponentially weighted moving average (EWMA) load estimate for each instance using the current queue depth, including both waiting and running requests. It then selects the most-loaded instance as the source and the least-loaded instance as the target. A rebalance is triggered only when both the absolute load gap and the relative load ratio exceed predefined thresholds, 𝜃 𝑎𝑏𝑠 and 𝜃 𝑟𝑒𝑙 , thereby avoiding unnecessary migrations when the cluster is already balanced. When rebalancing is needed, the policy considers only eligible requests on the source instance, excluding requests that are currently running or already under migration. Each eligible request is scored using three factors: request length in tokens, which estimates the potential KV reuse benefit; recency, which favors recently active requests; and a history penalty, which discourages repeatedly migrating the same request and avoids ping-pong behavior. The request with the highest score is selected as the migration candidate from the source to the target.

For Each Rebalance Tick: 2 foreach instance 𝑖 do 3 𝐿𝑖 ← 𝛼 · Load(𝑖) + (1 − 𝛼)𝐿𝑖 ⊲ Update instance load using EWMA 4 𝑠𝑟𝑐 ← arg max𝑖 𝐿𝑖 5 𝑑𝑠𝑡 ← arg min𝑖≠𝑠𝑟𝑐 𝐿𝑖 6 𝑔𝑎𝑝 ← 𝐿𝑠𝑟𝑐 − 𝐿𝑑𝑠𝑡 7 𝑟𝑎𝑡𝑖𝑜 ← 𝐿𝑠𝑟𝑐 /max(𝐿𝑑𝑠𝑡 , 1) 8 if 𝑔𝑎𝑝 < 𝜃 𝑎𝑏𝑠 or 𝑟𝑎𝑡𝑖𝑜 < 𝜃 𝑟𝑒𝑙 then 9 return ∅ ⊲ No rebalance 1

10 𝑅 ← EligibleRequests(𝑠𝑟𝑐)

⊲ Filter out running or

migrating requests 11 if 𝑅 = ∅ then 12 return ∅ foreach req 𝑟 ∈ 𝑅 do 𝑎𝑐𝑡𝑖𝑣𝑒𝑟 ← 𝑒 − (𝑛𝑜𝑤−𝑙𝑎𝑠𝑡 _𝑎𝑐𝑡𝑖𝑣𝑒𝑟 )/𝐻 15 𝑝𝑒𝑛𝑎𝑙𝑡𝑦𝑟 ← 1/(1 + 𝜆 · 𝑚𝑖𝑔𝑟𝑎𝑡𝑖𝑜𝑛𝑠𝑟 ) 16 𝑠𝑐𝑜𝑟𝑒𝑟 ← 𝑡𝑜𝑘𝑒𝑛𝑟 · 𝑎𝑐𝑡𝑖𝑣𝑒𝑟 · 𝑝𝑒𝑛𝑎𝑙𝑡𝑦𝑟

13

14

∗ ← arg max 𝑟 ∈𝑅 𝑠𝑐𝑜𝑟𝑒𝑟 ∗ 18 return (𝑟 , 𝑠𝑟𝑐, 𝑑𝑠𝑡)

Router Rebalancing Policy

17 𝑟

⊲ Migrate one session

19

Related documents

• TensorHub: Rethinking AI Model Hub with Tensor-Centric Compression#120482
• From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning#282788
• Direct Model State Migration for Elastic Training of Large Language Models#343431
• BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training#925514
• Relation-Aware Graph Foundation Model#1129079