RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

arXiv:2611.00967v1 [cs.OS] 31 Oct 2026

RTP-LLM: B UILDING AN E VOLVABLE AND R ELIABLE LLM I NFERENCE E NGINE AT A LIBABA S CALE Jiarui Guo * 1 2 Zhi Zhang * 2 Boyu Tan * 2 3 Zongwei Lv 1 Haobo Sun 2 3 Na He 2 Ziqing Wang 2 Xinfei Shi 2 Wei Zhang 2 Xi Yang 2 Haosheng Hu 2 Wending Bao 2 Zetao Hu 2 Dong Cao 2 Yong Li 2 Lining Wang 2 Chengjun Yang 2 Tianpeng Wang 2 Huanbing Lu 2 Yidan Wang 2 Hengyi Bao 2 Yaxin Yu 2 Hanyun Guo 2 Huidong Zhang 2 Haiyu Zhou 2 Zefeng Liu 2 Weixiang Huang 2 Ziyi Wang 2 Chenyi Song 2 Ruixuan Zhang 2 Zhengkai Ao 2 Haocheng Luo 2 Zhichen Xue 2 Jialin Zhang 2 Chi Zhang 2 Lingjun Zhu 2 Silu Zhou 2 Zongyuan Wu 2 Yinghao Yu 2 Yi Zheng 2 Miao Sun 2 Yuxing Yuan 2 Juncheng Yin 2 Guoding Li 2 Nan Wang 2 Xiyu Wang 2 Lin Yang 2 Zechao Zhang 2 Jianning Zhang 2 Bo Cai 2 Xiaolin Wang 1 Tao Lan 2 Tong Yang 1 Kan Liu 2 Lu Chen 3

A BSTRACT Large language models and inference techniques evolve rapidly, making model enablement a recurring production task rather than a one-time integration effort. At Alibaba, a shared inference infrastructure must keep pace with new models and serving capabilities while remaining reliable across heterogeneous production environments. To reconcile rapid evolution with production reliability, we present RTP-LLM, an open-source production inference engine developed and deployed at Alibaba. Its architecture separates model-specific execution from shared request processing, state management, and resource coordination. Explicit interfaces localize compatible model changes while allowing models and deployment configurations to reuse common serving mechanisms. RTP-LLM has evolved from an exploratory system into a shared production engine that supports ∼100 models and processes hundreds of trillions of tokens per day. We describe how its architecture and production practices accommodate model evolution and contain the impact of failures, and share lessons for building evolvable and reliable inference engines at enterprise scale. 1

I NTRODUCTION

In recent years, large language models (LLMs) have advanced at a remarkable pace (Team et al., 2025; OpenAI et al., 2023). Research institutions and technology companies continue to release open models spanning diverse parameter scales, architectures, and modalities (Jiang et al., 2023; DeepSeek-AI et al., 2024a; Wang et al., 2024). Even within a single model family, new checkpoints and variants are released frequently (Hui et al., 2024). In parallel, techniques for LLM inference continue to evolve, including low-bit quantization (Xiao et al., 2023; Lin et al., 2024), speculative decoding (Leviathan et al., 2023), diverse parallelization strategies (Narayanan et al., 2021; Rajbhandari et al., 2022; Liu et al., 2024a) and prefill-decode disaggregation (Zhong et al., 2024; Patel et al., 2024). Model enablement must therefore accommodate changes in both model computation and the techniques used to execute it. * Equal contribution 1 Peking University 2 Alibaba Group Zhejiang University. Correspondence to: Tong Yang <yangtong@pku.edu.cn>, Kan Liu <liukan.lk@alibaba-inc.com>, Lu Chen <luchen@zju.edu.cn>. 3

Preprint. November 3, 2026.

However, a publicly released model checkpoint does not by itself constitute a production-ready service; it must be integrated into an inference engine and qualified under production performance and reliability requirements (Yu et al., 2022; Kwon et al., 2023). At Alibaba, this task is amplified by the breadth and heterogeneity of its businesses. A shared inference infrastructure serves domestic and international commerce, logistics, digital mapping, and local consumer services, including Alibaba Cloud, Taobao, Tmall, Xianyu, Cainiao, Amap, Ele.me, AliExpress, and Lazada. These businesses differ in their models, hardware platforms, deployment configurations, workload characteristics, and service-level objectives. Model adaptation is therefore not an isolated compatibility exercise for a single application, but a recurring production requirement through which new model capabilities are made available across the company. Supporting this range of workloads through a common inference foundation imposes two competing requirements. New models and serving capabilities must be incorporated rapidly, while existing services must retain correctness, predictable performance, and availability through frequent changes. Insufficiently isolated modifications can disrupt

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

unrelated deployments, whereas interfaces made too rigid in pursuit of stability can make each new model or feature costly to integrate. The central challenge is therefore not merely to support many models or maximize peak performance, but to evolve continuously without compromising the reliability of existing services.

• We share experiences and lessons from the long-term development and operation of RTP-LLM. We analyze critical design trade-offs and provide practical guidelines for building evolvable and reliable production inference engines.

Existing inference frameworks have made substantial advances in performance, model coverage, and serving functionality (Kwon et al., 2023; Zheng et al., 2024; Nvidia, 2023b). However, adopting them for Alibaba’s production environment still requires substantial local validation and integration. First, upstream validation cannot cover every combination of workloads, hardware, and operating conditions encountered at Alibaba. In our experience, some correctness, cache-consistency, and overload-related bugs emerge only under sustained traffic, heterogeneous hardware, or fleet-scale deployment, and may remain invisible in functional tests and small-scale benchmarks. Also, the broad feature and component surface of a general-purpose engine may further expand the code that must be integrated, validated, and maintained, even when only a subset is used internally. Second, a deeply customized fork can address local requirements, but the necessary changes may span scheduling, cache management, hardware backends, and other shared components. As the upstream project evolves, these modifications must be repeatedly reconciled with upstream changes, creating persistent integration effort, regression risk, and maintenance cost. These trade-offs motivated our decision to own the engine’s architectural evolution and production validation.

2

BACKGROUND

2.1

From Model Release to Production Service

To realize this objective, we present RTP-LLM, an opensource production inference engine developed and deployed at Alibaba. Its central design principle is to separate rapidly changing model execution from relatively stable serving control. Explicit interfaces keep compatible model-specific changes within the execution layer while allowing serving mechanisms to be reused across models and deployment configurations. Our major contributions are summarized as follows: • We design, implement, and open-source RTP-LLM (Tan et al., 2026), a production inference engine that separates model execution from stateful serving control.1 We present its extensible model-execution layer and shared runtime mechanisms for request processing, state management, and resource coordination. • We report the large-scale deployment and continuous evolution of RTP-LLM at Alibaba. It serves multiple business domains and has supported successive generations of models, accelerator platforms, and serving capabilities. 1 The source code of RTP-LLM is available at https:// github.com/alibaba/rtp-llm.

A model release typically provides a checkpoint, model configuration, tokenizer, and reference implementation. These artifacts specify the model computation but not how it should operate as an online service. An inference engine (Kwon et al., 2023; Zheng et al., 2024; Nvidia, 2023b; DeepSeek-AI, 2026) bridges this gap by executing the model, coordinating parallel computation, scheduling and batching requests, managing KV caches and accelerator resources, and returning outputs under performance and reliability requirements. Based on our production experience at Alibaba, we organize model enablement into six readiness states. A model progresses from available, to correctness-ready when its outputs satisfy defined comparisons against a reference implementation, and then to performance-ready when it satisfies workload-specific latency, throughput, and resourceefficiency targets. It becomes integration-ready after connecting to the surrounding service infrastructure, canaryready when it can safely receive restricted production traffic, and finally production-ready when it can sustain production traffic under the required service-level objectives. Although validation activities may overlap and failures may trigger iteration, these states represent progressively stronger readiness gates. Throughout this paper, model enablement refers to this complete transition, rather than merely the successful execution of a model checkpoint. 2.2

Evolution towards RTP-LLM

The current architecture of RTP-LLM emerged through three stages, each prompted by assumptions that no longer held as models and production requirements evolved. Stage I: FasterTransformer-based exploration. The earliest system was developed between March 2023 and June 2024. Throughout this stage, the models targeted by the system were predominantly dense, decoder-only Transformers with largely similar attention structures, differing mainly in components such as positional encoding and feed-forward networks (Touvron et al., 2023; Bai et al., 2023). It used FasterTransformer’s C++/CUDA model implementations as its execution backend (Nvidia, 2023a), while a Python service layer handled tokenization, request scheduling, streaming responses, health checks, and request-level KV-cache lifecycle management. This design was sufficient for early

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

exploration, but it became inadequate as the system expanded. On some early workloads, the Python service layer incurred 10–20 ms of host-side overhead per model invocation. This overhead became significant when GPU execution was short. The model implementation inherited from FasterTransformer was also difficult to extend to new accelerator platforms, parallel configurations, and model structures. Stage II: C++-first integration. Starting in June 2024, RTPLLM moved batching, scheduling, and KV-cache management into C++ and replaced the inherited FasterTransformer backend with a C++-first model-execution path. Compared with the earlier Python service layer, this design substantially reduced host-side serving overhead. It also introduced a device abstraction through which the model-forward path selected backend-specific operators, enabling a common runtime to support heterogeneous accelerators. During its subsequent evolution, it incorporated optimized attention kernels, MoE parallelism, and prefill–decode disaggregation (Dao et al., 2022; Dao, 2024; DeepSeek-AI et al., 2024b; Zhao et al., 2025; Zhong et al., 2024). However, its C++-first organization became increasingly costly as new models and operators were primarily released as Python/PyTorch implementations in the Hugging Face ecosystem (Paszke et al., 2019; Wolf et al., 2020). Repeatedly translating modelspecific semantics into C++ made adaptation time difficult to predict, complicated the reuse of native operators, and increased maintenance effort, eventually motivating another architectural revision. Stage III: Decoupled model execution and serving mechanisms. In June 2025, the current RTP-LLM architecture moved model execution and model-specific semantics to Python and PyTorch, while retaining batching, scheduling, KV-cache management, and resource coordination in a shared high-performance C++ runtime. CUDA Graphs capture and replay repeated GPU execution (Nvidia, 2026), reducing host-side launch overhead while model logic remains in Python/PyTorch. This separation allows new model structures and operators to be expressed closer to their reference implementations, while reusing common serving mechanisms across models and deployments (GLM-5-Team et al., 2026; DeepSeek-AI et al., 2026; Kimi Team et al., 2026). The boundary is not immutable: model changes that alter shared state or resource requirements may still require coordinated runtime changes. Nevertheless, RTP-LLM has continued to incorporate new models and serving capabilities without another fundamental architectural redesign.

3

RTP-LLM S YSTEM OVERVIEW

Figure 1 shows the main components of RTP-LLM: a Frontend, a Master, a pool of inference workers, and a multi-tier KV cache subsystem. The Frontend provides the application-facing service interface, while the Master coor-

Multi-tier KV cache

Master

FlexLB Worker address

Request metadata

Client

Preprocess tokenize

Tokens & metadata

GPU memory Local memory Remote cache 3FS Load KV

Prepare cache / inputs Model computation

Stream response

Frontend

Generated output

Next iter.

Update progress

Workers

Figure 1. Request execution in RTP-LLM.

dinates request placement across the worker pool. Workers provide model execution capacity and maintain the execution context of the requests assigned to them. The cache subsystem supports the workers’ state requirements through a storage hierarchy spanning worker-local memory and external storage. For an autoregressive generation request, the Frontend first preprocesses the input and tokenizes the prompt. It sends the prompt length and cache keys to the Master. Flexible Load Balance (FlexLB), a component of the Master, uses cache-aware placement to select execution destinations, considering request characteristics, prefix-cache reuse, and worker load. After receiving the worker addresses, the Frontend submits the tokenized input and generation parameters directly to the selected workers. Each worker queues incoming requests and uses local scheduling to select the work for each model invocation. The multi-tier KV cache spans GPU memory, local memory, remote cache, and 3FS. Before processing the prompt, the runtime identifies reusable state for matching prefixes and loads it into GPU memory when needed. It allocates space for new state and prepares token inputs, sequence lengths, and cache-block mappings for the scheduled requests. The initial computation processes the uncached prompt tokens, establishing the state used by subsequent generation steps. Generation then advances through successive model invocations. After each invocation, the worker updates generated tokens and request progress and checks whether generation has completed. Unfinished requests continue with their accumulated state; their execution inputs are rebuilt for the next invocation, with additional state storage allocated as needed. As results become available, the Frontend converts the returned tokens into response content and streams it to the client until the request completes. Within this organization, workers can use different execution configurations to meet model requirements and deployment objectives. Configuration choices determine how model computation is distributed across devices and how the stages of request execution are assigned to workers. For example, initial prompt processing (prefill) and iterative token generation (decode) can run on the same worker or on separate

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale Model Execution

Parallel & Distributed Context

Checkpoint

Model Definition

weight rules

Weight Loader

Model Components Attention MoE

state requirements

Parallel config Process groups

parallel context

Linear

Shared Ops / Backends inputs / state

partition

Collectives

results

Model Wrapper

Serving Runtime

Cache Manager

Scheduler

Request Management

Common Utilities

IO / Protocol

Logging

Metrics / Monitoring

Figure 2. Model–serving separation of RTP-LLM.

worker groups. Table 2 in Appendix A provides a fuller inventory of supported capabilities and configurations.

4

These model-side changes connect to serving through a common model invocation interface. A shared model wrapper translates token and position inputs and sequence metadata into the inputs required by the Python model, including multimodal features when present. For ordinary autoregressive generation, the model returns hidden states; the wrapper applies the output projection and prepares logits for the common sampling path. When its invocation requirements fit this interface and its state requirements are supported by the runtime, changes to computation, weights, or operators can remain within the model and its associated components. The serving runtime continues to invoke the model through this interface, so those changes do not require rewriting request scheduling or progress tracking.

E NABLING C ONTINUOUS E VOLUTION

Continuous model integration requires controlling how changes propagate through the serving system. A new model can change computation and checkpoint layouts, as well as the state and distributed context needed for execution. As shown in Figure 2, RTP-LLM places computation and parameter interpretation in model-side components, while request coordination and resource management remain in the shared serving runtime. Explicit interfaces connect these responsibilities, allowing model-specific changes to remain local where possible and enabling different models to reuse common serving mechanisms. 4.1

ments, even when tensor shapes are unchanged.

Localizing Model-Specific Changes

Changes to model computation enter through Python/PyTorch definitions and components, keeping layer structure and operator choice separate from request coordination. A model definition specifies the forward computation and uses the shared operator library (Ops) through these components. Backend implementations register with component factories and expose support conditions on weights and execution configurations. The factories use these conditions to select an implementation. A hardware-specific kernel can therefore be added to the component library and reused by compatible models, while a new computation may require a dedicated component. In both cases, the extension is organized around model computation, without introducing a separate request-serving path for each implementation. Changes to checkpoint layouts are handled through modelsupplied weight descriptions and conversion logic. The descriptions map checkpoint tensors to internal parameters and specify required combinations or transformations. The common loader applies these rules together with the selected parallel and numerical configuration to prepare executable weights. This keeps parameter interpretation with the model while reusing the loading procedure. The physical layout of prepared weights must match the selected kernel’s require-

4.2

Reusing the Serving Runtime across Models

Reusing the serving runtime requires more than a common model invocation: the runtime also needs to know what resources execution requires. Models describe their state requirements and consume a shared parallel configuration defining their distributed context. These descriptions and configurations let the runtime accommodate different model requirements through shared state-management and distributed-initialization mechanisms. Persistent-state management illustrates how this reuse works. The model supplies per-layer descriptors of state types and layouts, together with supported capacity, retention, and reuse policies. Allocation and reuse depend on these policies as well as on state layout. Configuration builders combine these descriptions with execution parameters to form state groups, and common cache-management interfaces handle their allocation and reclamation. Allocated cache views are bound at model initialization; each invocation supplies sequence metadata and per-group block mappings that locate the current requests’ state. Models with different combinations of supported state types can therefore share allocation and lifecycle-management mechanisms through their descriptions. Cache connectors use resource metadata from the same organization; models with layouts supported by a connector can also reuse its cache lookup and transfer path. Parallel execution makes the distributed environment another point of reuse across models. Common startup and communication utilities establish process groups from the shared parallel configuration and provide collective operations. Model definitions and their components determine how these operations are combined with computation, allowing model-specific execution strategies to evolve while reusing common distributed initialization and communication support.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

However, such reuse is limited to requirements that the runtime can already express and support. A new state type or execution pattern may require changes to both the model’s descriptions and the shared runtime implementation; exposing a new descriptor does not by itself provide the corresponding execution or resource-management support. Such extensions incorporate the new requirements into shared serving mechanisms, preserving a common request lifecycle as supported models evolve. Appendix B illustrates this process through DeepSeek-V4 integration, where new model computation and heterogeneous state requirements led to extensions on both sides of the interface.

5

P RESERVING P RODUCTION R ELIABILITY

Reusing a serving runtime across models also makes its reliability a shared concern. Reliability must be maintained when requests acquire and release resources, while service instances fail or experience changing demand, and as shared components evolve. RTP-LLM addresses these concerns at three levels: state and resource protection within a worker, service availability and diagnosis across instances, and regression validation during evolution. 5.1

State and Resource Protection

Within a worker, request execution proceeds through scheduling, input and state preparation, model computation, and progress and output updates. Unfinished requests continue through further iterations, while completion or failure initiates cleanup. Autoregressive requests retain execution state across successive model invocations, so resource use extends beyond the work selected for a single iteration. The KV-cache subsystem stores this persistent state, whose footprint can grow as a request progresses. Before model execution, RTP-LLM constrains resource acquisition through scheduling and allocation checks. The worker-local scheduler bounds batch size and token work and, when configured, the number of requests with initialized KV resources. The cache manager checks block availability when requests first acquire state and as that state grows. Scheduling limits control current work; allocation checks enforce availability for the state that work requires. Requests can wait for scheduling eligibility, while an attempted allocation that fails at initialization or during growth terminates the affected request with an allocation error. During input and state preparation, the runtime checks data readiness. Storage ownership keeps buffers live, while completion tracking determines when their contents are usable. Cache transfers can be submitted asynchronously, so submission and publication are treated as distinct events. Transfer contexts retain the storage they access, and destination entries become reusable only after a successful copy. Before

Master

Requests Frontend

Gateway

Concurrency cap

Request records

FlexLB Queue limit (optional)

Cluster 1 · Site A Ingress

Place

Cluster 2 · Site B

Service replicas

Service replicas

worker worker

worker worker

Scale

Scale

k8s master

k8s master

OSS

Ingress

Request records

Models

Diagnosis

Figure 3. RTP-LLM deployment across clusters and sites.

execution consumes loaded state, the request checks that its load has completed successfully. These dependencies allow data movement to overlap with other work while preserving the ordering between production and consumption. During model execution and progress updates, asynchronous operations can continue using a request’s resources. For decode-only batches in non-speculative execution, optional asynchronous output processing uses device events and explicit waits to preserve dependencies between computation, output transfer, and request-state updates. When the request terminates, reclamation must preserve these completion dependencies. Completion and error use a common cleanup path, with guards against repeated release. Cleanup tracks outstanding output tasks and defers KV release until they complete. A timeout can therefore stop response production while pending local work finishes safely. State protection also extends beyond the originating request. Prefix reuse lets later requests share retained state, and asynchronous transfers can hold blocks independently of active computation. RTP-LLM separately tracks references held by requests, cache connectors, and retained cache entries. A block becomes available for unrelated allocation only after all references are released. Evicting a prefix releases its retained-cache reference without overriding ownership by active requests or transfers. Separating retention policy from active ownership allows cache-retention policies to change without invalidating ongoing work. Keeping allocation, ownership tracking, and cleanup in the shared runtime carries these protections to models using these shared statemanagement mechanisms. 5.2

Operational Reliability

Service availability depends on both worker capacity and the request-entry and coordination paths that make it reachable. Figure 3 illustrates a typical deployment of RTPLLM supporting Alibaba’s production services. The deployment combines redundant instances of the same service with placement across multiple clusters and sites. The Master uses FlexLB for request placement, while Gateway

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

handles request forwarding; Ingress supports service routing within a cluster, and Kubernetes supports deployment control. Worker replicas provide alternative capacity for subsequent requests after an instance failure, while placement across clusters and sites limits exposure to shared infrastructure failures. Replication of Gateway and Ingress protects request access from individual instance failures, while Kubernetes control-plane replication protects deployment management. To protect response latency as demand changes, Kubernetes manages worker replica scaling within a cluster. Scaling decisions can use time to first token (TTFT) and time per output token (TPOT) alongside input and output tokens per minute (TPM), relating response latency to the token workload handled by the service. These signals can guide adding workers when capacity is insufficient and reducing capacity under lower demand while maintaining latency objectives. Admission limits and resource-aware placement complement scaling by constraining pressure on the available worker pool. The Frontend enforces a configurable concurrency limit at request admission; when FlexLB queueing is enabled, its queue limit bounds requests awaiting placement. FlexLB makes a worker selectable only after initializing its scheduling state and excludes workers reported as unavailable. Within the available worker pool, resourceaware placement considers the work and state a request will require, with cache-aware policies also accounting for reusable prefixes. Placement estimates can include work already assigned but not yet reflected in worker reports, reducing the risk of overcommitting workers between updates. Worker-side allocation checks verify concrete resource availability during execution, complementing the estimates used for request placement. Operational diagnosis supports recovery by locating request failures and latency degradation. Request and error rates, response latency, queue length, and admission rejection expose failures and growing service pressure. The Frontend forwards request identifiers and, when present, trace identifiers with generation requests to workers. Access logs and execution-stage timings provide local evidence about slow or failed requests. In production, distributed tracing uses the propagated identifiers to correlate these records along the execution timeline of a request. This timeline lets operators locate delays and reported failures across the components that handle the request. Output behavior provides a complementary view of service reliability. An optional incremental monitor in the Frontend detects repeated token patterns. Tool-call loop detection uses configured output-format markers. When enabled, these checks report metrics and retain request context in logs. Tool-call parsing diagnostics provide complementary

evidence. These signals make output anomalies observable alongside request failures and latency, supporting investigation even when requests complete successfully. 5.3

Regression Validation during Evolution

Changes to a shared loader, operator, or serving component can affect configurations beyond the model that motivated them. Regression validation must therefore cover the existing configurations that depend on the changed component. The infrastructure combines component tests, end-to-end inference tests, and performance comparisons, with cases specifying model and execution settings and hardware profiles associating them with device types and counts. For each shared-component change, the team selects regression configurations according to the affected models, numerical formats, parallel layouts, and serving paths. Broader regression is performed before a production release to check interactions beyond the initial development configuration. Validation checks behavior, performance, and resource safety under explicit conditions. Behavioral comparisons specify request inputs, sampling settings, reference outputs, and tolerances; in validation mode, request and comparison errors propagate as failures. Performance comparisons require comparable workload and execution settings across versions. Resource-lifecycle tests check repeated allocation and release, cancellation and timeout states, and the ordering of completion visibility and reclamation, using expected block availability and release state. Together, these checks evaluate observable results and the resource-management guarantees that changes to the shared runtime must preserve. Validation also accounts for whether the intended comparisons were completed. Completed results are interpreted with their case identities, hardware settings, and workload parameters. The continuous integration system (CI) preserves reported failures. Regression and production validation check report completeness and verify that the expected performance parameter points are present, while regression records also account for absent baselines and invalid measurements. When the test infrastructure changes, coverage audits compare case identities, hardware assignments, and parameter settings. This ties the coverage claimed for a shared-runtime change to completed comparisons on the configurations. The release process includes resource-lifecycle checks. A candidate that passes the required regression then enters a limited production rollout, where error rates, output abnormalities, latency, and resource usage are compared with the serving version. Regressions pause further rollout and can trigger a return to the previous version. This connects regression validation to the operational signals used to protect existing services as shared components evolve.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale Table 1. Code changes for model enablement in RTP-LLM.2 Model DeepSeek-V4 GLM-5.2 MiniMax-M3 Qwen3.5

6

Model layer

Serving layer

19,749 (3.1671%) 423 (0.0678%) 10,448 (1.6755%) 884 (0.1418%)

5,207 (0.8350%) 38 (0.0061%) 91 (0.0146%) 2 (0.0003%)

D EPLOYMENT

RTP-LLM is deployed as part of Alibaba’s shared inference infrastructure, serving production applications across domestic and international commerce, logistics, digital mapping, and local consumer services. As of September 2026, RTP-LLM serves 6,337 business units and supports ∼100 models, with deployments spanning multiple clusters and processing hundreds of trillions of tokens (input and output) per day. Across successive engine versions, the system’s development and production use span more than three years, and RTP-LLM directly or indirectly serves hundreds of millions of users through the applications it supports. Production workloads differ in inference task, input modality, prompt and output lengths, and traffic patterns. They are served by models of different architectures and parameter scales running on heterogeneous accelerator platforms. Parallel configurations and serving capabilities are selected to match the available hardware and each business’s latency, throughput, and availability requirements.

7

E VALUATION

In this section, we draw on data from model enablement and production operation at Alibaba to answer the following questions about RTP-LLM: 1. How does it accommodate new model requirements? 2. How does it perform on production workloads? 3. How reliable is it in long-term production use? 7.1

Model Enablement

To assess how RTP-LLM supports model evolution, we examine the code changes for initial functional support of DeepSeek-V4, GLM-5.2, MiniMax-M3, and Qwen3.5. Table 1 reports the number of revised production-source lines, divided between the model and serving layers in Figure 2. For all four models, most changes are concentrated in the model layer and implement new computation, checkpoint interpretation, and supporting operators. For example, 2

Each percentage is the ratio of lines changed for that model in the corresponding layer to RTP-LLM’s total production-source line count.

MiniMax-M3 adds sparse attention and the associated support for quantized weights and operators, while DeepSeekV4 introduces new attention and residual computations and native FP8/MXFP4 weight loading. However, some models can be adapted using components already developed for earlier models. For example, GLM-5.2 reuses GLM-5’s MLA and expert components, while Qwen3.5 builds on Qwen3Next’s linear-attention components. These models thus require relatively few model-layer changes. Serving-layer changes are relatively small compared with model-layer changes across all four adaptations. GLM-5.2, MiniMax-M3, and Qwen3.5 each involve fewer than 100 changed serving-layer lines. They reuse the existing request scheduler and cache manager by supplying model-specific state layouts and configurations to the common allocation, reuse, and reclamation mechanisms. For DeepSeek-V4, we add the missing heterogeneous-state support by extending shared allocation, reuse, and cross-worker transfer mechanisms while retaining the common request-serving workflow. Even with these extensions, its serving-layer changes remain substantially smaller than its model-layer changes. Therefore, RTP-LLM’s design supports continuous model integration while keeping the serving layer largely stable. When new state or execution requirements demand runtime changes, the system extends shared mechanisms while preserving a common request-serving workflow across models. To further demonstrate RTP-LLM’s evolvability, we also report the model-enablement timelines in Appendix C, showing that these models reached production readiness in a short time after release. 7.2

Production Serving Performance

We examine three production applications: chat, an internal coding agent, and intelligent customer service, each served by a different model in a separate production cluster. Each set of measurements is collected from a single cluster over seven days to capture the recurring daily traffic cycle. Figures 4–7 present hourly observations of request volume, mean time to first token (TTFT), mean time per output token (TPOT), and mean cache hit rate. We observe recurring daily cycles in business demand, with request volumes rising during the daytime and falling late at night. Across these changing loads, the reported TTFT remains below 2 seconds and TPOT below 20 milliseconds for all three applications, and the services also meet their respective request-level SLOs in production over the observation period. We also observe cache reuse in all three applications, with hit rates of up to 80–90%. By reusing previously computed context, RTP-LLM avoids repeated computation for requests with shared prefixes and reduces the computational load of production serving.

400K 200K 0K

0

1

2

3

4

5

6

7

Requests/hour

600K

Requests/hour

Requests/hour

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale 150K 100K 50K 0K

0

1

2

Days

3

4

5

6

150K 100K 50K 0K

7

0

1

2

Days

(a) Chat

3

4

5

6

7

Days

(b) Coding agent

(c) Intelligent customer service

Figure 4. Request volume over seven days.

1500 1000 500

0

1

2

3

4

5

6

1500 1000 500 0

7

800

TTFT (ms)

2000

TTFT (ms)

TTFT (ms)

2000

0

1

2

Days

3

4

5

6

600 400 200 0

7

0

1

2

Days

(a) Chat

3

4

5

6

7

Days

(b) Coding agent

(c) Intelligent customer service

Figure 5. Mean time to first token (TTFT) over seven days.

12 9 6

0

1

2

3

4

5

6

16 14 12 10

7

20

TPOT (ms)

18

TPOT (ms)

TPOT (ms)

15

0

1

2

Days

3

4

5

6

15 10 5

7

0

1

2

Days

(a) Chat

3

4

5

6

7

Days

(b) Coding agent

(c) Intelligent customer service

80 70 60 50

0

1

2

3

4

5

6

7

90 80 70 60

0

1

2

(a) Chat

3

4

5

6

7

Days

Days

(b) Coding agent

Cache hit rate (%)

90

Cache hit rate (%)

Cache hit rate (%)

Figure 6. Mean time per output token (TPOT) over seven days. 90 80 70 60 50 40

0

1

2

3

4

5

6

7

Days

(c) Intelligent customer service

Figure 7. Mean cache hit rate over seven days.

Although applications differ in characteristics and traffic patterns, RTP-LLM allows each business to select a suitable model. The shared runtime then provides responsive execution and effective cache reuse, meeting practical requirements with high performance. Supporting these choices through a common runtime therefore combines evolvable model support with reliable production performance. 7.3

Long-Term Service Reliability

Figures 8(a) and 8(b) track a model-as-a-service (MaaS) offering over approximately 80 days, using measurements collected from a single production cluster. The vertical dashed lines mark version updates, which typically involve bug fixes, performance tuning, inference optimizations, or improvements to diagnostic logging and system stability. As

the service gains broader adoption, its daily request volume grows while the request error rate declines across successive updates. Some fixes markedly reduce errors associated with known bugs on the day of deployment, and the error rate remains below 0.01% on nearly all days in the later part of the observation period. These records illustrate RTP-LLM’s ability to evolve while improving production reliability. Figure 8(c) summarizes reported failure modes among failed requests over the same period. Requests may fail because a worker cannot be located or reached, an established connection is reset, or a remote procedure call (RPC) times out; these cases account for 40.9%. Reported execution errors and exceeded generation deadlines form another major share (34.6%), while failed memory or KV-cache allocations and reported overload or resource exhaustion contribute 19.4%.

100

100

80

10−2

Error rate

Normalized requests (%)

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

60 40 20 0

0

20

40

60

80

10

7.1% 13.6%

4.8%

10−6 10−8

0.2%

0

20

40

60

80

21.0%

21.1%

Conn. failure Gen. failure Gen. timeout Allocation RPC timeout Conn. reset Resource limit Cache I/O Other

Days

(b) Error rate Figure 8. Service reliability under frequent updates.

Cache loading and transfer can likewise encounter connection failures, timeouts, and data-transfer errors, accounting for a smaller 4.8%. The diversity of errors encountered in production requires complementary measures across the serving stack, combining bug fixes, resource protection, failure handling, and improved diagnostics to sustain reliable service as the system evolves.

8

7.8%

−4

Days

(a) Traffic

12.0%

12.3%

L ESSONS L EARNED

The long-term cost of owning an inference engine. Developing RTP-LLM around Alibaba’s requirements gives us flexibility in choosing which model, hardware, and serving capabilities to prioritize. That flexibility comes with responsibility for their implementation, validation, optimization, and long-term maintenance. Adopting an existing engine also requires local integration and production validation, so imperfections in an open-source framework are not sufficient grounds for starting an independent implementation. The decision should therefore weigh the benefits of organization-specific evolution against the engineering and maintenance costs over the engine’s lifetime. When existing engines can accommodate an organization’s requirements without extensive, recurring modifications, adopting and extending them may be more sustainable than maintaining an independent stack. Balancing execution efficiency with model-development cost. Our C++-first design prioritized execution efficiency, but model evolution exposed a recurring engineering cost. Supporting new models required translating their Python/PyTorch reference implementations into C++, adding extra development overhead. We therefore moved model execution to Python/PyTorch while retaining scheduling and KV-cache management in C++. This adjustment reflects a practical trade-off: runtime overhead must be weighed against the recurring work of translating, validating, and maintaining rapidly changing model code, rather than applying the same implementation strategy to every component. Generalizing after concrete model adaptation. Some of our model adaptations introduced requirements that the shared framework did not yet express. We have handled such changes by first establishing a working model adaptation, then generalizing its reusable capabilities into shared

(c) Error breakdown

framework mechanisms. Working through the adaptation makes the computation and state requirements concrete, helping distinguish model-specific details from functionality that belongs in a common abstraction. The initial adaptation allows us to understand a requirement before committing to a general interface, but the follow-up consolidation remains part of its engineering cost. Without that consolidation, locally successful adaptations risk accumulating as separate paths that must be maintained independently. Keeping cluster control understandable and recoverable. Our production experience highlights the importance of explicit component dependencies and failure boundaries in cluster control. A request-placement policy should make clear which components it depends on and how it behaves when those components time out or become unavailable. Coordination mechanisms should be designed to contain such failures, allowing operators to trace automated decisions and identify the affected part of the system. Cluster control should also preserve a way for operators to redirect traffic manually when automated decisions become ineffective or difficult to diagnose. This makes operational control an explicit design requirement: operators need a way to intervene even before the underlying fault has been fully diagnosed. Balancing request admission with service quality. We encountered resource overload when the Master admitted work without rate limiting. This experience showed that admitting more requests can compromise the service delivered to requests already in progress. We therefore treat admission limits as part of production reliability, rather than relying solely on redundancy and capacity expansion to absorb excess demand. Rejecting requests has an immediate service cost, but continuing to accept work beyond available capacity can spread that cost across a larger set of requests through contention, queueing, and timeouts. The appropriate balance depends on workload characteristics and service objectives: the aim is not to minimize rejection in isolation, but to preserve useful service under constrained capacity. Developing inference engines in the era of coding agents. LLM-based coding agents are changing the engineering effort required to support new models and optimize their execution in inference engines. With reference implemen-

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

tations and defined correctness comparisons, agents can quickly implement model support and help bring a newly released model to correctness-ready. However, reaching performance-ready requires assessing the model’s performance potential on the target hardware and identifying the most promising optimization directions, both of which depend on understanding how model computation, data movement, and communication interact under the intended workload and deployment configuration. At present, experienced engineers remain central to this assessment, while agents can assist with implementing and evaluating the chosen optimizations. As initial adaptation becomes easier, performance analysis and optimization design account for a greater share of the expertise required to develop an inference engine. RTP-LLM’s general applicability. Alibaba’s scale amplifies the challenges addressed by RTP-LLM, but these challenges do not arise from traffic volume alone. For an organization serving a small, stable set of models on uniform hardware, a deployment tailored to those configurations may be sufficient. However, as its model portfolio expands and release cycles accelerate, model support becomes a recurring integration task rather than a one-time implementation. When new capabilities require changes to shared serving code, each integration also introduces validation and maintenance work for existing services. Adding accelerator capacity does not remove this engineering burden, which can become significant well before an organization reaches Alibaba’s deployment scale. We therefore believe that the design space explored by RTP-LLM—accommodating rapid model evolution while preserving a reusable and reliable serving foundation—will become increasingly relevant to organizations operating diverse, continuously changing model portfolios, rather than remain a concern unique to hyperscale deployments.

9

R ELATED W ORK

LLM inference engines have developed complementary mechanisms for organizing autoregressive computation and managing its execution state. Orca (Yu et al., 2022) introduces iteration-level scheduling to coordinate requests at the granularity of individual model iterations. vLLM’s PagedAttention (Kwon et al., 2023) reduces KV-cache fragmentation and redundant copying while supporting memory sharing across requests. SGLang (Zheng et al., 2024) combines an interface for structured language model programs with runtime mechanisms such as prefix KV-cache reuse. TensorRT-LLM (Nvidia, 2023b) provides optimized execution through specialized kernels, numerical formats, and distributed execution support. These systems provide important architectural and implementation precedents for combining model execution, request scheduling, and state

management within an inference engine. Their relevance extends beyond individual optimizations to the broader organization of the serving stack. Complementary research improves model execution efficiency by reducing computation, data movement, and sequential decoding overhead. Hardware-aware attention implementations reduce memory traffic and improve the utilization of GPU execution resources (Dao et al., 2022; Dao, 2024; Shah et al., 2024; Ye et al., 2025). Low-bit quantization reduces the storage and bandwidth requirements of model weights and, in some methods, activations (Tseng et al., 2024; Ashkboos et al., 2024; Liu et al., 2025). Speculative and multi-token decoding approaches generate candidate continuations and verify them in parallel, amortizing target-model execution over multiple output tokens (Cai et al., 2024; Li et al., 2024; 2026). These directions address different sources of inference overhead and introduce execution choices whose applicability depends on model structure, numerical representation, and hardware capabilities. Together, they expand the set of execution mechanisms that inference engines must accommodate as models and accelerator platforms evolve. Beyond individual model execution, serving systems optimize how requests share computation, state, and distributed resources. Iteration-level scheduling and chunked-prefill batching organize prompt processing and token generation to balance throughput with request latency (Agrawal et al., 2024; Holmes et al., 2024). Other work reduces repeated prompt computation and state-loading overhead through prefix or segment-level reuse, selective recomputation, and compressed KV-cache transfer (Zheng et al., 2024; Gim et al., 2024; Yao et al., 2025; Liu et al., 2024b). Prefill– decode disaggregation separates execution phases to enable resource allocation and placement suited to their different demands (Qin et al., 2026; Patel et al., 2024; Zhong et al., 2024). Runtime request migration and elastic parallelism further adapt execution placement and resource assignments to workload variation (Sun et al., 2024; Wu et al., 2024). These approaches address complementary aspects of serving efficiency, from batch formation to cluster-wide resource organization. Our work complements them by documenting how a shared inference engine incorporates evolving execution and serving capabilities, together with the architectural trade-offs and operational lessons arising from long-term production deployment.

10

C ONCLUSION

We present RTP-LLM, an open-source inference engine developed for continuous evolution and reliable production serving at Alibaba. Our experience highlights the need to revisit the boundary between model execution and the serving runtime as computation and state requirements evolve.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

Over more than three years of development, RTP-LLM has grown from an exploratory system into shared production infrastructure serving diverse Alibaba businesses. Its current architecture has continued to accommodate new models and serving capabilities without another fundamental redesign. We share our engineering experience and lessons learned from this evolution. As model portfolios expand and release cycles accelerate, we hope these lessons and the idea of separating model execution from the serving runtime will inspire practitioners building inference infrastructure around their own workloads and operational constraints.

R EFERENCES Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in llm inference with sarathiserve. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 117–134, 2024. Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, volume 2024, pp. 35549– 35562, 2024. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022. DeepSeek-AI. Deepinfer: High-performance, large-scale inference framework for deepseek models. https:// github.com/deepseek-ai/DeepInfer, 2026.

DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, J., Dong, K., Gao, K., Guan, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng, S., Wang, T., Pei, T., Yuan, T., Sun, T., Xiao, W. L., Zeng, W., An, W., Liu, W., Liang, W., Gao, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nie, X., Sun, X., Wang, X., Liu, X., Xie, X., Yu, X., Song, X., Zhou, X., Yang, X., Lu, X., Su, X., Wu, Y., Li, Y. K., Wei, Y. X., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Zheng, Y., Zhang, Y., Xiong, Y., Zhao, Y., He, Y., Tang, Y., Piao, Y., Dong, Y., Tan, Y., Liu, Y., Wang, Y., Guo, Y., Zhu, Y., Wang, Y., Zou, Y., Zha, Y., Ma, Y., Yan, Y., You, Y., Liu, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Huang, Z., Zhang, Z., Xie, Z., Hao, Z., Shao, Z., Wen, Z., Xu, Z., Zhang, Z., Li, Z., Wang, Z., Gu, Z., Li, Z., and Xie, Z. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y., Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu,

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024b. DeepSeek-AI, Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., Ling, C., Lu, C., Zhao, C., Deng, C., Hou, C., Xu, C., Shao, C., Ruan, C., Sun, C., Dai, D., Guo, D., Yang, D., Chen, D., Li, D., Ji, D., Li, E., Wei, F., Lin, F., Yuan, F., Xia, F., Dai, F., Hao, G., Chen, G., Cao, G., Meng, G., Li, G., Yu, H., Zhang, H., Xu, H., Li, H., Liang, H., Zhang, H., Luo, H., Wei, H., Yuan, H., Zhang, H., Luo, H., Chen, H., Ji, H., Zhang, H., Ding, H., Tang, H., Cao, H., Gao, H., Qu, H., Zeng, H., Yang, J., Zhu, J. Q., Luo, J., Song, J., Yu, J., Huang, J., Cai, J., Liang, J., Zhou, J., Ye, J., Li, J., Xu, J., Hu, J., Yang, J., Chen, J., Yan, J., Chen, J., Zhou, J., Xiang, J., Yuan, J., Cheng, J., Zhou, J., Zhu, J., Yu, J., Sun, J., Ran, J., Jiang, J., Qiu, J., Li, J., Zheng, J., Song, J., Dong, K., Gao, K., Guan, K., Zhou, K., Huang, K., Yu, K., Wang, L., Zhang, L., Wang, L., Xia, L., Zhang, L., Zhao, L., Guo, L., Luo, L., Ma, L., Zhu, L., Wang, L., Cai, L., Zhang, L., Chen, L., Di, M. S., Xu, M. Y., Mei, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Zhou, M., Han, M., Wang, N., Huang, P., Wang, P., Cong, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Li, Q., Chen, Q., Du, Q., Jiang, Q., Tian, R., Xu, R., Lu, R., Xu, R., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R., Yin, R., Xu, R., Shen, R., Zhang, R., Chen, R., Liu, S. H., Lu, S., Sun, S., Zhou, S., Chen, S., Cai, S., Nie, S., Wu, S., Chen, S., Hu, S., Liu, S., Hu, S., Ma, S., Wang, S., Yu, S., Zhou, S., Pan, S., Yu, S., Zhou, S., Ni, T., Yun, T., Jin, T., Pei, T., Ye, T., Lin, T., Ji, T., Cui, T., Yue, T., Yu, T., Wang, T., Zhang, W., Xiao, W. L., Zeng, W., An, W., Zhao, W., Liu, W., Liang, W., Pang, W., Luo, W., Yao, W., Gao, W., Yang, W., Huang, W., Hou, W., Zhang, W., Ma, W., Gao, X., He, X., Wang, X., Wang, X., Bi, X., Liu, X., Wang, X., Chen, X., Zhang, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Liu, X., Yu, X., Li, X., Yang, X., Zhang, X., Chen, X., Wang, X., Su, X., Chen, X., Lin, X., Fu, X., Yan, Y. C., Wang, Y. Q., Ma, Y. W., Luo, Y., Zhang, Y., Xu, Y., Ma, Y., Huang, Y., Li, Y., Li, Y., Xu, Y., Zhao, Y., Sun, Y., Wang, Y., Qian, Y., Shao, Y., Yu, Y., Zhang, Y., Ding, Y., Shi, Y., Wu, Y., Xiong, Y., Ma, Y., He, Y., Tang, Y., Zhou, Y., Luo, Y., Zhong, Y., Piao, Y., Wang, Y., Zhang, Y., Chen, Y., Tan, Y., Wei, Y., Ma, Y., Liu, Y., Yang, Y., Guo, Y., Wu, Y., Wu, Y., Li, Y., Cheng, Y., Ou, Y., Xu,

Y., Li, Y., Wang, Y., Yang, Y., Xu, Y., Wu, Y., Meng, Y., Zou, Y., Zha, Y., Xiong, Y., Chen, Y., Lin, Y., Cao, Y., Wang, Y., Zhang, Y., Yan, Y., Lin, Y., Gu, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhou, Y., Huang, Y., Wu, Z. F., Wang, Z., Zhao, Z., Ren, Z., Zhang, Z., Sha, Z., Fu, Z., Ju, Z., Xu, Z., Xie, Z., Zhang, Z., Gao, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Huang, Z., Chen, Z., Wu, Z., Ren, Z., Wu, Z., Li, Z., Zhang, Z., Xu, Z., Wang, Z., Qu, Z., Gu, Z., Zhu, Z., Li, Z., Zhang, Z., Xie, Z., Gao, Z., Wan, Z., Pan, Z., and Yao, Z. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Gim, I., Chen, G., Lee, S.-s., Sarda, N., Khandelwal, A., and Zhong, L. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325–338, 2024. GLM-5-Team, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Huang, C., Xie, C., Zhu, C., Yin, C., Wang, C., Pan, G., Zeng, H., Zhang, H., Wang, H., Chen, H., Zhang, J., Jiao, J., Guo, J., Wang, J., Du, J., Wu, J., Wang, K., Li, L., Fan, L., Zhong, L., Liu, M., Zhao, M., Du, P., Dong, Q., Lu, R., Shuang-Li, Cao, S., Liu, S., Jiang, T., Chen, X., Zhang, X., Huang, X., Dong, X., Xu, Y., Wei, Y., An, Y., Niu, Y., Zhu, Y., Wen, Y., Cen, Y., Bai, Y., Qiao, Z., Wang, Z., Wang, Z., Zhu, Z., Liu, Z., Li, Z., Wang, B., Wen, B., Huang, C., Cai, C., Yu, C., Li, C., Hu, C., Zhang, C., Zhang, D., Lin, D., Yang, D., Wang, D., Ai, D., Zhu, E., Yi, F., Chen, F., Wen, G., Sun, H., Zhao, H., Hu, H., Zhang, H., Liu, H., Zhang, H., Peng, H., Tai, H., Zhang, H., Liu, H., Wang, H., Yan, H., Ge, H., Liu, H., Chu, H., Zhao, J., Wang, J., Zhao, J., Ren, J., Wang, J., Zhang, J., Gui, J., Zhao, J., Li, J., An, J., Li, J., Yuan, J., Du, J., Liu, J., Zhi, J., Duan, J., Zhou, K., Wei, K., Wang, K., Luo, K., Zhang, L., Sha, L., Xu, L., Wu, L., Ding, L., Chen, L., Li, M., Lin, N., Ta, P., Zou, Q., Song, R., Yang, R., Tu, S., Yang, S., Wu, S., Zhang, S., Li, S., Li, S., Fan, S., Qin, W., Tian, W., Zhang, W., Yu, W., Liang, W., Kuang, X., Cheng, X., Li, X., Yan, X., Hu, X., Ling, X., Fan, X., Xia, X., Zhang, X., Zhang, X., Pan, X., Zou, X., Zhang, X., Liu, Y., Wu, Y., Li, Y., Wang, Y., Zhu, Y., Tan, Y., Zhou, Y., Pan, Y., Zhang, Y., Su, Y., Geng, Y., Yan, Y., Tan, Y., Bi, Y., Shen, Y., Yang, Y., Li, Y., Liu, Y., Wang, Y., Li, Y., Wu, Y., Zhang, Y., Duan, Y., Zhang, Y., Liu, Z., Jiang, Z., Yan, Z., Zhang, Z., Wei, Z., Chen, Z., Feng, Z., Yao, Z., Chai, Z., Wang, Z., Zhang, Z., Xu, B., Huang, M., Wang, H., Li, J., Dong, Y., and Tang, J. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Holmes, C., Tanaka, M., Wyatt, M., Awan, A. A., Rasley, J., Rajbhandari, S., Aminabadi, R. Y., Qin, H., Bakhtiari, A., Kurilenko, L., and He, Y. Deepspeed-fastgen: High-

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

throughput text generation for llms via mii and deepspeedinference. arXiv preprint arXiv:2401.08671, 2024. Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y., Zhang, Y., Yang, A., Men, R., Huang, F., Zheng, B., Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. Kimi Team, Bai, T., Bai, Y., Bao, Y., C, M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, G., Chen, G., Chen, G., Chen, H., Chen, J., Chen, J., Chen, J., Chen, K., Chen, P., Chen, R., Chen, W., Chen, X., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Z., Cheng, D., Cheng, Y., Cui, J., Cui, J., Dai, A., Deng, J., Ding, H., Ding, R., Ding, S., Dong, M., Dong, M., Dong, Y., Dong, Y., Du, A., Du, C., Du, D., Du, J., Du, Y., Fan, Y., Feng, J., Feng, Q., Feng, Y., Fu, K., Fu, Q., Gao, F., Gao, H., Gao, J., Gao, T., Gao, W., Geng, S., Gong, J., Gong, L., Gong, S., Gong, X., Gu, Q., Gu, Y., Guan, S., Guo, H., Guo, S., Guo, X., Guo, Z., Hao, B., Hao, W., Hao, X., He, D., He, H., He, L., He, Q., He, W., He, X., He, X., He, Y., He, Y., Hong, C., Hong, T., Hu, H., Hu, J., Hu, R., Hu, W., Hu, Y., Hu, Z., Hua, L., Huang, J., Huang, K., Huang, R., Huang, S., Huang, W., Huang, Y., Huang, Z., Huang, Z., Hui, Y., Jia, C., Jiang, Y., Jiang, Z., Jiang, Z., Jin, W., Jin, X., Jing, Y., Kong, H., Lai, G., Li, A., Li, C., Li, C., Li, C., Li, F., Li, G., Li, H., Li, J., Li, J., Li, L., Li, L., Li, L., Li, W., Li, W., Li, X., Li, Y., Li, Y., Li, Y., Li, Y., Li, Z., Li, Z., Li, Z., Li, Z., Li, Z., Lin, J., Lin, X., Lin, Y., Lin, Z., Lin, Z., Liu, B., Liu, B., Liu, C., Liu, L., Liu, S., Liu, S., Liu, S., Liu, T., Liu, W., Liu, Y., Liu, Y., Liu, Y., Liu, Y., Liu, Z., Liu, Z., Lu, E., Lu, H., Lu, L., Lu, T., Lu, Z., Luo, A., Luo, G., Luo, J., Luo, Y., Lyu, B., Lyu, W., Mao, S., Mei, Y., Men, X., Ni, M., Niu, Y., Pan, S., Peng, S., Qi, Z., Qin, R., Qin, Z., Qin, Z., Qiu, H., Qiu, J., Qiu, J., Qu, B., Qu, Y., Shang, Z., Shao, Y., Shen, H., Shi, J., Shi, J., Shi, L., Shi, S., Siu, W., Song, P., Song, X., Su, J., Su, Y., Su, Z., Sui, L., Sun, J., Sun, J., Sun, S., Sun, S., Sun, T., Sun, Y., Tai, Y., Tang, C., Tang, H., Tang, S., Tang, Z., Tian, C., Tian, R., Tian, Y., Tu, W., Wang, C., Wang, C., Wang, C., Wang, D., Wang, F., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, J., Wang, J., Wang, J., Wang, J., Wang, L., Wang, S., Wang, S., Wang, S., Wang, S., Wang, S., Wang, T., Wang, W., Wang, X., Wang, X., Wang, X., Wang, X.,

Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wei, C., Wei, M., Wei, S., Wen, Z., Wu, F., Wu, H., Wu, R., Wu, W., Wu, X., Wu, Y., Wu, Y., Wu, Y., Wu, Z., Xian, X., Xiang, C., Xiang, Y., Xiao, B., Xiao, C., Xiao, X., Xie, J., Xie, X., Xie, Y., Xie, Z., Xing, B., Xiong, Y., Xu, B., Xu, B., Xu, J., Xu, J., Xu, J., Xu, J., Xu, L. H., Xu, Q., Xu, S., Xu, S., Xu, T., Xu, T., Xu, W., Xu, X., Xu, Y., Xu, Y., Xu, Y., Xu, Z., Xue, H., Yan, J., Yan, Y., Yang, F., Yang, G., Yang, H., Yang, J., Yang, R., Yang, W., Yang, X., Yang, X., Yang, Y., Yang, Y., Yang, Y., Yang, Y., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yao, H., Ye, D., Ye, H., Ye, W., Ye, Z., Yin, B., Yin, H., Yin, X., Yu, C., Yu, H., Yu, L., Yu, S., Yu, S., Yu, T., Yuan, E., Yuan, M., Yue, T., Yue, W., Yue, Y., Zha, D., Zhan, H., Zhang, B. H., Zhang, D., Zhang, F., Zhang, H., Zhang, H., Zhang, H., Zhang, J., Zhang, J., Zhang, J., Zhang, K., Zhang, M., Zhang, P., Zhang, Q., Zhang, R., Zhang, R., Zhang, S., Zhang, S., Zhang, X., Zhang, X., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Z., Zhang, Z., Zhao, B., Zhao, C., Zhao, F., Zhao, J., Zhao, J., Zhao, S., Zhao, W., Zhao, X., Zhao, X., Zhao, Y., Zhao, Z., Zheng, H., Zheng, H., Zheng, R., Zheng, S., Zheng, T., Zhong, H., Zhong, L., Zhong, L., Zhou, M., Zhou, Q., Zhou, R., Zhou, R., Zhou, X., Zhou, Y., Zhou, Z., Zhu, J., Zhu, L., Zhu, X., Zhu, Y., Zhu, Y., Zhu, Z., Zhuang, C., Zhuang, W., and Zu, X. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International conference on machine learning, pp. 19274–19286. PMLR, 2023. Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 7421–7432, 2024. Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle-3: Scaling up inference acceleration of large language models via training-time test. Advances in Neural Information Processing Systems, 38:136737–136756, 2026. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

compression and acceleration. Proceedings of machine learning and systems, 6:87–100, 2024. Liu, H., Zaharia, M., and Abbeel, P. Ringattention with blockwise transformers for near-infinite context. In International Conference on Learning Representations, volume 2024, pp. 3992–4008, 2024a. Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., and Jiang, J. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pp. 38–56, 2024b. Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Llm quantization with learned rotations. In International Conference on Learning Representations, volume 2025, pp. 92009–92032, 2025. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1–15, 2021. Nvidia. Fastertransformer. https://github.com/ NVIDIA/FasterTransformer, 2023a. Nvidia. Tensorrt llm. https://github.com/ NVIDIA/TensorRT-LLM, 2023b. Nvidia. CUDA C++ Programming Guide, 2026. URL https://docs.nvidia.com/cuda/ cuda-c-programming-guide/index.html# cuda-graphs. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S.,

Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Kaiser, Ł., Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Kondraciuk, Ł., Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., Peres, F. d. A. B., Petrov, M., Pinto, H. P. d. O., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C. J., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132. IEEE, 2024. Qin, R., Li, Z., He, W., Cui, J., Tang, H., Ren, F., Ma, T., Cai, S., Zhang, Y., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kvcache-centric disaggregated architecture for llm serving. ACM Transactions on Storage, 22(4): 1–38, 2026. Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning, pp. 18332–18346. PMLR, 2022. Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68685, 2024. Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 173–191, 2024. Tan, B., Guo, J., Lv, Z., Sun, H., Yang, T., Liu, K., Shi, X., Hu, Z., Yu, Y., Zhang, C., Zhang, J., Yang, X., Zhang, W., Cai, B., Zhou, S., Wang, X., He, N., Yu, Y., Bao, W., Huang, G., Yuan, Y., Yin, J., Wang, N., Yang, L., Zhang, Z., Chen, L., Li, G., Lan, T., and Qu, L. Rtp-llm: High-performance alibaba llm inference engine. arXiv preprint arXiv:2605.29639, 2026. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., Rouillard, L., Mesnard, T., Cideron, G., Grill, J.-b., Ramos, S., Yvinec, E., Casbon, M., Pot, E., Penchev, I., Liu, G., Visin, F., Kenealy, K., Beyer, L., Zhai, X., Tsitsulin, A., Busa-Fekete, R., Feng, A., Sachdeva, N., Coleman, B., Gao, Y., Mustafa, B., Barr, I., Parisotto, E., Tian, D., Eyal, M., Cherry, C., Peter, J.-T., Sinopalnikov, D., Bhupatiraju, S., Agarwal, R., Kazemi, M., Malkin, D., Kumar, R., Vilar, D., Brusilovsky, I., Luo, J., Steiner, A., Friesen, A., Sharma, A., Sharma, A., Gilady, A. M., Goedeckemeyer, A., Saade, A., Feng, A., Kolesnikov, A., Bendebury, A., Abdagic, A., Vadi, A., György, A., Pinto, A. S., Das, A., Bapna, A., Miech, A., Yang, A., Paterson, A., Shenoy, A., Chakrabarti, A., Piot, B., Wu, B., Shahriari, B., Petrini, B., Chen, C., Lan, C. L., Choquette-Choo, C. A., Carey, C. J., Brick, C., Deutsch, D., Eisenbud, D., Cattle, D., Cheng, D., Paparas, D., Sreepathihalli, D. S., Reid, D., Tran, D., Zelle, D., Noland, E., Huizenga, E., Kharitonov, E., Liu, F., Amirkhanyan, G., Cameron, G.,

Hashemi, H., Klimczak-Plucińska, H., Singh, H., Mehta, H., Lehri, H. T., Hazimeh, H., Ballantyne, I., Szpektor, I., Nardini, I., Pouget-Abadie, J., Chan, J., Stanton, J., Wieting, J., Lai, J., Orbay, J., Fernandez, J., Newlan, J., Ji, J.-y., Singh, J., Black, K., Yu, K., Hui, K., Vodrahalli, K., Greff, K., Qiu, L., Valentine, M., Coelho, M., Ritter, M., Hoffman, M., Watson, M., Chaturvedi, M., Moynihan, M., Ma, M., Babar, N., Noy, N., Byrd, N., Roy, N., Momchev, N., Chauhan, N., Sachdeva, N., Bunyan, O., Botarda, P., Caron, P., Rubenstein, P. K., Culliton, P., Schmid, P., Sessa, P. G., Xu, P., Stanczyk, P., Tafti, P., Shivanna, R., Wu, R., Pan, R., Rokni, R., Willoughby, R., Vallu, R., Mullins, R., Jerome, S., Smoot, S., Girgin, S., Iqbal, S., Reddy, S., Sheth, S., Põder, S., Bhatnagar, S., Panyam, S. R., Eiger, S., Zhang, S., Liu, T., Yacovone, T., Liechty, T., Kalra, U., Evci, U., Misra, V., Roseberry, V., Feinberg, V., Kolesnikov, V., Han, W., Kwon, W., Chen, X., Chow, Y., Zhu, Y., Wei, Z., Egyed, Z., Cotruta, V., Giang, M., Kirk, P., Rao, A., Black, K., Babar, N., Lo, J., Moreira, E., Martins, L. G., Sanseviero, O., Gonzalez, L., Gleicher, Z., Warkentin, T., Mirrokni, V., Senter, E., Collins, E., Barral, J., Ghahramani, Z., Hadsell, R., Matias, Y., Sculley, D., Petrov, S., Fiedel, N., Shazeer, N., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Alayrac, J.-B., Anil, R., Dmitry, Lepikhin, Borgeaud, S., Bachem, O., Joulin, A., Andreev, A., Hardin, C., Dadashi, R., and Hussenot, L. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and De Sa, C. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. Proceedings of machine learning research, 235:48630, 2024. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45, 2020.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

Wu, B., Liu, S., Zhong, Y., Sun, P., Liu, X., and Jin, X. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pp. 640–654, 2024. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087–38099. PMLR, 2023. Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., and Jiang, J. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the twentieth European conference on computer systems, pp. 94–109, 2025. Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems, 7, 2025. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for transformerbased generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), pp. 521–538, 2022. Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y., Yu, K., Li, J., and Zhao, L. Deepep: an efficient expertparallel communication library. https://github. com/deepseek-ai/DeepEP, 2025. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 193–210, 2024.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale Table 2. Representative configuration dimensions and capabilities of RTP-LLM. Dimension

Representative capabilities and configurations

Model-Specific Capabilities and Requirements Workloads and modalities Generation, embedding, reranking, and classification across text, vision–language, and audio inputs Model architectures Dense and MoE models; MHA/MLA, compressed sparse attention, sliding-window attention, and recurrent/attention hybrids Numerical formats FP16/BF16 and representative INT8/INT4, FP8, and FP4 formats for weights, activations, and KV state Speculative decoding MTP, EAGLE/EAGLE-3, and DSpark block-draft decoding Shared Runtime and Serving Mechanisms Hardware backends NVIDIA GPUs (CUDA) and AMD GPUs (ROCm) Shared operators Reusable attention, MoE, and quantized-computation operators Parallel execution Tensor, pipeline, data, and expert parallelism; prefill-context and FFN-sequence parallelism; multi-node execution Execution acceleration CUDA Graphs; asynchronous output processing; expert load balancing and communication– computation overlap Service interfaces Streaming OpenAI-compatible chat API; batch chat extension; native task APIs Generation controls Sampling and beam search; JSON/schema, regex, EBNF, and structural-tag constraints Request scheduling Cache-aware load balancing; worker-local continuous batching and chunked prefill Inference-state management Grouped state pools; paged allocation; prefix reuse with prefix-tree indexing Cache storage Device memory, host memory, local disk, and remote/distributed cache, including 3FS Deployment topology Fused or disaggregated computation; co-located or separated service components Startup acceleration Parallel and GPUDirect weight loading; reuse of JIT compilation artifacts

A

S UPPORTED C ONFIGURATIONS

Table 2 organizes representative capabilities of RTP-LLM from two perspectives: model-specific capabilities and requirements, and shared runtime and serving mechanisms. Together, they summarize the range of model requirements accommodated by RTP-LLM and the common mechanisms it provides for production serving.

B

A M ODEL -I NTEGRATION C ASE S TUDY

We use DeepSeek-V4 (DeepSeek-AI et al., 2026) as a case study to illustrate how RTP-LLM accommodates new models and their associated execution and serving requirements. DeepSeek-V4 required more than extending an existing model configuration. Its layer-dependent combination of compressed sparse attention, heavily compressed attention, and sliding-window attention, together with mHC residual transformations and revised MoE routing, could not be directly expressed by the existing DeepSeek execution path. These components also introduced new checkpoint layouts and quantization requirements. Beyond computation, individual layers required multiple persistent states with different layouts, growth patterns, and reuse conditions. Supporting the model therefore required both new model implementations and extensions to shared state-management and execution mechanisms. RTP-LLM accommodated these changes at distinct extension points. A dedicated Python model implementation composed the new attention, residual, and MoE computations while integrating with the existing model initialization and

invocation framework. Per-layer weight descriptions map V4’s parameters into the common loading pipeline, while V4-specific conversion strategies handle its FP8 and MXFP4 weights, UE8M0 scales, and execution-specific layouts. For example, its FP8 loading strategy preserves checkpointprovided scales rather than re-deriving them through requantization; this behavior is restricted to V4 weights. On the serving side, heterogeneous cache groups represent compressed KV entries, indexer state, and sliding-window state, with corresponding allocation and reuse policies. The integration also adapted per-group block mappings and crossworker cache transfer to V4’s heterogeneous states, and adjusted MTP hidden-state buffers and CUDA Graph input buffers to accommodate its mHC-expanded representations. The integration thus involved substantial model-specific code and meaningful runtime changes, but retained the common request-serving organization rather than introducing a separate serving architecture for DeepSeek-V4. Requests continued to use the existing request queues, batchscheduling framework, and output-dispatch path, with V4specific computation invoked through the common model interface. New computation and parameter interpretation were implemented in the model and associated execution and loading modules, while cross-cutting state requirements were accommodated through extensions to shared mechanisms. This case illustrates that evolvability is not about keeping the runtime fixed, but about ensuring that modeldriven changes can be absorbed through explicit extension points without fragmenting the shared serving path.

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale Table 3. Model-enablement timelines for representative models in RTP-LLM.

C

Model

Available

Correctness-ready

Performance-ready

Integration/ Canary-ready

Production-ready

DeepSeek-V4 GLM-5.2 MiniMax-M3 Qwen3.5

2026-04-24 2026-06-17 2026-06-12 2026-02-16

2026-04-30 2026-06-18 2026-06-15 2026-02-18

2026-05-20 2026-06-28 2026-06-26 2026-03-02

2026-05-21 2026-06-30 2026-07-03 2026-03-09

2026-05-22 2026-07-01 2026-07-06 2026-03-11

M ODEL E NABLEMENT T IMELINES

We report readiness timelines for the four models examined in Section 7.1. Table 3 summarizes their enablement timelines using the readiness milestones from Section 2.1, where we distinguish the time spent establishing model correctness, optimizing execution for production, and carrying out business integration and canary testing. Dates indicate when each readiness milestone was reached.3 The table shows that reaching correctness-ready takes one to three days for GLM-5.2, MiniMax-M3, and Qwen3.5, and about one week for DeepSeek-V4. By comparison, performance optimization accounts for the longest interval, at approximately three weeks for DeepSeek-V4 and two weeks for the other three models. Since our goal is to match and improve upon community implementations while meeting production SLOs, this stage requires bottleneck analysis and optimization, performance benchmarking under comparable conditions, and checks for correctness and performance regressions. The results often prompt further tuning, so reaching performance-ready requires multiple rounds of optimization and testing. The table also shows shorter integration/canary intervals, ranging from one day to about one week. This is partly because integration with production service interfaces can often proceed alongside correctness validation and performance optimization. Canary testing can also be interleaved with optimization, with successive optimized versions receiving limited production traffic. After these checks and canary validation, the models become production-ready for Alibaba’s internal applications and customer-facing services. In general, the full process usually takes 14–28 days from community release. These timelines illustrate RTP-LLM’s ability to evolve with new model releases while completing the optimization and validation required for reliable production service.

3

Timelines depend on team size and the engineering time allocated to each model, and are not intended as controlled comparisons of adaptation difficulty or engineering effort across models.

Related documents

• RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale#1536782
• Escaping Redundant Reasoning: Structure-Aware Search for Inference-Time LLMs#820751
• Optimal Model Activation Policies for Inference Networks of Large Language Models#1110390
• LLM Inference in a Flash!#1110540
• A Unified Approach to Routing and Cascading for LLMs#1327205