Intelligence Delivery Network: Toward an Internet Architecture for the AI Age Hanling Wang1 , Qing Li1,* , Dan Zhao1 , Yuhong Song1 , Xingchi Chen1 , Teng Gao1 , Peiyuan Zong1 , Zhuyun Qi1 , Yue Yu1 , Yong Jiang2,1
arXiv:2605.13235v1 [cs.NI] 13 May 2026
{wanghl03,liq,zhaod01,songyh,chenxch01,gaot,zongpy,qizy,yuy}@pcl.ac.cn,[email protected] 1 Pengcheng Laboratory, Shenzhen, Guangdong, China 2 Tsinghua SIGS, Shenzhen, Guangdong, China * Corresponding author
Abstract
1 Endpoint Reachability
The rapid emergence of AI-powered applications is reshaping the role of the Internet. Users increasingly rely on the network to obtain intelligence services derived from large foundation models, rather than merely to reach remote endpoints or retrieve specific content. Today’s dominant deployment paradigm for AI services remains cloud-centric, where user requests are transmitted to remote data centers for centralized inference. Although operationally convenient, this paradigm suffers from latency and jitter, heavy wide-area traffic, limited utilization of distributed heterogeneous compute resources, and growing privacy and governance concerns. In this paper, we propose the Intelligence Delivery Network (IDN), an Internet architecture that treats AI capabilities as deliverable network services. The key idea is to position, select, reuse, and verify intelligence across cloud, regional, edge, and local environments according to demand locality, resource availability, and policy constraints. We present the system assumptions of IDN, define its core architectural mechanisms, and discuss how capability abstraction, compute resource integration, demand-driven deployment, service routing, state-aware caching, and trust management can jointly support distributed AI services. We believe that IDN provides a practical path toward an Internet architecture for the AI age, making AI capabilities more accessible, efficient, trustworthy, and responsive to diverse application needs.
IP 198.51.100.20
Content Delivery
3 Intelligence Service
Origin Server
Cloud Model Serving
O1
Edge Cache (Region A)
O1
Edge Cache (Region B)
O1
Edge Cache (Region C)
Edge DC (Region A)
Edge DC (Region B)
Edge DC (Region C)
Retrieve Q1 from nearby cache
O1
IP
IP
203.0.113.30
203.0.113.40
User in Region A
O1
User in Region B
O1
User in Region C
User devices & applications
Figure 1: A high-level shift in network abstractions: from endpoint reachability, to content delivery, to intelligence delivery.
host-to-host conversations. This observation shaped industrial Content Delivery Networks (CDNs) [9, 30] and motivated informationcentric networking research [18, 40, 43]. A comparable shift is now emerging with the prosperity of large foundation models. Transformer-based Large Language Models (LLM) have enabled general-purpose Artificial Intelligence (AI) capabilities that can be invoked by a variety of applications, including interactive assistants, retrieval-augmented search, code copilots, real-time translation, robotics, and multimodal perception [7, 20, 22, 42]. In these settings, the central question is no longer simply whether a server is reachable. What matters is whether the system can deliver the appropriate intelligence with acceptable latency, quality, cost, privacy, and operational stability. Today, however, AI services remain largely cloud-centric. User inputs are sent across the wide-area network to remote data centers, where inference is performed inside provider-controlled serving stacks. Such stacks have made major progress in batching, scheduling, memory management, and model-variant selection [7, 14, 20, 32, 42]. Although this approach is operationally attractive, it also reveals several fundamental limitations: (i) It introduces nontrivial end-to-end latency and jitter, which are difficult to tolerate in interactive or time-sensitive applications; (ii) It increases wide-area traffic by repeatedly transporting prompts, retrieved documents, media streams, and other input data to distant locations; (iii) It leaves many distributed and heterogeneous compute resources at the edge, on premises, and within local networks underutilized. (iv) It complicates privacy protection, policy compliance, and data
CCS Concepts • Networks → Network architectures; • Computer systems organization → Cloud computing; • Security and privacy → Distributed systems security.
Keywords Internet architecture, distributed AI services, edge intelligence, service routing, model serving, caching
1
IP 192.0.2.10
2
Vision: Intelligence as a Networked Service
Over the past several decades, the Internet has been repeatedly reshaped by the abstraction it makes common, as shown in Fig. 1. The early Internet standardized global communication around endpoint addressing and packet delivery, leaving higher-level semantics to applications and end systems [4, 5, 33]. Later, web-scale content distribution showed that many workloads are better understood as retrieving and replicating named objects than as maintaining 1
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
Wang, Li, et al.
governance, since sensitive inputs and execution context may need to cross administrative or geographic boundaries. These limitations point to the need for a new network architecture designed specifically for delivering intelligence. We observe that distributed AI services resemble content in CDNs in several important respects: demand is often geographically clustered, requests frequently invoke related capabilities, and many applications benefit when execution remains close to users, data sources, or policy boundaries. CDNs improve performance by moving popular objects closer to demand through locality-aware placement, hierarchical coordination, and selective replication [9, 30]. This naturally raises the question: Can intelligence be delivered in a similar way as CDNs? However, AI service may depend on a base model, adapters, tokenization logic, safety filters, retrieval indices, and reusable runtime state such as prompt prefixes or key-value (KV) caches. These elements have compatibility constraints, hardware requirements, loading costs, trust relationships, and service-level implications. As a result, delivering intelligence requires more than object placement. It requires a common way to describe capabilities, expose execution conditions, guide deployment and routing, reuse state, and establish trust in distributed execution. To this end, we propose the Intelligence Delivery Network (IDN), an Internet architecture for distributed AI services. IDN is designed to make intelligence available where and when it is needed by enabling the network to position, select, and deliver AI capabilities according to demand locality, resource availability, and policy constraints. In this paper, we present the architectural vision of IDN through six core mechanisms: intelligence capability abstraction, compute resource integration, demand-driven capability deployment, capability-aware service routing, state-aware caching, and secure and trustworthy management. We believe that IDN offers a practical path toward an intelligence-centric Internet and establishes an architectural foundation for delivering intelligence as a network service.
2
Cloud
Network operator
Edge
Vision Models
Speech Models
NLP Models
…
Object Det.
Speech ASR
Anomaly Det.
…
Keyword Spotting
Face Recog.
Sensor Analysis
…
ISP/enterprise
Local User/local device
Closer to users, lower latency, stronger privacy
Higher capacity, compute resource, memory
Regional
Figure 2: System assumptions of IDN. AI capabilities are delivered over a hierarchy of cloud, regional, edge, and local resources that differ in capacity, proximity, and administrative control. and local nodes operate under tighter resource constraints but can provide lower latency, better locality, and stronger alignment with local privacy or policy requirements. In some cases, enterprise premises or end devices may also contribute limited capabilities when local execution is required. These locations differ not only in scale, but also in accelerator type, memory capacity, startup overhead, connectivity, reliability, and distance to demand. As a result, intelligence cannot be treated as a uniform deployment object. A monolithic model that fits naturally in a cloud cluster may be infeasible to place at the edge, while a compact task-specialized capability may be more efficient near users than repeatedly invoking a remote foundation model. Similar to CDN, the goal of IDN is not to replicate everything everywhere, but to place the right capability at the right level of the hierarchy.
System Assumptions
Compute resources in the Internet are inherently hierarchical and heterogeneous, residing at different levels of the network and offering different tradeoffs in capacity, locality, and administrative control [27, 34, 36, 46]. Modern CDNs already exploit hierarchical deployment for content delivery by replicating popular objects across multiple layers of infrastructure [9, 30]. IDN considers an analogous setting for AI services, i.e., intelligence should not be served only from distant cloud clusters, but positioned across multiple levels of the network according to resource availability, proximity to demand, and policy constraints, as shown in Fig. 2.
2.1
Large Foundation Model
Data center
2.2
Administrative Domains and Control Boundaries
The deployment environment considered by IDN is also multidomain. Cloud infrastructure may belong to model providers, regional sites may be operated by service providers or large network operators, edge resources may be managed by ISPs, enterprises, or campus networks, and local resources may be controlled by users with their own policies. Consequently, IDN cannot assume a single global controller with complete authority over all resources. Instead, each domain may authenticate nodes, admit deployment objects, expose resource conditions, and enforce local security or compliance rules under its own control. Cross-domain intelligence delivery therefore requires explicit capability descriptions, common deployment interfaces, policy-aware request steering, and verifiable execution evidence rather than implicit trust in a provider-local
Hierarchical and Heterogeneous Deployment
We consider an Internet-scale service environment in which AI capabilities are deployed over a hierarchy of resource domains. Cloud data centers host the largest foundation models, maintain the most complete set of capabilities, and provide abundant compute, memory, and storage. Regional or metro-scale nodes offer more limited but still substantial resources and are suitable for hosting frequently used capabilities within a geographic or administrative region. Edge 2
Intelligence Delivery Network: Toward an Internet Architecture for the AI Age
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
AI Service Request
IDN Control Plane 4. Service Router
1. Capability Registry
5. State Manager
Class, variants, realizations
Plan selection under latency, load, policy
2. Resource Broker
3. Deployment Controller
Resource summary, sensing
Placement, update, rollback
Artifact, prefix, tensor-state, result-caches
6. Trust Manager Identity, attestation, policy, provenance
Capability placement, execution plan
Monitoring data, inference results
Distributed Execution Substrate Cloud
Regional
Edge
Local
large-scale
metro
near-user
private/local
Control flow
Trust influence
Service flow
Figure 3: IDN architecture. The six components form a system for describing, placing, routing, reusing, and securing intelligence across heterogeneous resources.
Service Model
We model an intelligence request as a demand for a class of AI capability under performance and policy constraints. A request may specify, explicitly or implicitly, the desired task, quality target, response-time requirement, privacy or locality constraint, and affinity to reusable state. The purpose of IDN is not merely to identify a reachable endpoint, but to determine where and how this requested capability should be realized within distributed infrastructure. A request may be served in several ways. A nearby node may host a suitable lightweight capability, a regional node may provide a stronger variant with acceptable latency, and a cloud node may be needed for the full capability of a foundation model. The system may also activate a capability, reuse cached state, or route through multiple stages when split execution is beneficial. Thus, service delivery is jointly determined by capability availability, deployment state, network conditions, node resource status, and policy constraints. These considerations motivate the architectural mechanisms described in the next section.
3
Decomposition Domain-specific Model
Foundation LLM
Task-specific Model Low-latency Model
Task: chat Latency: 30ms Mem.: 4GB
User
Task: text gen. Latency: 200ms Mem.: 256GB
Higher model generalizability
2.3
Lower response time
serving stack. This assumption aligns with current computingaware traffic steering works [23, 41].
Task: coding Latency: 60ms Mem.: 16GB
Cloud
Local Regional Edge
Figure 4: Capability abstraction and deployment in IDN. Foundation LLMs are decomposed into multiple capability variants, which are then placed on suitable computing nodes according to resource availability, service requirements, and policy constraints.
IDN Architecture
IDN organizes intelligence delivery around six architectural functions: intelligence capability abstraction, compute resource integration, demand-driven capability deployment, capability-aware service routing, state-aware caching, and security and trust management, as shown in Fig. 3. These functions are tightly coupled. Capability abstraction defines what can be delivered; resource integration determines where it can execute; deployment prepares the substrate on which routing operates; routing determines whether cached state is useful; and security constraints influence every stage of the service path. Table 1 lists the representative architectural descriptors in IDN.
3.1
Intelligence Capability Abstraction
A fundamental challenge for IDN is that the intelligence embodied in today’s foundation models is not naturally aligned with the unit of network delivery. A large model may support many tasks, but as a deployment object it is too coarse-grained for efficient distribution. If intelligence were exposed only as a monolithic checkpoint, nodes would either need to replicate the same full model or give up serving that capability. This is inefficient in heterogeneous environments where requests differ in latency tolerance, accuracy 3
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
Wang, Li, et al.
Table 1: Representative architectural descriptors in IDN. Descriptor
Representative fields
Architectural role
Request descriptor 𝑄
capability, quality target, policy constraints, affinity token, cost budget task, quality tier, response-time requirement, security label, resource requirement, lineage accelerator type, memory, storage, runtime, locality, load, trust level state type, compatibility hash, sharing scope, size, reuse statistics, migration cost selected plan, capability version, node attestations, cache usage, policy verdict
Expresses what intelligence is requested and under which service constraints. Describes a deployable intelligence capability that can be advertised, placed, and matched to requests. Summarizes what a node or domain can execute and under which current conditions. Describes reusable artifacts, prefixes, tensor states, or results for cache management and routing. Records how a request was served for debugging, auditing, and compliance.
Capability descriptor 𝐶 Resource profile 𝑅 State descriptor 𝑆 Execution receipt 𝐸
targets, privacy requirements, and resource budgets. IDN therefore abstracts intelligence into smaller capability units that are meaningful for deployment, as shown in Fig. 4. An intelligence capability is a bounded and deployable unit of service competence derived from one or more foundation models. It does not preserve the full generality of the original model. Instead, it captures the subset of intelligence needed to support a class of requests under explicit quality, latency, security, and resource constraints. Such capabilities may be obtained through distillation, specialization, compression, adapter-based refinement, quantization, or other provider-specific mechanisms [8, 10, 16, 17, 39]. The IDN architecture does not mandate a specific extraction method, only that the resulting unit is semantically meaningful, deployable, and stable enough to support placement, discovery, and routing. IDN views capabilities at three levels. At the top level are capability classes, such as translation, code assistance, visual recognition, or vertical-domain question answering. At the middle level are capability variants, which encode tradeoffs among accuracy, latency, model size, privacy, and trust requirements. At the bottom level are deployable realizations, such as distilled, quantized, or otherwise optimized model instances that implement a variant on a particular hardware type. This separation allows the network to reason about capability classes and variants while allowing local execution platforms to choose the concrete realization that best fits their resources. We represent a capability by a descriptor
Unified Resource Pool
FLOPS: 275TOPS Mem: 16GB Disk: 100GB
FLOPS: 275TOPS Mem: 16GB Disk: 100GB FLOPS: 1000TOPS Mem: 256GB Disk: 100TB
Cloud Data Center Node 1
Node 2
Resource declaration
State exchange
User request
FLOPS: 40TOPS Mem: 8GB Disk: 32GB
Node n Service flow
Figure 5: Unified compute resource pool in IDN. Compute nodes register with the resource pool by advertising their available compute resources and continuously reporting dynamic state information for capability-aware request routing.
granularity, i.e., a capability should be small enough to enable differentiated deployment across heterogeneous nodes, yet large enough to remain stable and meaningful as a service object. The granularity should also be demand-driven. Recurring or high-demand service patterns provide a natural basis for extracting and packaging capabilities because they identify the parts of intelligence most worth placing and reusing in the network. This abstraction makes hierarchical delivery possible. Once intelligence is decomposed into capability units, the network can advertise, deploy, route, and reuse them independently.
𝐶 = ⟨name, task, quality, latency, security, resource, lineage⟩. (1) Here, name denotes the capability class; task specifies the supported function or domain; quality records the expected service level, such as an accuracy tier or context limit; latency captures the response-time characteristic; security captures privacy and trust requirements; resource summarizes compute, memory, and storage requirements; and lineage records how the capability is derived from its originating model family. This descriptor provides the minimal information needed for capability advertisement, deployment planning, request matching, and compatibility checking. Granularity is a key design issue. If the unit is too coarse, deployment degenerates into whole-model replication. If it is too fine, the control overhead of naming, advertising, and composing units may dominate any reuse benefit. IDN therefore adopts a service-effective
3.2
Compute Resource Integration
Distributed compute nodes must be incorporated into IDN as manageable suppliers of execution capability, as shown in Fig. 5. We adopt a hierarchical integration model rather than direct client-side selection of arbitrary nodes. Similar to CDN operations, nodes are admitted, described, and monitored by the control plane of their administrative domain, while requests are steered using aggregated capability views rather than raw node-level exposure. This design 4
Intelligence Delivery Network: Toward an Internet Architecture for the AI Age
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
improves stability and scalability because fast-changing node details do not need to be globally visible. A compute node joins IDN by registering with a domain controller or broker responsible for local resource management. The registration creates a resource profile
or domain 𝑛. A deployment controller can be viewed as minimizing ∑︁ min 𝐿(𝑞, 𝑥) + 𝜆𝐶 deploy (𝑥) + 𝜇𝐶 net (𝑥) 𝑥
𝑞∈ Q
s.t.
+ 𝜈𝐶 risk (𝑥) ∑︁ 𝑥𝑐,𝑛𝑚𝑐 ≤ 𝑀𝑛 ,
(3) 𝑥𝑐,𝑛 ∈ {0, 1}.
𝑐
𝑅 = ⟨hardware, runtime, capacity, state, locality, trust⟩, (2) where hardware captures accelerator type and memory capacity, runtime captures supported serving environments, capacity summarizes static resource limits, state summarizes dynamic load and model residency, locality expresses network and geographic position, and trust records identity and attestation properties. The profile is service-oriented. It exposes what the node can provide to the scheduler, not every hardware detail. Because resource conditions change quickly, integration is not a one-time registration step. Local agents should continuously sense health, queueing condition, available memory, model residency, recent performance, and relevant network measurements. Detailed telemetry remains inside the local domain, while coarse capability summaries are propagated upward or across domains. This hierarchical sensing mirrors the fast local control loops and slower inter-domain coordination in [23, 41]. Once nodes are represented through a common resource view, IDN can support higher-level coordination, including staged execution, multi-node service realization, and adaptive deployment, without requiring a single global optimizer. Resource integration must also be coupled with trust. A node that advertises resources should present verifiable identity and policyrelevant properties before serving protected requests or hosting sensitive capabilities. In this sense, integration is the mechanism by which heterogeneous compute resources become visible, manageable, and trustworthy service suppliers.
3.3
Here, 𝐿(𝑞, 𝑥) captures the service latency or loss for request 𝑞 under placement 𝑥, 𝐶 deploy captures model loading and storage cost, 𝐶 net captures transfer or egress cost, 𝐶 risk captures policy and trust exposure, 𝑚𝑐 is the memory footprint of capability 𝑐, and 𝑀𝑛 is the resource budget of node 𝑛. The exact solver is provider-specific. The key point is that deployment must jointly consider demand locality, resource constraints, network cost, and policy risk. Deployment must also be adaptive. Demand surges, resource fluctuations, node failures, and policy changes can invalidate a previously effective placement. IDN therefore treats deployment as a closed-loop process. It senses demand and infrastructure dynamics, places capabilities accordingly, and revises placement through migration, scale-out, scale-back, and rollback when conditions change. Existing model-serving systems already optimize placement and scheduling within a provider boundary [11, 15, 24, 32, 37], but IDN exposes the interfaces to extend such decisions across a distributed network substrate.
3.4
Capability-Aware Service Routing
Once capabilities are deployed, IDN must decide how an incoming request should be realized. Unlike conventional routing, the goal is not merely to forward a packet to a destination, but to select the execution location, capability variant, and scheduling decision that together satisfy the request. Some requests require stronger models or stricter quality guarantees, while others can be served by lightweight variants with lower latency or cost. Service routing is therefore capability-aware by design, as shown in Fig. 6. We represent a request as
Demand-Driven Capability Deployment
𝑄 = ⟨𝑐, 𝑞, 𝑝, 𝑎, 𝑏⟩,
After capabilities have been abstracted and resources integrated, IDN must decide which capabilities should be placed at which nodes. The objective is not to replicate all capabilities everywhere, but to position the right capability at the right level of the hierarchy. A capability placed too far from demand may incur unnecessary latency and bandwidth cost, while a capability placed too aggressively at constrained nodes may waste scarce memory or reduce system-wide efficiency. Deployment in IDN is demand-driven. User demand is dynamic, unevenly distributed, and often concentrated around recurring service patterns. Therefore, the system should decide not only where existing capabilities are placed, but also when new variants should be constructed or activated. If a task pattern becomes frequent in a region, the system may derive a compact specialization, push it to suitable regional or edge nodes, and later withdraw it when demand subsides. This links capability abstraction to deployment, i.e., the units worth creating are often the ones with enough reuse value under observed demand. A useful high-level formulation is a constrained placement problem. Let 𝑥𝑐,𝑛 ∈ {0, 1} denote whether capability 𝑐 is placed at node
where 𝑐 denotes the requested capability, 𝑞 captures the target service quality, 𝑝 specifies policy or privacy constraints, 𝑎 is an optional state-affinity token, and 𝑏 is an optional resource or cost budget. Given 𝑄, the scheduler determines the feasible set of execution plans Π(𝑄) = {𝜋 | 𝜋 satisfies 𝑐, 𝑞, 𝑝, 𝑏}, (5) where a plan 𝜋 may correspond to a single node or a multi-stage path across nodes. IDN selects 𝜋 ∗ (𝑄) = arg min 𝐽 (𝜋, 𝑄), 𝜋 ∈Π (𝑄 )
(4)
(6)
with 𝐽 (𝜋, 𝑄) = 𝛼𝑇net (𝜋) + 𝛽𝑇queue (𝜋) + 𝛾𝑇exec (𝜋) + 𝛿𝑇state (𝜋, 𝑎) + 𝜖𝐶 load (𝜋) + 𝜁 𝑃policy (𝜋, 𝑄).
(7)
Here, 𝑇net denotes network transfer delay, 𝑇queue queueing delay, 𝑇exec execution time, 𝑇state state transfer or reconstruction cost, 𝐶 load a load-balancing penalty, and 𝑃policy a policy or trust penalty. The weights are deployment-specific. The key is that routing, scheduling, and policy enforcement are correlated with each other for AI services. 5
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
Wang, Li, et al.
We distinguish four classes of cacheable objects. Artifact caches store relatively stable deployment objects such as model weights, adapters, quantized variants, and runtime packages. Prefix caches store reusable prompt modules, system instructions, retrieved-context templates, or other common input prefixes. Tensor-state caches store intermediate execution state, such as KV caches or other activations, that can accelerate subsequent inference. Result caches store reusable outputs for identical or semantically equivalent inputs under compatible model, decoding, and policy settings. They may include full responses when execution is deterministic or constrained, as well as deterministic subcomputations such as embeddings, retrieval outputs, or tool results [12, 20, 25]. Orthogonal to type, each cached state has a sharing scope: public, tenant-shared, sessionprivate, or hardware-bound. A cached state object is described by a state descriptor containing its type, compatibility hash, size, sharing scope, reuse statistics, privacy label, decoding configuration when applicable, and migration cost. Cache admission can be expressed as
Redirec t to clou
d
Model Pool On the Cloud Capability distribution Node 1
Node 2
Node 3
Node 4
Model: A
Model: B
Cache miss
CPU: 50%
CPU: 90%
BW: 5Mbps
BW: 1Gbps
Node 5 Task: driving Model: A
Request 1
Priority: High
Request 2
Task: chat Model: B Priority: Low
Figure 6: Service routing in IDN. IDN selects suitable compute nodes by jointly considering task requirements, deployed node capabilities, current resource states, and policy constraints.
Benefit(𝑥, 𝑄) = 𝑃 hit (𝑥, 𝑄)Δ𝐿 −𝐶 transfer (𝑥) −𝐶 storage (𝑥) −𝐶 privacy (𝑥), (8) where 𝑃hit (𝑥, 𝑄) is the predicted reuse probability of state 𝑥 for request class 𝑄, Δ𝐿 is the expected latency reduction, and the remaining terms capture transfer, storage, and privacy costs. This formulation highlights that caching cannot be designed independently from routing and deployment. A cache is valuable only if future requests are likely to be routed to locations where it can be reused safely and efficiently. IDN also supports cooperative caching across nodes. In generative AI workloads, state may be too large or too expensive to move as a single object. Prefill-decode separation, for example, may let compute-rich nodes perform prefill while latency- or memoryoptimized nodes perform decode. The associated KV state must then be transferred, compressed, partitioned, or selectively retained along the execution path [25, 31, 45]. Network awareness is necessary because reuse cost depends on bandwidth, congestion, path latency, and the communication pattern required to access or move state. The IDN architecture requires that cache policy must jointly consider model structure, compute resources, network conditions, routing decisions, and service objectives.
State-awareness is especially important. Interactive sessions, long-context inference, and split execution accumulate valuable intermediate state, including reusable prefixes and KV caches. If execution switches nodes, the system may need to move, reconstruct, or discard this state, which can dominate the latency gain from a shorter path. IDN therefore treats state affinity as a key routing consideration: the scheduler should preserve locality when reuse is valuable, but gradually divert traffic when queueing delay, contention, or resource pressure dominates. This is consistent with recent LLM-serving work showing that prefill and decode phases, KV-cache management, and state movement strongly affect tail latency and goodput [1, 20, 21, 31, 35, 42, 45]. Routing must also avoid hotspot formation. Demand, node load, and network conditions vary over time, and naive affinity-based routing may overload popular nodes. The load penalty in 𝐽 (𝜋, 𝑄) provides an architectural handle for balancing short-term locality against long-term service stability. Under overload, the system may route to a compatible warm instance, select a cheaper capability variant, reduce the admissible context window, or reject requests early rather than allowing unbounded queue growth. Thus, IDN routing is a service-quality control loop, not only a path-selection mechanism.
3.5
3.6
Security and Trust Management
Security in IDN must be considered in the context of distributed and collaborative inference. A service request may involve capabilities derived from different models, nodes operated by different domains, cached states produced by previous requests, and intermediate results exchanged across the network. Protecting only the transport channel is therefore insufficient. IDN must provide architectural support for ensuring that data is handled under appropriate constraints, model components are trustworthy, and inference results can be associated with a reliable execution process. A key principle is to make trust visible to the architecture. Compute nodes should expose verifiable identities and security-relevant properties before serving protected requests. Similarly, intelligence capabilities should carry authenticated metadata about origin, version, dependency chain, and admissible policy scope. This allows the system to reason not only about whether a node has enough
State-Aware Caching
Caching is essential for improving the efficiency and service quality of distributed AI systems. Unlike conventional content caching, AI-serving caches contain heterogeneous objects with different correctness, privacy, and mobility constraints. IDN therefore treats caching as state management rather than simple object reuse. The goal is to decide what state should be cached, where it should be placed, how it should be shared, and when it should be migrated or invalidated. 6
Intelligence Delivery Network: Toward an Internet Architecture for the AI Age
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
resources, but also about whether it is authorized and trusted to execute a particular request. In multi-domain deployments, such information is necessary for coordinating authentication and authorization across operators with different policies and trust assumptions. Security constraints should participate in deployment and routing. Different requests impose different privacy, locality, and isolation requirements. A latency-critical request without strict locality constraints may be routed to a nearby lightweight capability, whereas a request involving private or regulated data may require local execution, stronger isolation, or a node within a specific administrative domain. These choices trade off privacy, performance, and resource efficiency. IDN therefore treats security and privacy as scheduling constraints rather than after-the-fact application checks. Trusted execution environments, confidential VMs, and verifiable inference techniques provide useful building blocks, but they should be exposed through capability and resource descriptors rather than hidden inside provider-specific implementations [2, 3, 6, 28, 38, 44]. IDN must also account for the full lifecycle of intelligence delivery. Model components, adapters, runtime packages, cached states, and execution environments can all become attack surfaces. Model backdoors and poisoned components show that the ML supply chain itself can be compromised [13, 26]. The architecture should therefore support provenance tracking, policy-based access control, runtime monitoring, and invalidation or rollback of unsafe components. Execution receipts can record capability versions, node attestations, cache usage, and policy decisions for auditing and incident response. The goal is not to mandate a single defense mechanism, but to provide common hooks for maintaining trust in distributed inference.
4
provider-local model serving is insufficient for network-level intelligence delivery.
4.2
A practical IDN prototype should separate architectural functions from provider-specific serving implementations. At a high level, such a prototype would include a capability registry, resource broker, deployment controller, service router, state/cache manager, and trust manager. These components can be built as an overlay on existing infrastructure such as Kubernetes, service meshes, Ray, GPU inference runtimes, and model-serving frameworks [7, 19, 20, 29, 42]. IDN need not replace these systems. Rather, it exposes capability, resource, state, and policy information across them. A useful testbed should span cloud clusters, regional nodes, edge servers, and local or on-premise nodes. Such a testbed would allow researchers to study cross-layer questions that single-cluster inference benchmarks cannot capture, including capability placement, state migration, load-aware routing, privacy-aware scheduling, failure recovery, and multi-domain trust.
4.3
Evaluation and Incremental Deployment
Evaluating IDN requires metrics beyond network throughput or single-cluster inference speed. From the user perspective, the metrics include time-to-first-token, time-per-output-token, tail latency, completion rate, and degradation under overload. From the system perspective, IDN should be evaluated by accelerator utilization, memory pressure, model loading overhead, cache hit ratio, statetransfer cost, and load-balancing effectiveness. From the network perspective, relevant metrics include wide-area traffic reduction, inter-domain transfer cost, and sensitivity to congestion or failures. The deployment of IDN shall proceed incrementally. The first stage may occur within a single provider domain that already controls cloud and edge resources. The next stage is integration with CDN-like and edge infrastructures, extending their control planes from content objects to intelligence capabilities and reusable state. Cross-domain intelligence delivery requires stronger standardization of capability descriptions, resource summaries, state descriptors, and execution evidence. Therefore, it requires gradually standardizing the interface needed to discover, deploy, route, reuse, and trust distributed AI services.
IDN Architecture Development
The previous sections describe the core architectural mechanisms of IDN. A natural next question is how such an architecture can be developed, validated, and deployed. We view IDN as an architecture that should evolve through application-driven design, prototype systems, experimental testbeds, and incremental deployment.
4.1
Prototype System and Testbed
Application-Driven Design
IDN should be developed toward concrete AI applications. Different applications stress different parts of the architecture. Interactive assistants and copilots emphasize low latency, session continuity, and state reuse. Real-time translation and AR/VR applications require predictable response time and close placement of lightweight capabilities. Enterprise retrieval-augmented generation stresses policy compliance, locality of private data, and controlled reuse of retrieved context [22]. Robotics and multimodal perception introduce stronger requirements on reliability, edge execution, and graceful degradation. These applications are design drivers as well as evaluation workloads. They help determine the right granularity of capability abstraction, the fields exposed in descriptors, the resource states that must be advertised, and the security constraints that should influence deployment and routing. In this sense, application development provides feedback into the architecture by revealing where
5
Open Questions
IDN raises several research questions. A central issue is the abstraction boundary exposed to the network. This paper argues for deployable capabilities and associated descriptors because they make placement, routing, and state reuse visible to the architecture. However, future AI services may involve multi-agent workflows, tool-using pipelines, or dynamically composed reasoning processes. The appropriate network-visible unit for such services may be richer than a single capability. Another challenge is the economic and operational model of cross-domain intelligence delivery. CDNs and cloud platforms have historically hidden settlement, peering, and placement logic behind provider-specific contracts. IDN makes the situation more complex because execution capacity, cached state, model placement, and policy compliance may all carry economic value. If intelligence 7
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
Wang, Li, et al.
can be placed and served across domains, the architecture may require explicit mechanisms for accounting, incentive alignment, and policy negotiation. State mobility is also underexplored. Distributed inference creates many forms of reusable state, including prompt prefixes, KV caches, intermediate activations, retrieval results, and session context. Moving such state can reduce repeated computation, but it may consume bandwidth, increase latency, violate compatibility constraints, or weaken privacy guarantees. The question of when state should move, be recomputed, or be discarded is both a system problem and a networking problem. Trust and verification introduce another set of open problems. Provenance, signed capability metadata, and node attestation can establish where a request executed and which components were involved, but they do not prove semantic correctness or eliminate all side channels. Practical IDN deployments will need combinations of attestation, runtime monitoring, selective verification, privacypreserving inference, and auditing [28, 38, 44]. Finally, progress on IDN will require shared traces and benchmarks. Without workloads that capture model popularity, request locality, context reuse, accelerator heterogeneity, privacy constraints, and multi-domain operation, it will be difficult to compare placement algorithms, routing policies, caching strategies, or fairness mechanisms. Building this empirical foundation may be as important as designing any single control algorithm.
6
[6] Victor Costan, Ilia Lebedev, and Srinivas Devadas. 2016. Sanctum: Minimal hardware extensions for strong software isolation. In 25th USENIX Security Symposium (USENIX Security 16). USENIX Association, Austin, United States, 857–874. [7] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, United States, 613–627. [8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in neural information processing systems. Curran Associates, New Orleans, United States, 30318–30332. [9] John Dilley, Bruce M Maggs, Jay Parikh, Harald Prokop, Ramesh Sitaraman, and Bill Weihl. 2002. Globally distributed content delivery. IEEE Internet Computing 6, 5 (2002), 50–58. [10] William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. [11] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low-Latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, United States, 135–153. [12] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems. PMLR, Santa Clara, United States, 325–338. [13] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. ArXiv abs/1708.06733 (2017), 1–13. [14] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Virtual Event, 443–462. [15] Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R Das. 2022. Cocktail: A multidimensional optimization for model serving in cloud. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, United States, 1041–1057. [16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. ArXiv abs/1503.02531 (2015), 1–9. [17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ArXiv abs/2106.09685 (2022), 1–26. [18] Van Jacobson, Diana K Smetters, James D Thornton, Michael F Plass, Nicholas H Briggs, and Rebecca L Braynard. 2009. Networking named content. In Proceedings of the 5th international conference on Emerging networking experiments and technologies. Association for Computing Machinery, Rome, Italy, 1–12. [19] Kubernetes Authors. 2026. Kubernetes: Production-Grade Container Orchestration. https://kubernetes.io/ Accessed May 6, 2026. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. Association for Computing Machinery, Koblenz, Germany, 611–626. [21] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, United States, 155–172. [22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in neural information processing systems. Curran Associates, Inc., Vancouver, Canada, 9459–9474. [23] Cheng Li, Zongpeng Du, Mohamed Boucadair, Luis M Contreras, and J Drake. 2024. A framework for computing-aware traffic steering (CATS). https: //datatracker.ietf.org/doc/draft-ietf-cats-framework/ Accessed May 6, 2026. [24] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. 2023. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, United States, 663–679. [25] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference. Association for Computing Machinery, Sydney, Australia, 38–56.
Looking Forward
IDN is not intended to replace today’s Internet or centralized AI clouds. Rather, it identifies a path for extending the Internet so that distributed AI services can be delivered more efficiently, reliably, and securely. As AI workloads become increasingly interactive, personalized, multimodal, and privacy-sensitive, the network must reason not only about reachability, but also about capability, resource availability, reusable state, and policy constraints. This paper presented IDN as an architectural framework for this transition. Its key idea is to make intelligence a deliverable network service through the six interdependent mechanisms. Looking forward, IDN should be viewed as the next generation of Internet architecture for the AI age, which defines the abstractions and interfaces needed for the Internet to actively participate in delivering intelligence, rather than merely transporting requests to remote AI services.
References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). USENIX Association, Santa Clara, United States, 117–134. [2] AMD. 2020. AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More. https://docs.amd.com/v/u/en-US/SEV-SNP-strengthening-vmisolation-with-integrity-protection-and-more Accessed May 6, 2026. [3] Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O’keeffe, Mark L Stillwell, et al. 2016. SCONE: Secure linux containers with intel SGX. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, United States, 689–703. [4] Vinton Cerf and Robert Kahn. 1974. A protocol for packet network intercommunication. IEEE Transactions on communications 22, 5 (1974), 637–648. [5] David Clark. 1988. The design philosophy of the DARPA Internet protocols. In Symposium proceedings on Communications architectures and protocols. Association for Computing Machinery, Stanford, United States, 106–114. 8
Intelligence Delivery Network: Toward an Internet Architecture for the AI Age
under submission to ACM SIGCOMM CCR, Volume xx Issue x, 2026
[37] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). USENIX Association, Santa Clara, United States, 173–191. [38] Florian Tramer and Dan Boneh. 2018. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. ArXiv abs/1806.03287 (2018), 1–19. [39] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning. PMLR, Honolulu, United States, 38087–38099. [40] George Xylomenos, Christopher N Ververidis, Vasilios A Siris, Nikos Fotiou, Christos Tsilopoulos, Xenofon Vasilakos, Konstantinos V Katsaros, and George C Polyzos. 2013. A survey of information-centric networking research. IEEE communications surveys & tutorials 16, 2 (2013), 1024–1049. [41] Kehan Yao, Dirk Trossen, Mohamed Boucadair, Luis M Contreras, Hang Shi, Yizhou Li, Shuai Zhang, and Qing An. 2024. Computing-aware traffic steering (CATS) problem statement, use cases, and requirements. https://datatracker.ietf. org/doc/draft-ietf-cats-usecases-requirements/ Accessed May 6, 2026. [42] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and ByungGon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22). USENIX Association, Carlsbad, United States, 521–538. [43] Lixia Zhang, Alexander Afanasyev, Jeffrey Burke, Van Jacobson, KC Claffy, Patrick Crowley, Christos Papadopoulos, Lan Wang, and Beichuan Zhang. 2014. Named data networking. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 66–73. [44] Wenting Zheng, Ankur Dave, Jethro G Beekman, Raluca Ada Popa, Joseph E Gonzalez, and Ion Stoica. 2017. Opaque: An oblivious and encrypted distributed analytics platform. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, United States, 283–298. [45] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, United States, 193–210. [46] Zhi Zhou, Xu Chen, En Li, Liekang Zeng, Ke Luo, and Junshan Zhang. 2019. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proc. IEEE 107, 8 (2019), 1738–1762.
[26] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Society, San Diego, United States, 1–15. [27] Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled B Letaief. 2017. A survey on mobile edge computing: The communication perspective. IEEE communications surveys & tutorials 19, 4 (2017), 2322–2358. [28] Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. 2020. Delphi: A cryptographic inference system for neural networks. In Proceedings of the 2020 workshop on privacy-preserving machine learning in practice. Association for Computing Machinery, Virtual Event, 27–30. [29] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. 2018. Ray: A distributed framework for emerging AI applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18). USENIX Association, Carlsbad, United States, 561–577. [30] Erik Nygren, Ramesh K Sitaraman, and Jennifer Sun. 2010. The akamai network: a platform for high-performance internet applications. ACM SIGOPS Operating Systems Review 44, 3 (2010), 2–19. [31] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 118–132. [32] Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, Virtual Event, 397–411. [33] Jerome H Saltzer, David P Reed, and David D Clark. 1984. End-to-end arguments in system design. ACM Transactions on Computer Systems (TOCS) 2, 4 (1984), 277–288. [34] Mahadev Satyanarayanan. 2017. The emergence of edge computing. computer 50, 1 (2017), 30–39. [35] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. 2024. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, United States, 965–988. [36] Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. 2016. Edge computing: Vision and challenges. IEEE internet of things journal 3, 5 (2016), 637–646.
9