arXiv CSopen access

[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows

arXiv:2607.10987v1 [cs.DC] 13 Jul 2026

[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows Arup Kumar Sarker Alexander James Halpern Mills Staylor

Gregor von Laszewski Geoffrey Fox Yue Cheng

University of Virginia, Biocomplexity Institute and Initiative Charlottesville, VA, USA [email protected] [email protected] [email protected]

Biocomplexity Institute and Initiative University of Virginia Charlottesville, VA, USA [email protected] [email protected] [email protected]

Aymen Alsaadi Shantenu Jha Rutgers University Princeton Plasma Physics Laboratory Princeton, NJ, USA [email protected] [email protected]

ABSTRACT

1

Multi-agent LLM systems increasingly integrate retrieval, planning, and reasoning, but remain fundamentally text-centric, requiring agents to repeatedly recompute shared context through expensive prefill. Although single-request inference is known to be accelerated by KV-cache management, it is usually restricted to local serving scopes. We introduce AAFLOW+, a stateful extension of agentic workflow operators that makes KV cache a first-class distributed systems object. AAFLOW+ builds processes into communication-aware graphs that concurrently optimize data, prompts, and reusable model state. It also provides operators for KV materialization, transfer, fork, composition, and eviction. Its runtime enables zero-copy, transfer-aware execution, allowing agents to reuse long context without recomputation. AAFLOW+ reduces TTFT by up to 50.2×, achieves up to 7.63× reduced multi-agent compute cost at 16-agent scale, reduces KV memory by 1.72–6.10×, and increases throughput by more than 7.74×, based on an analytical cost model parameterized by empirical hardware microbenchmarks. The results demonstrate that KV transmission outperforms recomputation on networks with moderate to high bandwidth, making sure KV-state sharing greatly increases efficiency in multi-agent LLM systems by replacing text passing.

Large language models are increasingly deployed as agentic systems that interleave retrieval, reasoning, tool invocation, and memory across multiple stages. Frameworks such as ReAct, Reflexion, AutoGen, and DSPy expand this design space, while RAG improves factual grounding by conditioning generation on external data [16, 20, 38, 41, 42]. However, as these workflows become deeper and more collaborative, a fundamental systems bottleneck emerges: agents continue to communicate primarily through text. Even when shared context has already been computed, downstream agents must replay it through costly prefill computation, repeatedly reconstructing identical model state. This inefficiency is closely tied to the treatment of the key-value (KV) cache, a central object in LLM inference and a key component of AI memory [22]. Recent serving systems such as vLLM [19], SGLang [47], DistServe, MemServe, Mooncake, ChunkAttention, KVCOMM, and RelayCaching [12, 15, 29, 43, 44, 49] demonstrate that KV cache management is critical for reducing latency and improving throughput. These systems optimize KV reuse through techniques such as block-level allocation, prefix sharing, and decoupled prefill and decode stages. However, their scope is largely confined to single-request or single-cluster execution. They do not provide a workflow-level abstraction for explicitly transferring, branching, and reusing KV state across multiple agents, which impacts pipeline execution. In planner–retriever–solver workflows, multiple agents often operate over the same long context prefix. In tree-of-thought or debate settings, agents branch from shared prefixes and explore alternative reasoning paths. In collaborative RAG, agents independently retrieve overlapping evidence and reprocess similar context. In all cases, current systems serialize shared state back into text, transforming a stateful execution problem into repeated prompt replay. The result is increased time-to-first-token (TTFT), duplicated computation, higher framework overhead, and limited control over state placement and reuse. Recent frameworks like AAFLOW [34], model agentic workflows as compositions of distributed operators and reduce orchestration overhead via communication-aware DAG execution and zero-copy data transfer. However, AAFLOW remains data-centric and does not expose internal model state, such as KV cache, as part of its abstraction. We introduce AAFLOW+, a stateful extension that elevates KV cache to a first-class distributed systems

PVLDB Reference Format: Arup Kumar Sarker, Alexander James Halpern, Mills Staylor, Gregor von Laszewski, Geoffrey Fox, Yue Cheng, Aymen Alsaadi, and Shantenu Jha. [AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows. PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available in AAFLOW/stateful_agentic_algebra directory at https://github.com/aru pcsedu/AAFLOW. This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX

INTRODUCTION

object. AAFLOW+ extends operator abstraction from dataflow to stateflow, enabling explicit modeling of KV-state lifecycle through operators for materialization, transfer, fork, restricted composition, and eviction (Figure 1). This lets the compiler choose between text passing, KV reuse, state transfer, and eviction, shifting optimization from prompt construction to explicit state management. Our runtime makes the KV cache transportable by using explicit metadata, zero-copy communication, and transfer-aware scheduling. By separating metadata from tensor buffers and using Arrow-based formats, the system minimizes serialization costs and enables efficient state transfer between distributed components [2]. AAFLOW+ ensures correctness by enforcing compatibility on model identity, positional encoding, and execution history, allowing safe prefix reuse and controlled state composition.

tool use, memory access, and response generation interact dynamically. While frameworks like DSPy treat language-model applications as compositional programs that can be optimized through compilation [16, 41], systems like AutoGen and LangGraph offer flexible multi-agent programming abstractions. Nonetheless, the majority of agent frameworks continue to be largely text-centric, with agents exchanging serialized strings representing tool outputs, recovered excerpts, intermediate summaries, and natural language messages. Although this approach is practical at the application layer, it conceals a more serious inefficiency in the system. A downstream agent typically needs to re-tokenize and re-prefill the same context before it can produce fresh output when it receives text that has previously been handled by an upstream model invocation. Instead of viewing agentic workflows as loosely connected framework callbacks, AAFLOW [34] treated them as compilable distributed programs. In AAFLOW, a workflow is represented as a set of operators, 𝑊 = {𝑂𝑝𝑒𝑚𝑏𝑒𝑑 , 𝑂𝑝𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒 , 𝑂𝑝𝑟𝑒𝑎𝑠𝑜𝑛 , 𝑂𝑝𝑚𝑒𝑚𝑜𝑟 𝑦 , 𝑂𝑝𝑢𝑝𝑠𝑒𝑟𝑡 },

(1)

where each operator is defined as 𝑂𝑝𝑖 = (𝐼𝑖 , 𝑂𝑖 , 𝑓𝑖 , 𝑃𝑖 ).

(2)

Here, 𝐼𝑖 and 𝑂𝑖 denote the input and output objects, 𝑓𝑖 is the transformation function, and 𝑃𝑖 is the communication pattern associated with the operator. This abstraction allows the runtime to lower an agentic workflow into an execution graph, 𝐺 = 𝐶𝑜𝑚𝑝𝑖𝑙𝑒 (𝑊 ),

(3)

where communication patterns such as broadcast, shuffle, reduce, and embarrassingly parallel execution become explicit parts of the execution plan. The key insight of this paper is that the same principle should apply not only to external workflow data, such as documents, embeddings, and vector indices, but also to internal model execution state. The state created during LLM prefill is typically locked inside a local serving runtime in current agentic systems. Because of this, multi-agent workflows maximize the flow of data throughout the model, but they are unable to optimize the transportation and reuse of the key-value (KV) cache, the model’s most costly intermediate artifact.

Figure 1: Layered architecture of stateful operator abstraction. The design extends AAFLOW-style operator execution with explicit KV-state materialization, fork, transfer, merge, and eviction. We evaluate AAFLOW+ on GPU clusters using Mistral-7B and Llama-3-8B. Results show that stateful execution significantly improves performance over text-based orchestration. AAFLOW+ reduces TTFT by up to 50.2× at long context, achieves up to 7.63× lower multi-agent latency at 16-agent scale, reduces peak KV memory by 1.72×–6.10×, and improves throughput by over 7.74×. Transfer–recompute analysis shows that KV transfer dominates recomputation on moderate-to-high bandwidth networks, while consistency experiments preserve deterministic output agreement. These results show that workflow-level KV-state sharing reduces latency, memory pressure, and framework overhead while complementing existing LLM serving systems.

Figure 2: Textflow forces downstream agents to replay context, whereas stateflow transfers reusable KV execution state.

2.2

Cost of Text-Based Agent Communication

Consider a two-agent workflow where an upstream agent gets documents and creates a lengthy context, and a downstream agent uses that context to perform reasoning in order to comprehend the limitations of text-based communication. In existing systems, text is usually emitted by the upstream agent and consumed as a new prompt by the downstream agent. Before generating its first output

2 BACKGROUND AND MOTIVATION 2.1 From Agentic Dataflow to Stateful Execution Agentic large language model (LLM) applications are increasingly organized as multi-stage workflows in which retrieval, reasoning, 2

2.4

token, the downstream model must make a prefill pass over the whole context if the prompt length is 𝐿. We denote this cost as 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿).

(4)

The total execution time for a single agent invocation can therefore be approximated as 𝑇𝑎𝑔𝑒𝑛𝑡 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) + 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 ) + Ω,

(5)

where 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 ) is the cost of autoregressively generating an output sequence of length 𝑌 , and Ω captures framework overhead from scheduling, serialization, synchronization, and data movement. The inefficiency becomes more severe in multi-agent settings. If 𝑘 agents consume the same context independently, the system may pay the prefill cost 𝑘 times: 𝑘 𝑇𝑡𝑒𝑥𝑡 ≈ 𝑘 · 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) +

𝑘 ∑︁

𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 𝑗 ) + Ω𝑡𝑒𝑥𝑡 .

Motivating Stateflow Example

Figure 2 illustrates the difference between textflow and stateflow. Agent B needs to prefill the textual context that Agent A emits in the textflow scenario. Agent A produces a reusable KV state object in addition to a logical output in the stateflow scenario. Then, instead of paying for complete prompt replay, Agent B can resume execution from the transferred state, simply paying for state movement and continuation. By contrasting the cost of state transfer with the cost of text replay, the anticipated benefit may be stated. Text-based execution is beneficially replaced by stateful execution when 𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 (𝐾𝑉 ) + 𝑇𝑟𝑒𝑠𝑢𝑚𝑒 + Ω𝑠𝑡𝑎𝑡𝑒 < 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) + Ω𝑡𝑒𝑥𝑡 .

(7)

Equation 7 defines the central systems condition studied in this paper. Multi-agent workflows can lower TTFT and overall latency by working over state rather than text if transferring and resuming from KV state is less expensive than replaying the prompt.

(6)

𝑗=1

3 STATEFUL OPERATOR ABSTRACTION 3.1 Extending the Operator Model Each operator is described by AAFLOW [34] as a tuple with inputs, outputs, a transformation function, and a communication pattern. For data-centric workflow steps including embedding, retrieval, reasoning, memory lookup, and index updating, this approach is adequate. It does not, however, specifically differentiate between model execution state and ordinary data. Because KV cache is connected to model settings, token placements, attention layout, device placement, and branch lineage, it is distinct from regular data. Reusing it inappropriately can modify model meaning, while discarding it unnecessarily produces wasteful processing. We therefore extend the original operator definition into a stateful operator:

Figure 3: Text-based multi-agent execution duplicates prefill computation across branches that share the same context. Tree-of-Thought reasoning, multi-agent argument, self-consistency sampling, speculative planning, and parallel retrieval synthesis all exhibit this execution pattern. Although the process in each scenario has a shared prefix or shared context, text-based communication compels each branch to separately recreate the model state. This redundancy is shown in Figure 3. The execution system handles each branch as a separate model invocation even though the agents conceptually share a shared context. Poor branch factor scalability is the outcome, particularly for long-context workloads where time-to-first-token (TTFT) is dominated by prefill.

2.3

𝑂𝑝𝑖𝑠 = (𝐼𝑖 , 𝑂𝑖 , 𝑆𝑖𝑖𝑛 , 𝑆𝑖𝑜𝑢𝑡 , 𝑓𝑖 , 𝑃𝑖 , 𝜎𝑖 ).

(8)

Here, 𝐼𝑖 and 𝑂𝑖 remain the ordinary data inputs and outputs, while 𝑆𝑖𝑖𝑛 and 𝑆𝑖𝑜𝑢𝑡 represent input and output state objects. The function 𝑓𝑖 describes the logical transformation performed by the operator, and 𝑃𝑖 describes the communication pattern used to execute it. The new term 𝜎𝑖 denotes the state policy associated with the operator. This policy determines whether state is transferred, aliased, forked, merged, pinned, evicted, or recomputed. Making state movement transparent to the compiler is the goal of Equation 8. The runtime only observes strings and tensors moving between operators in a text-based workflow. The runtime also detects state dependencies in a stateful workflow, which may affect placement and scheduling. Thus, the execution graph becomes

KV Cache as Reusable Execution State

Serving systems have already demonstrated that performance depends on appropriate KV-cache management. To manage KV cache using block-level paging and minimize memory fragmentation [19], vLLM provides PagedAttention. To enhance KV-cache reuse across structured language-model programs, SGLang introduces RadixAttention [47]. These systems show that effective LLM execution depends on the KV cache. Their optimizations, however, mostly function within a serving runtime. A universal distributed abstraction for exposing KV cache as a schedulable state object across multi-agent workflows is not offered by them. Our work is motivated by this distinction. We don’t try to replace the current serving systems. Rather, we investigate the possibility of lifting KV cache from a local serving optimization into a distributed workflow abstraction. According to this perspective, a multi-agent workflow should be able to manage partitions, buffers, and interim outputs in the same manner as distributed data systems may materialize, transmit, fork, reuse, and ultimately evict KV state.

𝐺𝑠 = (𝑉 , 𝐸𝑑 , 𝐸𝑠 ),

(9)

where 𝑉 is the set of operators, 𝐸𝑑 is the set of data edges, and 𝐸𝑠 is the set of state edges. Data edges describe conventional workflow dependencies, while state edges describe KV-cache reuse and movement.

3.2

KV State Object

A KV cache object must carry enough information to determine whether it can be safely reused. We define a state object as 𝑆𝐾𝑉 = (𝑀, Θ, 𝐵, Π, Λ, Γ), 3

(10)

where 𝑀 is the model identifier, Θ is the model and tokenizer configuration, 𝐵 is the set of KV blocks, Π is positional metadata, Λ is lineage metadata, and Γ describes placement and ownership. The block set 𝐵 is represented as 𝐵 = {𝑏 1, 𝑏 2, . . . , 𝑏𝑚 },

The merge discipline is specified by the policy 𝜇. Since arbitrary KV-cache merging is not typically semantically correct, we purposefully limit this operation. Merge is only permitted under specific structural limitations, such as prefix-compatible concatenation, segment-aware assembly, or reduction through a summarizing model invocation, because KV state is position-dependent and model-layout-dependent. KV cache is not treated as a general tensor object by this cautious approach.

(11)

where each block has the form 𝑏 𝑗 = (𝐾 𝑗 , 𝑉𝑗 , ℓ 𝑗 , 𝑟 𝑗 , 𝑑 𝑗 ).

(12)

3.4

The key and value tensors in this representation are 𝐾 𝑗 and 𝑉𝑗 , the layer index is ℓ 𝑗 , the token-position range covered by the block is 𝑟 𝑗 , and the device or memory domain where the block is now located is 𝑑 𝑗 . The idea behind this block-level representation is similar to that of paged KV-cache management in vLLM, which manages cache storage in blocks as opposed to a single monolithic sequence [19]. The lineage term Λ is particularly important for multi-agent workflows. It records whether a state was produced by direct prefill, by transfer, by fork, or by restricted composition. This prevents the runtime from treating all KV states as interchangeable. Two states are compatible only if they satisfy the predicate

The operator abstraction is useful only if it exposes an optimization target. For text-based execution, the cost of a branch with context length 𝐿 can be written as 𝑇𝑡𝑒𝑥𝑡 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) + 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 ) + Ω𝑡𝑒𝑥𝑡 .

(18)

For stateful execution, the corresponding cost is 𝑇𝑠𝑡𝑎𝑡𝑒 = 𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 (𝑆𝐾𝑉 ) + 𝑇𝑟𝑒𝑠𝑢𝑚𝑒 + 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 ) + Ω𝑠𝑡𝑎𝑡𝑒 .

(19)

The system should choose stateful execution when (20)

𝑇𝑠𝑡𝑎𝑡𝑒 < 𝑇𝑡𝑒𝑥𝑡 . Expanding this condition gives

𝐶𝑜𝑚𝑝𝑎𝑡 (𝑆𝑎 , 𝑆𝑏 ) = ⊮[𝑀𝑎 = 𝑀𝑏 ] · ⊮[Θ𝑎 = Θ𝑏 ] · ⊮[Π𝑎 ∼ Π𝑏 ], (13)

𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 (𝑆𝐾𝑉 ) + 𝑇𝑟𝑒𝑠𝑢𝑚𝑒 + Ω𝑠𝑡𝑎𝑡𝑒 < 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) + Ω𝑡𝑒𝑥𝑡 .

where Π𝑎 ∼ Π𝑏 denotes positional compatibility. This predicate is conservative by design: when compatibility cannot be established, the runtime falls back to text replay or recomputation.

3.3

Cost Model for Stateful Execution

(21)

For a branching workflow with 𝑘 branches sharing a prefix, the text-based cost is approximately 𝑘 𝑇𝑡𝑒𝑥𝑡 = 𝑘 · 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) +

Stateful Operator Semantics

𝑘 ∑︁

𝑘 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 𝑗 ) + Ω𝑡𝑒𝑥𝑡 .

(22)

𝑗=1

The abstraction introduces a family of KV-state operators that extend the original AAFLOW workflow. The first operator is materialization: 𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 (𝑥, 𝑀) → 𝑆𝐾𝑉 .

With KV-state fork, the prefix is materialized once: 𝑘 𝑇𝑠𝑡𝑎𝑡𝑒 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) +𝑇𝑓 𝑜𝑟𝑘 (𝑆𝐾𝑉 , 𝑘) +

(14)

𝑘 ∑︁

𝑘 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 (𝑌 𝑗 ) + Ω𝑠𝑡𝑎𝑡𝑒 . (23)

𝑗=1

This operator creates a reusable KV state object after consuming a tokenized context 𝑥 and a model 𝑀. Materialization distinguishes between the cost of decoding future tokens and the cost of processing the input context, as contrast to regular generation. The second operator is transfer: ′ 𝑂𝑝𝑘𝑣_𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 (𝑆𝐾𝑉 , 𝑛𝑎 , 𝑛𝑏 ) → 𝑆𝐾𝑉 .

The expected savings therefore come from replacing repeated prefill with a single materialization and a cheaper fork operation: 𝑘 𝑘 Δ𝑇 = (𝑘 − 1)𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) − 𝑇𝑓 𝑜𝑟𝑘 (𝑆𝐾𝑉 , 𝑘) − (Ω𝑠𝑡𝑎𝑡𝑒 − Ω𝑡𝑒𝑥𝑡 ). (24)

This equation makes the performance intuition precise. Stateful execution is most beneficial when the shared context is long, the branch factor is high, and state management overhead remains lower than the avoided prefill cost.

(15)

A state object is moved or aliased from node 𝑛𝑎 to node 𝑛𝑏 by this operation. Depending on where the source and destination are located, its communication pattern could be point-to-point transfer, RDMA, or a collective operation. The transfer operator modifies the state’s location and ownership metadata but not its logical content. The third operator is fork: (1) (2) (𝑘 ) 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 (𝑆𝐾𝑉 , 𝑘) → {𝑆𝐾𝑉 , 𝑆𝐾𝑉 , . . . , 𝑆𝐾𝑉 }.

3.5

The inability to treat KV cache like arbitrary data is a major problem in stateful abstraction. Only model-specific and sequence-specific requirements make it valid. As a result, three invariants must be maintained in every state transition. Model compatibility must first be maintained. One incompatible model, tokenizer, or attention layout cannot reuse a KV state created by another. Secondly, positional compatibility needs to be maintained. Fork and merge operations must maintain position semantics since attention state is linked to word order and positional encoding. Third, lineage consistency needs to be maintained. In order for the scheduler to assess if future reuse is safe, a branch formed from a prefix state must document its pedigree. Equation 10 contains metadata terms that encode these requirements. The abstraction does not try unsafe reuse if any invariant fails. Rather,

(16)

The abstraction procedure that makes branching effective is called a fork. It generates several logical offspring of a common prefix state. When branches diverge, copy-on-write behavior is applied, even if these descendants may initially share the same physical blocks. Self-consistency sampling, Tree-of-Thought reasoning, multi-agent argument, and parallel plan exploration all directly benefit from this operation. The fourth operator is restricted merge: 𝑂𝑝𝑘𝑣_𝑚𝑒𝑟𝑔𝑒 (𝑆 1, 𝑆 2, 𝜇) → 𝑆 ∗ .

Correctness and Validity Constraints

(17) 4

4.3

the runtime resorts to text-based recomputation. The abstraction is useful for real systems because of this cautious design, which is crucial for accuracy. AAFLOW is extended from dataflow to stateflow by the stateful abstraction. The abstraction treats the KV cache as a structured state object with explicit operators, compatibility criteria, placement metadata, and cost models rather than as an opaque local artifact inside the serving engine. This makes it possible for the runtime and compiler to decide when to materialize, transfer, fork, merge, or evict model state. The outcome is a workflow architecture that allows for the effective reuse of costly LLM prefill computation while maintaining the analyzability of operator-driven execution.

4

Figure 4: Lifecycle of KV state from creation to reuse and eviction. The lifecycle of KV state in our system follows a structured sequence, as shown in Figure 4. State is first materialized during the prefill phase of LLM execution, then can be forked to enable parallel reasoning (e.g., in Tree-of-Thought workflows). Forked states may be transferred between nodes for distributed processing. When computation resumes, agents use the transferred state directly, avoiding redundant prefill. If necessary, state may be evicted due to memory constraints or policy. This lifecycle promotes state reuse, reduces redundant computation, and maintains execution flexibility.

DESIGN

We design a distributed runtime that extends AAFLOW’s operatordriven execution to a stateful execution system capable of managing KV cache across multi-agent workflows. The key principle is to treat model execution state as a first-class distributed object, rather than a temporary artifact limited to one node. The runtime integrates a stateful compiler, a distributed KV-state layer, and a state-aware scheduler to coordinate data and state dependencies, as well as transfer/recompute decisions.

4.1

4.4

Memory Management and Eviction

Managing KV state for multiple agents strains GPU memory. To mitigate this, a cost-aware eviction strategy balances reuse with memory use. Each state receives a score based on reuse and size. States with lower scores are evicted first under memory pressure, similar to distributed cache strategies but tailored for KV state reuse. To mitigate GPU memory exhaustion, the runtime implements a heuristic eviction strategy prioritizing KV states based on a combination of memory footprint size and historical reuse frequency. Formalizing and profiling optimal distributed KV eviction policies remains future work.

Stateful Workflow Compilation

The first step in our system is to extend the compilation process from a purely data-driven model to a hybrid data-state model. Given a workflow 𝑊 , AAFLOW compiles it into an execution graph 𝐺. We extend this to construct a stateful execution graph: 𝐺𝑠 = (𝑉 , 𝐸𝑑 , 𝐸𝑠 ), where 𝑉 represents operator nodes, 𝐸𝑑 captures data dependencies, and 𝐸𝑠 captures state dependencies corresponding to KV cache flow. Unlike traditional DAGs that only encode data movement, 𝐺𝑠 exposes KV-state propagation. State edges allow the scheduler to reason about locality, reuse, and transfer before recomputing shared context. This enables optimizations such as avoiding recomputation when compatible state already exists elsewhere in the system.

4.2

KV State Lifecycle

4.5

Stateful Workflow Composition

A stateful workflow composes data operators and state operators into a single graph. A typical branching workflow can be written as

KV State Representation

To support distributed state management, we define a structured representation of KV cache. Conceptually, KV cache consists of a sequence of blocks: 𝐾𝑉 = {𝐵 1, 𝐵 2, . . . , 𝐵𝑛 }

𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 → 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 → (1) (𝑘 ) {𝑂𝑝𝑟𝑒𝑎𝑠𝑜𝑛 , . . . , 𝑂𝑝𝑟𝑒𝑎𝑠𝑜𝑛 } → 𝑂𝑝𝑘𝑣_𝑚𝑒𝑟𝑔𝑒 .

(26)

This expression models a workflow where shared context is processed once, branched into multiple reasoning paths, and later merged under specific policies. As illustrated in Figure 5, dashed edges indicate KV state flow and solid edges denote standard operator outputs. Explicitly distinguishing state from data dependencies enables the compiler to schedule reasoning branches efficiently, avoiding redundant prefill computation for each branch. Currently, restricted merge supports strictly non-overlapping sequential concatenation of independent branch outputs, avoiding arbitrary tensor-blending of divergent attention states, which would violate positional encoding semantics.

(25)

where each block corresponds to a segment of the input sequence and contains key-value tensors 𝐵𝑖 = (𝐾𝑖 , 𝑉𝑖 , 𝑙𝑖 , 𝑝𝑖 ). Here, 𝐾𝑖 and 𝑉𝑖 denote the key and value tensors, 𝑙𝑖 identifies the layer index, and 𝑝𝑖 encodes the positional range associated with the block. We use Apache Arrow [3] to represent metadata and block layouts in a columnar, zero-copy format. This allows different components of the system to share state descriptors without serialization overhead. The underlying tensor buffers are transferred using highperformance communication frameworks such as UCX [35, 39] and MPI [6], ensuring efficient movement across nodes. The transport subsystem overlaps communication and computation, allowing operators to begin execution as KV blocks arrive, improving throughput.

4.6

State-Aware Scheduling

The scheduler plays a central role in determining how state is managed during execution. For each operator, the scheduler evaluates 5

Figure 6: System architecture illustrating the interaction between compiler, runtime, KV state layer, and transport.

Figure 5: Stateful operator abstraction composes conventional reasoning operators with KV-state operators. Dashed edges represent state dependencies.

5.2

The compiler extends the workflow of AAFLOW compilation process by incorporating state dependencies into the execution graph. Given a workflow 𝑊 , the compiler constructs a stateful graph 𝐺𝑠 = (𝑉 , 𝐸𝑑 , 𝐸𝑠 ), where 𝐸𝑑 represents data dependencies and 𝐸𝑠 represents KV-state dependencies. Each operator is instantiated as:

whether to transfer existing state or recompute it from scratch. This decision is guided by a cost model: ( 𝜋 (𝑆) =

transfer, recompute,

if 𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 < 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 otherwise

(27)

𝑣𝑖 = (𝑂𝑝𝑖𝑠 , R𝑖 , L𝑖 )

(28)

where R𝑖 captures resource requirements and L𝑖 encodes locality constraints derived from KV-state placement. During compilation, the system identifies shared context across agents and inserts explicit state operators such as 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 and 𝑂𝑝𝑘𝑣_𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 . This transformation enables the reuse of model execution state across agents rather than recomputation from text. The compilation process preserves the determinism guarantees of AAFLOW while extending its execution model to include stateflow alongside dataflow.

Here, 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 represents the cost of recomputing the state from input text. When transfer is cheaper, the system reuses existing state; otherwise, it falls back to recomputation. More generally, the Í scheduler optimizes the objective: min 𝑖 𝑇𝑖 + 𝜆 · 𝑀𝑒𝑚𝑖 where 𝑇𝑖 is the execution time of operator 𝑖, 𝑀𝑒𝑚𝑖 is its memory footprint, and 𝜆 controls the trade-off between latency and memory usage. This formulation allows the system to adapt to different workloads and resource constraints. For example, in memory-constrained environments, the scheduler may choose to recompute state rather than store and transfer it.

5.3 4.7

Build Compiler and Execution Graph

Runtime Integration

KV-State Manager

The KV-state manager is responsible for representing and maintaining KV cache across distributed resources. Inspired by block-based KV management in vLLM [19], the system partitions KV cache into fixed-size blocks to enable efficient reuse and transfer. Each KV state is represented as a collection of blocks with associated metadata, as defined in Section 3. Internally, the system maintains a distributed mapping:

The runtime extends the capabilities of existing LLM serving systems like vLLM [18] and SGLang [48], which already manage KV caches efficiently on single nodes, to distributed workflows. Execution involves four key stages: generating the stateful execution graph 𝐺𝑠 , scheduling operators and state placement, executing operators with KV reuse when possible, and dynamically updating the state graph as states are created or consumed. This approach maintains compatibility with current model-serving infrastructure while supporting advanced distributed optimization.

M : (𝑠𝑡𝑎𝑡𝑒_𝑖𝑑, 𝑏𝑙𝑜𝑐𝑘_𝑖𝑑) → (𝑑𝑒𝑣𝑖𝑐𝑒, 𝑎𝑑𝑑𝑟𝑒𝑠𝑠)

(29)

This mapping allows the runtime to locate KV blocks without scanning global state. The manager also tracks lineage information, ensuring that forked states maintain consistent ancestry relationships. Metadata is encoded using Apache Arrow [3], which provides a columnar, zero-copy representation that can be shared across components without serialization overhead. This design aligns with prior work demonstrating that zero-copy data exchange is critical for high-performance distributed pipelines.

5 IMPLEMENTATION 5.1 Overview We implement the proposed stateful agentic abstraction as a distributed runtime that extends AAFLOW with explicit support for KV-state orchestration. The implementation builds on the operatordriven execution model introduced in AAFLOW, where workflows are compiled into communication-aware execution graphs over distributed resources. In contrast to existing agent frameworks, which treat model execution as a black box, our system externalizes KV cache as a manipulable distributed state object. The system consists of four tightly integrated layers: a state-aware compiler, a KV-state manager, a transport subsystem, and an execution runtime (Figure 6). These components collectively enable the system to materialize, transfer, reuse, and evict KV state while preserving compatibility with existing LLM serving infrastructures such as vLLM [19] and SGLang [47].

5.4

Transport Subsystem

The transport subsystem enables efficient movement of KV state across nodes and devices. It leverages high-performance communication frameworks such as UCX [35] and MPI [6] to perform zero-copy transfers of tensor buffers. Unlike traditional distributed systems that serialize objects into intermediate formats, our system transfers raw KV blocks directly between memory regions. Metadata is transmitted separately using Arrow descriptors, while large tensor buffers are transferred using RDMA when available. The 6

transfer cost is modeled as:

rather than attempting to mathematically blend divergent parallel attention matrices, which would corrupt RoPE positional encodings.

|𝐾𝑉 | +𝛿 (30) 𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 = 𝐵𝑊 To reduce communication overhead, the system operates at block granularity and transfers only the subset of KV state required by downstream operators. This approach is particularly effective for prefix-based reuse, where only early segments of the sequence are needed. Additionally, the transport subsystem supports overlapping communication with computation. As KV blocks arrive, operators can begin partial execution, improving overall pipeline throughput.

5.5

6

To isolate the performance of distributed KV-transfer from the variance of application-level framework overhead, we evaluate our abstraction using deterministic synthetic prompts and natural questions [17] datasets and a trace-driven analytical model parameterized by rigorous empirical microbenchmarks (TTFT, multi-agent computing cost, transfer/recompute tradeoffs, memory footprint, throughput, and framework overhead). AAFLOW+ is also compatible with vLLM, and SGLang backends

Execution Runtime

The execution runtime orchestrates the execution of the compiled graph across distributed resources. It schedules operators based on both data availability and KV-state availability, ensuring that tasks are executed only when required inputs and state are ready. Each operator invocation interacts with the underlying LLM serving system through KV-aware APIs. Specifically, when executing a reasoning operator, the runtime injects precomputed KV state into the model, bypassing the prefill stage and directly initiating decoding. This mechanism builds upon the KV reuse capabilities of systems such as vLLM and SGLang, extending them from singlenode execution to distributed workflows [19, 47].

5.6

6.1

Fault Tolerance and Consistency

Branching Optimization

Branching workloads are particularly well-suited for stateful execution. Consider a workflow that generates 𝑘 reasoning branches from a shared prefix. In text-based systems, each branch must independently process the full context: 𝑘 𝑇𝑡𝑒𝑥𝑡 = 𝑘 · 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿)

Baselines

We compare against representative systems from three categories: LLM serving, KV-cache optimization, and multi-agent orchestration. (1) vLLM (PagedAttention) [19]: vLLM is a state-of-the-art LLM serving system that optimizes KV cache using block-based memory management and PagedAttention. It represents the strongest baseline for single-node KV reuse and is widely used in production. (2) SGLang (RadixAttention) [47]: SGLang improves KV reuse by identifying shared prefixes across structured language programs and exploiting radix-tree-style attention reuse. It provides a natural comparison for prefix-sharing optimizations. (3) DistServe [50]: DistServe is a distributed LLM serving system that decouples prefill and decode stages across nodes. While it improves resource utilization, it does not expose KV cache as a reusable distributed object across agents. (4) AAFLOW-text [34]: AAFLOW-text focuses on memoryefficient inference through offloading from RAG and scheduling techniques. It highlights that prefill from vectorstore through RAG could improve, but does not optimize inter-agent KV reuse. (5) Text-based Dense Prefil Orchestration (Sarathi-style) [1]: This baseline represents current agent frameworks where agents exchange text messages without KV reuse. Each agent performs independent prefill. (6) Communication-oriented KV-cache reuse (KVCOMM) [43]: KVCOMM shares reusable cache context across related agent interactions instead of fully replaying text prompts. Unlike AAFLOW+, it models KV communication without exposing a full workflowlevel abstraction for state materialization, fork, transfer, restricted composition, and eviction.

Maintaining correctness in the presence of distributed state requires careful validation. Each KV state carries metadata that ensures compatibility across operators. In particular, lineage tracking ensures that state is only reused in contexts where it remains semantically valid. When failures occur, the system can recover by recomputing state from the nearest valid prefix. This fallback mechanism ensures robustness without requiring full checkpointing of intermediate states. The proposed system generalizes agentic execution from a stateless, text-based model to a stateful, distributed model. By explicitly representing, transferring, and reusing KV cache, the system eliminates redundant prefill computation and enables scalable multiagent workflows. The combination of state-aware compilation, structured state representation, and cost-driven scheduling forms the foundation for efficient stateful execution.

5.7

EVALUATION

6.2

Metrics

We evaluate systems using metrics that capture both model-level performance and system-level efficiency. (31)

In contrast, our system computes the prefix once and then forks the resulting KV state:

Time-to-First-Token (TTFT):. TTFT measures the latency between request arrival and the generation of the first output token:

𝑘 𝑇𝑠𝑡𝑎𝑡𝑒 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 (𝐿) + 𝑘 · 𝑇𝑑𝑒𝑐𝑜𝑑𝑒

𝑇𝑇 𝐹𝑇 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 + 𝑇𝑞𝑢𝑒𝑢𝑒 + Ω

(32)

Since decoding is significantly cheaper than prefill, this leads to substantial performance gains. Figure 5 illustrates this optimization. Restricted merge is used to append non-overlapping segments,

(33)

Since prefill dominates TTFT for long contexts, reducing redundant prefill is critical. 7

Aggregate Compute Cost: The aggregate compute cost is defined

Cost and network parameters: For stateful runtime experiments, the configured state-transfer model uses measured KV bytes. Experiment 1 used bandwidth_bytes_per_sec = 25,000,000,000, equivalent to 25 GB/s or 200 Gbps, with network_latency_sec = 50us, resume_overhead_sec = 0.1ms, and very small framework overheads (omega_state_sec = omega_text_sec = 50us). Experiment 3 sweeps network bandwidth from 10 Gbps to 400 Gbps and uses both RDMA-like 10us and Ethernet-like 100us latency settings.

as: 𝑇𝑡𝑜𝑡𝑎𝑙 = 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 + 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 + Ω

(34)

Framework Overhead (Ω): Following AAFLOW, we isolate system overhead as: Ω = 𝑇𝑡𝑜𝑡𝑎𝑙 − (𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 + 𝑇𝑑𝑒𝑐𝑜𝑑𝑒 )

(35)

This includes scheduling, communication, serialization, and synchronization costs. Throughput: Throughput is measured as tokens per second: Total tokens generated (36) 𝑇ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡 = 𝑇𝑡𝑜𝑡𝑎𝑙 Memory Footprint: We measure peak KV memory usage: ∑︁ |𝐾𝑉𝑖 (𝑡)| 𝑀𝑒𝑚𝐾𝑉 = max

6.4

Workloads and Topologies

The benchmarked scenario is a shared-prefix agentic workflow, not independent stateless chat. The main topology is a parallel broadcast / Tree-of-Thought style DAG:

(37)

𝑡

root context → KV materialize → KV fork

𝑖

KV Reuse Ratio: To quantify reuse effectiveness, we define: Tokens served from KV cache 𝑅𝑒𝑢𝑠𝑒 = (38) Total tokens processed

→ {agent1, . . . , agent𝑘 } → merge

A single root context is processed once, and 𝑘 downstream agents evaluate different reasoning paths using the same prefix. The multiagent scaling experiment sweeps 𝑘 ∈ {1, 2, 4, 8, 16}. The main realmodel runs use 𝑌 = 64 generated tokens per branch. This output length is important: AAFLOW+ primarily reduces prefill and TTFT. If 𝑌 were thousands of tokens, decode would dominate and the relative TTFT advantage would be smaller. The prompts are deterministic synthetic long-context prompts. They are used to control context length and branch count precisely. These experiments therefore measure systems behavior—state reuse, transfer, memory, and overhead—rather than dataset answer quality. We evaluate three representative workloads: Multi-Agent Debate: multiple agents iteratively refine responses using shared context. Tree-of-Thought Reasoning: branching reasoning paths with shared prefixes. Retrieval-Augmented Generation (RAG): agents share retrieved context and perform downstream reasoning. It is not expected to help independent short stateless prompts, because such prompts have no shared state to fork or transfer. Each workload varies in context length (1K–32K tokens), number of agents (2–16), and branching factor (2–16).

Transfer Efficiency: We evaluate the ratio between transfer cost and avoided recomputation: 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙 − 𝑇𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟 𝐸 𝑓 𝑓 𝑖𝑐𝑖𝑒𝑛𝑐𝑦 = (39) 𝑇𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙

6.3

Experimental Setup

All experiments are conducted on a distributed GPU cluster. Hardware: We use a cluster of 4–16 nodes, each equipped with NVIDIA A100 (80GB and 40GB), 32–64 CPU cores, and RDMAenabled InfiniBand interconnects. Models and backend: The HF backend uses AutoTokenizer and AutoModelForCausalLM with deterministic decoding. KV metadata is extracted from past_key_values; tensors are not serialized by default. The measured real-model values used by the benchmark matrix include prefill time, decode time, TTFT approximation, generated-token count, KV shape metadata, and KV byte size. Table 1: HF, vLLM, SGLang backend use below model coverage in the completed run. Name

HF model id HF model id

Max context

Output tokens

Mistral Llama3

Mistral-7B-Instruct-v0.3 Meta-Llama-3-8B-Instruct

32,640 8,128

64 64

(40)

6.5

Meaning of total latency

The total_latency_sec field in these tables is a modeled aggregate workload cost, not the observed Slurm wall-clock runtime of the experiment script. For text baselines, the benchmark multiplies measured per-prompt prefill and decode costs by the number of branch instances and prompts. For example, Experiment 1 uses 16 agents, branch factor 8, and 32 prompts, so dense text passing is charged for 16 × 8 × 32 = 4096 full-context prefill/decode executions. AAFLOW+ is charged for one shared materialized state plus reused state transfers and continuations. Therefore large total_latency_sec values should be interpreted as aggregate work avoided by state reuse, not as the elapsed time of the Slurm job.

Mistral was tested up to 32,640 context tokens so that context_tokens + output_tokens stayed within the configured 32,768 token window. Llama3 was tested up to 8,128 context tokens with 64 output tokens, matching the configured 8,192-token limit (Table 1). Our runtime integrates with the backend’s KV-cache APIs while extending them to support distributed state transfer. Communication is implemented using UCX and NCCL. Mistral-7B, vLLM, and SGLang are licensed under Apache 2.0. Llama-3-8B is licensed under the Meta Llama 3 Community License.

6.6

Experiment 1: TTFT Reduction

The first experiment measures time to first token (TTFT) as a function of context length. Text-passing systems repeatedly rebuild the 8

TTFT vs Context Length by Baseline

4.0 3.5 3.0 2.5 TTFT (s)

The TTFT curves validate the expected asymptotic behavior. Dense and text baselines scale with context length because every agent or branch repeats prefix prefill. AAFLOW+ grows much more slowly because the marginal branch cost is the KV transfer/resume path. Mistral shows the strongest visible spread because the run reaches 32K context; DistServe-style is the nearest TTFT competitor because it also separates prefill from decode, but it does not expose the same explicit state abstraction and therefore remains slower in these state-transfer workloads.

AAFLOW+ aaflow_text dense_prefill distserve_style kvcomm_prefix sglang_prefix vllm_local_prefix

2.0 1.5 1.0 0.5

Speedup vs Number of Agents

0.0

12 5000

10000

15000 20000 Context Length (tokens)

25000

30000

Speedup (baseline / AAFLOW+)

0

Figure 7: Experiment-1: TTFT vs context length across multiple baselines. The curves confirm that text baselines scale with context length, while AAFLOW+ grows slowly because branch cost follows the transfer/resume path rather than repeated prefill

10

8 AAFLOW+ aaflow_text dense_prefill distserve_style kvcomm_prefix sglang_prefix vllm_local_prefix

6

4

2

Table 2: Experiment-1: Mean TTFT reduction and Throughput with variation context grid and fixed 16 agents. Model

Baseline

Mean TTFT (s)

Relative to AAFLOW+

Mistral Mistral Mistral Mistral Mistral Mistral Mistral

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

0.041 2.017 2.057 0.437 0.280 0.704 0.123

1.00× 49.2× slower 50.2× slower 10.7× slower 6.8× slower 17.2× slower 3.0× slower

Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

0.030 0.499 0.509 0.124 0.086 0.187 0.052

1.00× 16.6× slower 17.0× slower 4.1× slower 2.9× slower 6.2× slower 1.7× slower

2

4

6

8 10 Number of Agents

12

14

16

Figure 8: Experiment-2: Scaling impact of operator abstraction for multi-agent jobs with HF+Mistral. Total latency and speedup versus the nearest competitor with HF backend. The scaling result is the clearest evidence that the KV state abstraction matters for multi-agent workloads.

Table 3: Experiment-2: Aggregate Compute Cost and Efficiency Gain (EG) vs nearest competitor in HF backend

same prefix and pay a TTFT cost that scales with context length: 𝑇𝑇 𝐹𝑇text ∝ 𝐿. AAFLOW+ pays the prefill cost once and then resumes from the distributed state: 𝑇𝑇 𝐹𝑇state ≈ 𝑇transfer + 𝑇resume . Table 2(HF+Mistral, Llama3) and Figure 7(HF+Mistral) reports the mean TTFT across all tested context lengths for each model. AAFLOW+ has the lowest mean TTFT for both models. On Mistral, dense text prefill, AAFLOW-text and vLLM-local, SGLang, share almost same cold-start TTFT because each must first materialize the prompt prefix locally, while AAFLOW+ has 0.041s, a 49.2× reduction. Local-prefix baselines reduce total repeated work after cache population, which is reflected in aggregate compute cost rather than cold TTFT. The nearest competitor by mean TTFT is DistServe-style at 0.123s, still 2.99× slower than AAFLOW+. The largest contexts show the widening gap most clearly.

Model

Agents AAFLOW+ (s)

Mistral Mistral Mistral Mistral Mistral

1 2 4 8 16

30.700 169.757 30.700 339.514 58.371 679.027 113.714 1358.055 224.399 2716.109

119.135 224.850 436.281 859.142 1704.865

3.88× 7.32× 7.47× 7.56× 7.60×

Llama3 Llama3 Llama3 Llama3 Llama3

1 2 4 8 16

29.067 143.959 29.067 287.918 56.351 575.836 110.918 1151.671 220.052 2303.342

112.396 216.891 425.880 843.858 1679.815

3.87× 7.46× 7.56× 7.61× 7.63×

6.7

Dense SGLang (s) EG to (s) Competitor SGLang

Experiment 2: Multi-Agent Scaling

To evaluate multi-agent scaling, we extrapolate aggregate multiagent compute time using an analytical cost model parameterized by empirical microbenchmarks gathered on our GPU cluster. In textpassing systems, each agent pays for repeated context processing, so cost increases approximately linearly with the agent count. In 9

Table 4: Experiment-4: Mean peak KV memory and memory ratio with HF backend

high-branch-factor workflows (e.g., 16 parallel branches), a single node can’t run all decodes without OOM errors, so the workload must be distributed across nodes, requiring KV network transfer. AAFLOW+ materializes the shared context once and then fork the state, which should improve the speedup as 𝑘 grows: 𝑘 𝑇state ≈ 𝑇prefill + 𝑘 · 𝑇decode,

Table 3 and Figure 8 show aggregate compute cost by agent count. The scaling table shows two effects. First, AAFLOW+ uses the available stateful parallel width for the first two agents, so 1agent and 2-agent costs are equal for both models. Second, after the parallel width is filled, costs grow by waves rather than by full repeated prefill. Mistral grows from 30.700s at 1–2 agents to 224.399s at 16 agents. Llama3 grows from 29.067s at 1–2 agents to 220.052s at 16 agents. The nearest non-AAFLOW+ competitor at every agent count is SGLang prefix, but it remains 3.87×–7.63× slower because local prefix reuse does not remove workflow-level branch duplication. The dense baseline is much worse: at 16 agents, the dense total aggregated compute cost is 2716.109 for Mistral. AAFLOW+ converts the agent dimension from repeated prefill into repeated decode continuation from an already materialized state. Even a strong local-prefix baseline cannot fully remove duplicated branch work once many agents need the same prefix.

AAFLOW+ transfer @ 100Gbps AAFLOW+ transfer @ 10Gbps AAFLOW+ transfer @ 200Gbps AAFLOW+ transfer @ 25Gbps AAFLOW+ transfer @ 400Gbps Dense/text recompute Local-prefix reuse (vLLM/SGLang/KVCOMM)

3.0

Time (s)

2.5

Baseline

Peak KV (GiB)

Memory Ratio

Mistral Mistral Mistral Mistral Mistral Mistral Mistral

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

8.355 49.987 53.986 14.351 14.591 15.702 50.995

1.00× 5.98× larger 6.46× larger 1.72× larger 1.74× larger 1.88× larger 6.10× larger

Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

4.210 25.188 27.202 7.231 7.311 7.912 25.695

1.00× 5.98× larger 6.46× larger 1.72× larger 1.73× larger 1.88× larger 6.10× larger

At 25 Gbps and above, transfer wins at every tested context for both models, measured HF prefill is already more expensive than transferring the measured KV bytes across every tested context. AAFLOW+, therefore, needs both the stateful operators and the cost model: KVTransfer is not always correct, but it is usually correct on high-bandwidth GPU clusters and can be selected explicitly by the scheduler.

Transfer/Recompute Crossover by Baseline Policy 3.5

Model

2.0

KV Memory Footprint vs Branch Factor

1.5 1.0

120

Peak KV Memory (GiB)

0.5 0.0 0

5000

10000

15000 20000 Context Length (tokens)

25000

30000

Figure 9: Experiment-3: RDMA-like transfer benefit summary with HF backend and Mistral model. This experiment gives a concrete scheduling rule for stateful execution. The maximum transfer-vs-recompute speedup grows with bandwidth.

6.8

100 80

AAFLOW+ aaflow_text dense_prefill distserve_style kvcomm_prefix sglang_prefix vllm_local_prefix

60 40 20 0 2

Experiment 3: Transfer vs. Recomputation

4

6

8 10 Branch Factor

12

14

16

Figure 10: Experiment-4: Mean peak KV memory with HF backend and Mistral Model. AAFLOW+ reduces memory because it treats KV as a first-class object with lineage and ownership.

The third experiment compares KV transfer cost to recomputing the prompt prefill where 𝑇recompute = 𝑇prefill : KV bytes + latency, bandwidth The scheduler prefer state transfer when 𝑇transfer < 𝑇recompute . Figure 9 summarizes the RDMA-like latency sweep. On slow 10 Gbps links, the KV object can be large enough that recomputation may be preferable for some contexts (5 out of 8). This is expected: low bandwidth makes the serialized KV payload expensive. 𝑇transfer =

6.9

Experiment 4: Memory Efficiency

The fourth experiment compares peak KV memory footprint across baselines. The stateful design should avoid allocating redundant prefix KV for every branch. AAFLOW+ stores one materialized prefix 10

Table 5: Experiment-5: Mean throughput and framework overhead with fixed context grid and variation agents.

plus branch continuation state; dense and text baselines effectively duplicate the prefix across branches. Table 4 and Figure 10 reports mean peak KV memory across branch factors 1, 2, 4, 8, and 16. For Mistral, AAFLOW+ averages 8.355 GiB peak KV memory. Dense, AAFLOW-text, and DistServe-style average 49.987 GiB, or 5.98× the AAFLOW+ memory. vLLM-local and SGLang average 14.351 GiB (1.72× AAFLOW+), and 14.591 GiB (1.74× AAFLOW+), while KVCOMM averages 15.702 GiB (1.88×). The nearest competitor varies with the branch factor. At branch factors 1, 2, and 4, KVCOMM is closest to AAFLOW+. At branch factors 8 and 16, vLLM-local becomes closest. Even then, AAFLOW+ uses less memory at every branch factor. Forked states initially share block references, so memory grows with branch continuation rather than with full-context duplication. The vLLM-local and SGLang profiles also reduce duplicate prefix storage, but they are local-prefix baselines rather than distributed state-object schedulers. SKVCOMM captures communication-oriented prefix reuse but has a lower reuse fraction in this benchmark. The result is that AAFLOW+ has the lowest peak memory in every tested branch-factor setting.

6.10

Experiment 5: Throughput and Framework Overhead

The fifth experiment measures effective throughput and framework overhead Ω shown in Table 5. AAFLOW+ should improve throughput by removing repeated prefill work and reducing text serialization. In the benchmark schema, throughput is reported as generated tokens per second, and Ω captures framework-side scheduling, serialization, and orchestration overhead modeled by the baseline adapters. AAFLOW+ achieves approximately 302 tokens/s on Mistral. The nearest throughput competitor is vLLMlocal/SGLang/DistServe-style at about 38–39 tokens/s. The resulting throughput advantage is about 7.63×–8.04× against those nearest competitors and roughly 9.47× against dense text prefill, although dense text prefill has low overhead. The throughput numbers are consistent with the aggregated compute costs experiments. AAFLOW+ executes one measured prefill and then reuses state; text baselines repeatedly pay the full context construction and prefill cost. The explicit abstraction also keeps framework overhead small in this simulation profile. KVCOMM has a meaningful modeled communication overhead in these runs, which explains the larger Ω values and lower throughput relative to AAFLOW+.

7

Throughput (tok/s)

Ω (s)

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

302.61 31.94 31.84 38.81 39.62 35.59 37.62

0.0075 0.0019 11.1659 0.0627 0.0657 21.6770 0.0597

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

300.57 33.61 33.54 38.46 39.27 36.16 37.46

0.0075 0.0019 6.7272 0.0627 0.0657 13.0375 0.0597

Model

Baseline

Mistral Mistral Mistral Mistral Mistral Mistral Mistral Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

workflows by treating embedding, retrieval, reasoning, and memory operations as composable operators. Our work further extends this abstraction by introducing stateful operators over KV cache, effectively generalizing dataflow systems into stateflow systems. Workflow Runtimes and Distributed Execution: A large body of work has explored distributed workflow execution and task scheduling. Systems such as Ray [24], Dask [30], and Parsl [4] provide flexible task-based execution frameworks for parallel and distributed workloads. HPC-oriented systems such as RADICALPilot [23, 31–33] and Pegasus [9] focus on large-scale scientific workflows. While these systems provide powerful execution substrates, they treat tasks as independent units and do not explicitly model LLM-specific state such as KV cache. As a result, they cannot exploit opportunities for state reuse across agents. Our work complements these systems by introducing an abstraction that exposes state dependencies and enables scheduling decisions based on both data and model state. LLM Programming and Agent Frameworks: Recent work on LLM programming has focused on improving the expressiveness and composability of language model applications. Frameworks such as LangChain [7], LangGraph [10], and AutoGen [41] enable multi-agent workflows, tool invocation, and iterative reasoning. DSPy [16] further proposes a declarative programming model for optimizing LLM pipelines. These systems treat orchestration as a high-level programming problem but largely ignore the underlying systems cost of data movement and state recomputation. In particular, they rely on text-based communication between agents, leading to repeated prefill computation. Our work differs by introducing a systems-level abstraction that allows agents to exchange execution state directly.

RELATED WORK

Distributed Data Systems and Operator Abstractions: The design of AAFLOW [34] and our extension builds upon a long line of work in distributed data systems, where computation is expressed as compositions of operators over structured data. Systems such as MapReduce [8], Spark [46], and Flink [5] demonstrate how declarative data transformations can be compiled into efficient distributed execution plans. More recent frameworks such as Modin [27] and Cylon [40] extend this paradigm to high-performance dataframes, emphasizing communication-aware execution and parallel processing patterns [25, 26, 36]. AAFLOW adapts these ideas to agentic 11

LLM Serving and KV-Cache Optimization: A growing body of work has focused on optimizing LLM inference through efficient KV-cache management. vLLM [19] introduces PagedAttention, which uses block-based memory allocation to improve throughput and reduce fragmentation. SGLang [47] proposes RadixAttention, enabling prefix sharing across structured programs. FlexGen [37] explores memory-compute trade-offs through offloading strategies, while DistServe [50] and LMCache [21] separate prefill and decode stages to improve resource utilization. These systems demonstrate that KV cache is a critical performance factor in LLM serving. However, their optimizations are confined to single-node or singlerequest execution contexts with efficient storage, movement, and management of KV cache across heterogeneous memory tiers and serving backends. They do not expose KV cache as a distributed systems object that can be shared across multiple agents. Our work extends KV-cache optimization from local serving to distributed multi-agent workflows rather than focusing solely on cache management. Distributed and Multi-Tenant LLM Serving: Recent work has explored scaling LLM serving across multiple tenants and distributed environments. Systems such as Orca [45] and Sarathi [1] improve GPU utilization through scheduling and batching techniques. Helix [14] focuses on multi-tenant inference with efficient resource sharing. While these systems address throughput and fairness, they do not explicitly consider KV-state reuse across independent requests or agents. Our approach is orthogonal, focusing on reducing redundant computation by reusing state rather than improving scheduling alone. Retrieval-Augmented Generation and Memory Systems: Retrieval-Augmented Generation (RAG) systems integrate external knowledge with LLM inference to improve factual accuracy [20]. Recent work has extended this paradigm with more sophisticated memory and retrieval mechanisms, such as MemoRAG [28], HippoRAG [13], and CueRAG [11]. These systems demonstrate the importance of persistent memory for multi-step reasoning. However, most RAG systems treat memory as external data rather than internal model state. They optimize retrieval and indexing but do not address the cost of reprocessing retrieved context within the model. Our work complements these approaches by enabling the reuse of the internal KV representation of retrieved context. Existing research has focused on two areas: improving LLM efficiency via KV-cache optimization and enhancing workflow expressiveness with agent frameworks. We unify these by treating the KV cache as a distributed state within an operator abstraction, enabling optimizations like cross-agent state reuse, distributed KV transfer, and state-aware scheduling. Extending AAFLOW’s operator-driven model with stateflow lays the groundwork for scalable, efficient, and reproducible multi-agent LLM systems.

8

inference. Infrastructure assessments are based on controlled experiments rather than production cluster data. We define architectural objectives for eviction and memory-constrained scheduling, but currently evaluate only unconstrained memory. AAFLOW+ opens several directions for future research. First, extending the workflow to support heterogeneous model environments remains an important challenge. Current compatibility constraints assume identical model architectures and positional encodings, but real-world systems may involve mixtures of models, adapters, or fine-tuned variants. Generalizing state reuse across heterogeneous settings would broaden applicability. Second, costaware scheduling can be further developed. While this work demonstrates the tradeoff between KV transfer and recomputation, future systems could incorporate dynamic bandwidth estimation, workload prediction, and adaptive placement strategies to optimize execution decisions in real time. Third, integrating stateful execution with emerging model architectures is a promising direction. While this work focuses on transformer-based KV cache, alternative architectures such as state space models introduce different notions of internal state. Extending the proposed abstraction to support these models could enable a unified treatment of execution state beyond KV representations. Fourth, fault tolerance and persistence for stateful workflows require deeper investigation. Although recomputation provides a fallback mechanism, efficient checkpointing, partial state recovery, and durable storage of KV state remain open problems for large-scale deployments. Finally, future work can explore programming abstractions and developer tooling for state-aware workflows. Providing higher-level interfaces, debugging tools, and visualization systems for stateflow execution would make these capabilities accessible to a broader class of applications.

9

CONCLUSION

We presented AAFLOW+, a stateful extension of agentic workflow abstraction that makes KV cache a first-class distributed systems object in AI memory instead of just a local inference optimization. AAFLOW+ presents a stateflow abstraction that allows agents to directly interchange and reuse execution state, in contrast to current multi-agent LLM systems that rely on text-based communication and frequently recalculate shared context. Our method exposes state reuse at the workflow level by defining operators for KV-state materialization, transfer, fork, restricted composition, and eviction, and assembling workflows into communication-aware graphs. The runtime externalizes KV cache using explicit metadata descriptors and zero-copy communication pathways, enabling efficient state reuse across distributed agents while preserving correctness through compatibility constraints. Experimental results on Mistral7B and Llama-3-8B show that AAFLOW+ significantly improves efficiency, achieving up to 50.2× reduction in TTFT, 7.63× lower multi-agent latency at 16-agent scale, 1.72×–6.10× reduction in peak KV memory, and over 7.74× improvement in throughput. These findings show that duplicate computation and framework overhead in multi-agent LLM execution can be significantly reduced by substituting explicit KV-state sharing for text forwarding in AI memory.

LIMITATIONS AND FUTURE WORK

These results should be viewed within defined limits. The workloads use synthetic, deterministic prompts [17], so answer quality is not evaluated against external datasets. The main output length is 𝑌 = 64 tokens; longer outputs would reduce prefill reuse benefits and place greater emphasis on decoding. AAFLOW+ targets shared-prefix multi-agent workflows, not independent stateless 12

REFERENCES

generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages. https://dl.acm.org/doi/abs/10.5555/3495724.3496517 [21] Yuhan Liu, Jiayi Yao, Yihua Cheng, Yuwei An, Xiaokun Chen, Shaoting Feng, et al. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv preprint arXiv:2510.09665 (2025). https://arxiv.org/pdf/2510.09665 [22] LLMS3. 2026. When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory Tier. Project website. https://llms3. com/blog/when-ai-memory-became-an-architecture-may-2026 KV-Cache Persistence, MCP, and the Night S3 Got Its Memory Tier. [23] A. Merzky, M. Turilli, M. Titov, A. Al-Saadi, and S. Jha. 2022. Design and Performance Characterization of RADICAL-Pilot on Leadership-Class Platforms. IEEE Transactions on Parallel and amp; Distributed Systems 33, 04 (apr 2022), 818–829. https://doi.org/10.1109/TPDS.2021.3105994 [24] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. 2018. Ray: A distributed framework for emerging { AI } applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 561–577. https://www.usenix.org/system/files/osdi18-moritz.pdf [25] Niranda Perera, Arup Kumar Sarker, Kaiying Shan, Alex Fetea, Supun Kamburugamuve, Thejaka Amila Kanewala, Chathura Widanage, Mills Staylor, Tianle Zhong, Vibhatha Abeykoon, Gregor von Laszewski, and Geoffrey Fox. 2024. Supercharging distributed computing environments for high-performance data engineering. Frontiers in High Performance Computing Volume 2 - 2024 (2024). https://doi.org/10.3389/fhpcp.2024.1384619 [26] Niranda Perera, Arup Kumar Sarker, Mills Staylor, Gregor von Laszewski, Kaiying Shan, Supun Kamburugamuve, Chathura Widanage, Vibhatha Abeykoon, Thejaka Amila Kanewela, and Geoffrey Fox. 2023. In-depth analysis on parallel processing patterns for high-performance Dataframes. Future Generation Computer Systems 149 (2023), 250–264. https://doi.org/10.1016/j.future.2023.07.007 [27] Maximilian Petersohn, Stephen Macke, Doris Xin, William Ma, J. K. Wittenauer, Stephen Hoyer, Ryan Marcus, Matei Zaharia, and Benjamin Recht. 2020. Towards Scalable Dataframe Systems. Proceedings of the VLDB Endowment (PVLDB) 13, 12 (2020), 2033–2046. https://doi.org/10.14778/3407790.3407807 [28] Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. 2025. MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation. In Proceedings of the ACM on Web Conference 2025 (Sydney NSW, Australia) (WWW ’25). Association for Computing Machinery, New York, NY, USA, 2366–2377. https://doi.org/10.114 5/3696410.3714805 [29] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. ACM Trans. Storage (Nov. 2025). https://doi.org/10.1145/3773 772 Just Accepted. [30] Matthew Rocklin. 2015. Dask: Parallel computation with blocked algorithms and task scheduling. In Proceedings of the 14th python in science conference, Vol. 130. Citeseer, 136. https://proceedings.scipy.org/articles/Majora-7b98e3ed-013.pdf [31] Arup Kumar Sarker, Aymen Alsaadi, Alexander James Halpern, Prabhath Tangella, Mikhail Titov, Niranda Perera, Mills Staylor, Gregor von Laszewski, Shantenu Jha, and Geoffrey Fox. 2025. Deep RC: A Scalable Data Engineering and Deep Learning Pipeline. In Job Scheduling Strategies for Parallel Processing: 28th International Workshop, JSSPP 2025, Milan, Italy, June 3–4, 2025, Revised Selected Papers (Milan, Italy). Springer-Verlag, Berlin, Heidelberg, 205–223. https://doi.org/10.1007/978-3-032-10507-3_11 [32] Arup Kumar Sarker, Aymen Alsaadi, Niranda Perera, Mills Staylor, Gregor von Laszewski, Matteo Turilli, Ozgur Ozan Kilic, Mikhail Titov, Andre Merzky, Shantenu Jha, et al. 2024. Design and implementation of an analysis pipeline for heterogeneous data. arXiv preprint arXiv:2403.15721 (2024). [33] Arup Kumar Sarker, Aymen Alsaadi, Niranda Perera, Mills Staylor, Gregor von Laszewski, Matteo Turilli, Ozgur Ozan Kilic, Mikhail Titov, Andre Merzky, Shantenu Jha, et al. 2024. Radical-Cylon: A Heterogeneous Data Pipeline for Scientific Computing. In Job Scheduling Strategies for Parallel Processing. Springer Nature Switzerland, 84–102. https://doi.org/10.1007/978-3-031-74430-3_5 [34] Arup Kumar Sarker, Mills Staylor, Aymen Alsaadi, Gregor von Laszewski, Shantenu Jha, and Geoffrey Fox. 2026. AAFLOW: Scalable Patterns for Agentic AI Workflows. arXiv preprint arXiv:2605.02162 (2026). Under Submission to SC2026. [35] Pavel Shamis, Manjunath Gorentla Venkata, M. Graham Lopez, Matthew B. Baker, Oscar Hernandez, Yossi Itigin, Mike Dubman, Gilad Shainer, Richard L. Graham, Liran Liss, Yiftah Shahar, Sreeram Potluri, Davide Rossetti, Donald Becker, Duncan Poole, Christopher Lamb, Sameer Kumar, Craig Stunkel, George Bosilca, and Aurelien Bouteiller. 2015. UCX: An Open Source Framework for HPC Network APIs and Beyond. In 2015 IEEE 23rd Annual Symposium on HighPerformance Interconnects. 40–43. https://doi.org/10.1109/HOTI.2015.13

[1] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369 (2023). [2] Apache Arrow Project. 2025. Apache Arrow. Project website. https://arrow. apache.org/ Language-independent columnar memory format with zero-copy reads. [3] Apache Software Foundation. 2016. Apache Arrow: A cross-language development platform for in-memory data. https://arrow.apache.org [4] Yadu Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin M. Wozniak, Ian Foster, Michael Wilde, and Kyle Chard. 2019. Parsl: Pervasive Parallel Programming in Python. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing (Phoenix, AZ, USA) (HPDC ’19). Association for Computing Machinery, New York, NY, USA, 25–36. https://doi.org/10.1145/33 07681.3325400 [5] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering 38, 4 (2015). https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf [6] Lisandro Dalcin, Rodrigo Paz, and Mario Storti. 2005. MPI for Python. J. Parallel and Distrib. Comput. 65, 9 (1 Sept. 2005), 1108–1115. https://doi.org/10.1016/j.jp dc.2005.03.010 [7] Emily Davis. 2024. Building Custom AI Workflows Using LangChain Tools. ThinkTide Global Research Journal 5, 4 (2024), 54–62. https://thinktidejournal.c om/index.php/TGRJ/article/view/53/63 [8] Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters. Commun. ACM 51, 1 (Jan. 2008), 107–113. https://doi.org/10.1 145/1327452.1327492 [9] Ewa Deelman, Karan Vahi, Gideon Juve, Mats Rynge, Scott Callaghan, Philip J. Maechling, Rajiv Mayani, Weiwei Chen, Rafael Ferreira da Silva, Miron Livny, and Kent Wenger. 2015. Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46 (2015), 17–35. https: //doi.org/10.1016/j.future.2014.10.008 [10] LangChain Developer. 2024. LangGraph: Stateful Multi-Agent Workflows. Technical Report. LangChain Inc. https://blog.langchain.com/langgraph-multi-agentworkflows [11] Yuanshuang Fu, Dan Liu, Bonan Zhang, Zhuotong Jiang, Haibo Mei, and Jiajin Guan. 2025. Cue RAG: Dynamic multi-output cue memory under H framework for retrieval-augmented generation. Neurocomputing 639 (2025), 130235. https: //doi.org/10.1016/j.neucom.2025.130235 [12] Yingsheng Geng, Yuchong Gao, Weihong Wu, Guyue Liu, and Jiang Liu. 2026. RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse. arXiv preprint arXiv:2603.13289 (2026). [13] Bernal J Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems 37 (2024), 59532– 59569. [14] Bodun Hu, Jiamin Li, Le Xu, Myungjin Lee, Akshay Jajoo, Geon-Woo Kim, Hong Xu, and Aditya Akella. 2024. Blockllm: Multi-tenant finer-grained serving for large language models. arXiv preprint arXiv:2404.18322 (2024). [15] Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. arXiv preprint arXiv:2406.17565 (2024). https://arxiv.org/pdf/2406.17565 [16] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint arXiv:2310.03714 (2023). https://arxiv.org/pdf/2310.03714 [17] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453–466. [18] Woosuk Kwon et al. 2023. vLLM: Easy, Fast, and Cheap LLM Serving. https: //github.com/vllm-project/vllm [19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. https://doi.org/10.1145/3600006.3613165 [20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented

13

APPENDIX A EXTENDED SYSTEM DESIGN

[36] Kaiying Shan, Niranda Perera, Damitha Lenadora, Tianle Zhong, Arup Kumar Sarker, Supun Kamburugamuve, Thejaka Amila Kanewela, Chathura Widanage, and Geoffrey Fox. 2022. Hybrid Cloud and HPC Approach to HighPerformance Dataframes. In 2022 IEEE International Conference on Big Data (Big Data). 2728–2736. https://doi.org/10.1109/BigData55660.2022.10020958 [37] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Vol. 202. PMLR, 31094–31116. https://proceedings.mlr.press/v202/sheng23a.html [38] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 377, 19 pages. https://dl.acm.org/doi/10.5555/3666122.3666499 [39] Mills Staylor, Arup Kumar Sarker, Gregor von Laszewski, Geoffrey Fox, Yue Cheng, and Judy Fox. 2026. Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications. Frontiers in High Performance Computing (2026). [40] Chathura Widanage, Niranda Perera, Vibhatha Abeykoon, Supun Kamburugamuve, Thejaka Amila Kanewala, Hasara Maithree, Pulasthi Wickramasinghe, Ahmet Uyar, Gurhan Gunduz, and Geoffrey Fox. 2020. High performance data engineering everywhere. In 2020 IEEE International Conference on Smart Data Services (SMDS). IEEE, 122–132. https://doi.org/10.1109/SMDS49396.2020.00022 [41] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang (Eric) Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Ahmed Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In COLM 2024. https://www.mi crosof t.com/en-us/research/publication/autogen-enabling-next-gen-llmapplications-via-multi-agent-conversation-framework/ [42] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). https: //openreview.net/pdf?id=WE_vluYUL-X [43] Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, and Yiran Chen. 2025. KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems. [44] Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. ChunkAttention: Efficient SelfAttention with Prefix-Aware KV Cache and Two-Phase Partition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). 11608–11620. https://doi.org/10.18653/v1/2024.acl-long.623 [45] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and ByungGon Chun. 2022. Orca: A distributed serving system for { Transformer-Based } generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538. [46] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. 2016. Apache Spark: a unified engine for big data processing. Commun. ACM 59, 11 (Oct. 2016), 56–65. https://doi.org/10.1145/2934664 [47] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2000, 27 pages. https://dl.acm.org/doi/10.5555/3737916.3739916 [48] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2000, 27 pages. https://dl.acm.org/doi/10.5555/3737916.3739916 [49] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). https://www.usenix.o rg/conference/osdi24/presentation/zhong-yinmin [50] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. { DistServe } : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210.

This appendix expands the design and implementation details of AAFLOW+. The main paper presents the operator abstraction, state object, scheduler, and evaluation results. Here, we describe how the compiler, KV-state manager, transport layer, runtime, and backend adapters interact during execution.

A.1

Design Components

AAFLOW+ is organized around five cooperating components: the stateful compiler, the KV-state manager, the transport subsystem, the state-aware scheduler, and the execution runtime. The stateful compiler converts an agentic workflow into a graph containing both data edges and state edges. The KV-state manager tracks block-level KV metadata, ownership, lineage, and placement. The transport subsystem moves KV blocks across devices and nodes using Arrow metadata and UCX/RDMA-style communication. The scheduler decides whether to transfer, reuse, fork, evict, or recompute state. Finally, the execution runtime coordinates backend invocation through Hugging Face, vLLM, or SGLang-compatible execution paths. Given a workflow𝑊 , the compiler constructs a stateful execution graph 𝐺𝑠 = (𝑉 , 𝐸𝑑 , 𝐸𝑠 ), where 𝑉 is the set of operator instances, 𝐸𝑑 represents ordinary data dependencies, and 𝐸𝑠 represents KV-state dependencies. Each vertex is annotated as 𝑣𝑖 = (𝑂𝑝𝑖𝑠 , 𝑅𝑖 , 𝐿𝑖 , 𝐶𝑖 ), where 𝑂𝑝𝑖𝑠 is the stateful operator, 𝑅𝑖 describes resource requirements, 𝐿𝑖 captures locality constraints derived from KV-state placement, and 𝐶𝑖 stores compatibility constraints such as model identity, tokenizer configuration, attention layout, and positional encoding. The compiler inserts KV-state operators when it detects shared prompt prefixes or branchable contexts. For example, a text-centric workflow 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 → {𝐴𝑔𝑒𝑛𝑡 1, . . . , 𝐴𝑔𝑒𝑛𝑡𝑘 } is rewritten as

𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 (𝐶𝑜𝑛𝑡𝑒𝑥𝑡) → 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 (𝑆𝐾𝑉 , 𝑘) (1) (𝑘 ) → {𝐴𝑔𝑒𝑛𝑡 1 (𝑆𝐾𝑉 ), . . . , 𝐴𝑔𝑒𝑛𝑡𝑘 (𝑆𝐾𝑉 )}.

(41)

This transformation makes prefix reuse explicit and allows the runtime to schedule reasoning branches without forcing each branch to replay the same context.

A.2

KV-State Metadata Catalog

AAFLOW+ maintains a metadata catalog for every materialized KV state. The catalog separates logical state identity from physical tensor placement. This allows the scheduler to reason about state reuse without scanning device memory or reconstructing prompt text. Each KV state is represented as 𝑆𝐾𝑉 = (𝑀, Θ, 𝐵, Π, Λ, Γ), 14

where 𝑀 is the model identifier, Θ is the model and tokenizer configuration, 𝐵 is the set of KV blocks, Π stores positional metadata, Λ records lineage, and Γ records placement and ownership. Internally, the KV-state manager maintains the mapping

and only pays the cost of branch-specific continuation: 𝑘 𝑇state ≈ 𝑇prefill (𝐿) + 𝑇fork (𝑆𝐾𝑉 , 𝑘) +

𝑘 ∑︁

𝑇decode (𝑌 𝑗 ) + Ωstate .

𝑗=1

M : (𝑠𝑡𝑎𝑡𝑒_𝑖𝑑, 𝑏𝑙𝑜𝑐𝑘_𝑖𝑑) → (𝑑𝑒𝑣𝑖𝑐𝑒, 𝑛𝑜𝑑𝑒, 𝑎𝑑𝑑𝑟𝑒𝑠𝑠, 𝑟𝑎𝑛𝑔𝑒, 𝑜𝑤𝑛𝑒𝑟 ). The benefit is largest when the retrieved context is long, the number of agents is large, and the branch outputs are short relative to the shared prefix.

Table 6: Metadata maintained by the AAFLOW+ KV-state manager.

C Field

Purpose

state_id model_id

Unique logical identifier for a KV state. Model family and checkpoint used to create the state. Tokenizer and vocabulary compatibility. Token positions by each KV block. Transformer layer with the KV block. Block identifier for partial transfer and reuse. Materialize, fork, transfer, merge, or evict history. GPU, CPU, or remote memory location. Runtime worker currently responsible for the block. Number of logical states sharing the block. Timestamp or logical clock for eviction.

tokenizer_id position_range layer_id block_id lineage device owner refcount last_access

This section describes the full AAFLOW+ call flow for a sharedprefix multi-agent workflow. Workflow Submission: The user submits a workflow containing a shared context and multiple downstream agents: 𝑊 = (𝐶𝑜𝑛𝑡𝑒𝑥𝑡, {𝐴𝑔𝑒𝑛𝑡 1, . . . , 𝐴𝑔𝑒𝑛𝑡𝑘 }, 𝑃𝑜𝑙𝑖𝑐𝑦). The policy specifies whether state reuse is allowed, whether branches can share physical blocks, and whether transfer or recomputation should be preferred under memory pressure. Compilation: The compiler parses the workflow and constructs the stateful graph: 𝐺𝑠 = (𝑉 , 𝐸𝑑 , 𝐸𝑠 ). It identifies common prefixes, inserts 𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 before the first shared reasoning stage, inserts 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 before branch expansion, and annotates each branch with state compatibility constraints. State Materialization: The runtime sends the shared context to the backend model and executes the prefill stage once. The backend produces KV tensors, which are registered with the KV-state manager:

This design allows multiple forked states to share the same physical prefix blocks. When a branch diverges, the runtime applies copy-on-write behavior: shared prefix blocks remain aliased, while branch-specific continuation blocks are allocated separately.

B

END-TO-END EXECUTION

USE CASE: COLLABORATIVE RAG WITH SHARED EVIDENCE

𝐶𝑜𝑛𝑡𝑒𝑥𝑡 −−−−−→ 𝑆𝐾𝑉 .

A representative use case is collaborative retrieval-augmented generation over a long document collection. Suppose a user asks a complex scientific question requiring retrieval, risk analysis, verification, and final synthesis. A conventional multi-agent pipeline may use four agents: a retriever, a reasoning agent, a verifier, and a summarizer. All agents require the same retrieved evidence prefix, but they perform different downstream tasks. In a text-centric implementation, the retriever emits passages as text. Each downstream agent then receives the same passages and performs a separate prefill over the identical context:

The KV-state manager records block identifiers, token ranges, layer identifiers, placement, and lineage. Scheduling and Placement: The scheduler chooses placement for each branch. For a branch assigned to the same node as the materialized KV state, the scheduler prefers aliasing or local reuse. For a branch assigned to a remote node, it compares transfer cost against recomputation: ( 𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟, 𝑇transfer (𝑆) < 𝑇prefill (𝐿), 𝜋 (𝑆) = 𝑟𝑒𝑐𝑜𝑚𝑝𝑢𝑡𝑒, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒.

𝑘 ∑︁

Execution: Agents resume from the assigned KV state and generate branch-specific continuations. The runtime updates the state graph as new continuation blocks are created:

𝑘 𝑇text ≈ 𝑘 · 𝑇prefill (𝐿) +

𝑝𝑟𝑒 𝑓 𝑖𝑙𝑙

𝑇decode (𝑌 𝑗 ) + Ωtext .

𝑗=1

This duplicates both compute and KV memory. With AAFLOW+, the retrieved evidence is materialized once as KV state: 𝑆𝐾𝑉 = 𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 (𝐶𝑜𝑛𝑡𝑒𝑥𝑡, 𝑀). The runtime then forks this state:

(𝑗) 𝑆𝐾𝑉 → 𝐷𝑒𝑐𝑜𝑑𝑒 𝑗 (𝑌 𝑗 ) → 𝑂𝑢𝑡𝑝𝑢𝑡 𝑗 .

Merge and Finalization: The outputs are merged using a safe reduction policy. AAFLOW+ does not arbitrarily blend divergent attention states. Instead, it supports restricted merge patterns such as non-overlapping sequential concatenation, prefix-compatible extension, or text-level reduction through a summarizing model invocation.

(1) (𝑘 ) {𝑆𝐾𝑉 , . . . , 𝑆𝐾𝑉 } = 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 (𝑆𝐾𝑉 , 𝑘),

and assigns the forked states to the reasoning, verification, and synthesis agents. Each agent resumes from the shared context state 15

D

ALGORITHMS FOR KV OPERATORS

Algorithm 3 KV-Transfer Require: State 𝑆𝐾𝑉 , source node 𝑛𝑎 , destination node 𝑛𝑏 ′ Ensure: Destination state 𝑆𝐾𝑉 1: Estimate transfer cost 𝑇transfer = |𝐾𝑉 |/𝐵𝑊 + 𝛿. 2: Estimate recomputation cost 𝑇prefill (𝐿). 3: if 𝑇transfer ≥ 𝑇prefill (𝐿) then 4: Return RecomputeDecision. 5: end if 6: Select required KV blocks based on downstream position range. 7: Send Arrow metadata descriptor to 𝑛𝑏 . 8: Transfer KV tensor buffers using UCX/RDMA/MPI transport. 9: Register received blocks in the destination KV-state manager. 10: Update placement metadata Γ and lineage Λ. ′ . 11: Return transferred state 𝑆 𝐾𝑉

Each KV state follows a finite-state lifecycle: 𝐶𝑟𝑒𝑎𝑡𝑒𝑑 → 𝑅𝑒𝑔𝑖𝑠𝑡𝑒𝑟𝑒𝑑 → 𝐴𝑐𝑡𝑖𝑣𝑒 → 𝐹𝑜𝑟𝑘𝑒𝑑 → 𝑇𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟𝑟𝑒𝑑 → 𝑅𝑒𝑠𝑢𝑚𝑒𝑑 → 𝐸𝑣𝑖𝑐𝑡𝑒𝑑.

(42)

The lifecycle is maintained by the KV-state manager and updated after every operator invocation, shown in Table-7. Because lineage is explicit, the scheduler can distinguish original materialized prefixes, forked aliases, transferred blocks, and branch-specific continuation state.

D.1

KV Materialization

KV materialization converts a text context into reusable execution state. This operation is invoked once for a shared prefix.

D.3

Algorithm 1 KV-Materialize

KV Transfer

Require: Context tokens 𝑥, model 𝑀, tokenizer/configuration Θ Ensure: KV state 𝑆𝐾𝑉 1: Validate model and tokenizer configuration. 2: Execute backend prefill on 𝑥 using 𝑀. 3: Collect layer-wise KV tensors {(𝐾ℓ , 𝑉ℓ )}. 4: Partition tensors into KV blocks 𝐵 = {𝑏 1 , . . . , 𝑏𝑚 }. 5: Create metadata (𝑀, Θ, Π, Λ, Γ). 6: Register (𝑠𝑡𝑎𝑡𝑒_𝑖𝑑, 𝑏𝑙𝑜𝑐𝑘_𝑖𝑑) → (𝑑𝑒𝑣𝑖𝑐𝑒, 𝑛𝑜𝑑𝑒, 𝑎𝑑𝑑𝑟𝑒𝑠𝑠) in the KV-state manager. 7: Return 𝑆 𝐾𝑉 = (𝑀, Θ, 𝐵, Π, Λ, Γ).

KV transfer moves or aliases KV blocks between workers. The scheduler invokes transfer only when it is cheaper than recomputation. The operation works at block granularity, allowing the runtime to transfer only the prefix segments needed by the downstream branch.

Materialization is the only operation that pays the full prefill cost for the shared prefix. All downstream reuse is expressed as fork, transfer, alias, or resume.

Algorithm 4 KV-Resume

D.2

D.4

KV Resume

KV resume injects precomputed state into a backend invocation and skips repeated prefill.

Require: State 𝑆𝐾𝑉 , branch prompt suffix 𝑞, model backend 𝐵 Ensure: Generated output 𝑦 1: Validate compatibility of 𝑆 𝐾𝑉 with backend model and tokenizer. 2: Load or map KV blocks into the backend-visible memory domain. 3: Initialize decoding position from Π. 4: Append branch-specific suffix tokens 𝑞 if required. 5: Decode output tokens using reused KV state. 6: Register any newly generated continuation KV blocks. 7: Return generated output 𝑦.

KV Fork

KV fork creates multiple logical descendants from a shared prefix state.

Algorithm 2 KV-Fork Require: Parent state 𝑆𝐾𝑉 , number of branches 𝑘 (1) (𝑘 ) } Ensure: Forked states {𝑆𝐾𝑉 , . . . , 𝑆𝐾𝑉 1: Check that 𝑆 𝐾𝑉 is valid and not evicted. 2: for 𝑗 = 1 to 𝑘 do 3: Create child metadata with lineage Λ ( 𝑗 ) = Λ ∪ {𝑓 𝑜𝑟𝑘 ( 𝑗)}. 4: Alias parent prefix blocks by increasing block reference counts. 5: Assign branch-specific ownership and placement policy. 6: end for 7: Return 𝑘 logical child states.

Resume is the operation that converts state reuse into TTFT reduction. Instead of rebuilding the shared prefix, the backend continues from the imported or mapped KV state.

D.5

Restricted KV Merge

Restricted merge reconciles branch outputs without unsafe tensorlevel blending of divergent KV states. This conservative merge rule is necessary because KV state is position-dependent and model-layout-dependent. AAFLOW+ therefore treats merge as a controlled operator rather than as arbitrary tensor composition.

Fork does not copy the entire KV tensor by default. It creates logical children that share prefix blocks until branch-specific continuation requires new allocation. 16

Table 7: AAFLOW+ KV-state lifecycle transitions. Transition

Trigger

Runtime Action

Created → Registered Registered → Active Active → Forked Forked → Transferred Transferred → Resumed Resumed → Active Active → Evicted

Backend prefill completes Scheduler assigns owner Branch expansion Remote branch placement Agent execution begins Decode creates continuation Memory pressure

Insert metadata into catalog. Pin or map blocks to worker. Create child state descriptors. Transfer required blocks. Inject state into backend. Register new branch blocks. Free low-priority blocks.

E

Algorithm 5 Restricted-KV-Merge Require: States or outputs {𝑆 1, . . . , 𝑆𝑘 }, merge policy 𝜇 Ensure: Merged state or merged text output 1: Check model compatibility and positional compatibility. 2: if 𝜇 permits prefix-compatible concatenation then 3: Concatenate non-overlapping compatible state segments. 4: else if 𝜇 requires semantic reduction then 5: Convert branch outputs to text and call a summarizing/reduction operator. 6: else 7: Reject tensor-level merge and fall back to text-level reduction. 8: end if 9: Record merge lineage in Λ. 10: Return merged result.

EXAMPLE: PLANNER–RETRIEVER–SOLVER CALL FLOW

This section illustrates a complete AAFLOW+ execution path for a planner–retriever–solver workflow. Step 1 - Planning: The planner decomposes a user request into subquestions and determines that the downstream agents require a shared evidence context.

𝑈 𝑠𝑒𝑟𝑄𝑢𝑒𝑟𝑦 → 𝑂𝑝 𝑝𝑙𝑎𝑛 → {𝑆𝑢𝑏𝑡𝑎𝑠𝑘 1, . . . , 𝑆𝑢𝑏𝑡𝑎𝑠𝑘𝑘 }. Step 2 - Retrieval and Context Assembly: The retriever fetches documents and constructs a shared context: 𝑆𝑢𝑏𝑡𝑎𝑠𝑘𝑠 → 𝑂𝑝𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒 → 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 . Step 3 - KV Materialization : The context is prefilling once:

D.6

𝐶𝑜𝑛𝑡𝑒𝑥𝑡 → 𝑂𝑝𝑘𝑣_𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒 → 𝑆𝐾𝑉 .

KV Eviction

Step 4 - Fork Across Solvers: The state is forked across multiple solver agents:

KV eviction releases state when memory pressure exceeds the configured threshold.

(1) (𝑘 ) 𝑆𝐾𝑉 → 𝑂𝑝𝑘𝑣_𝑓 𝑜𝑟𝑘 (𝑘) → {𝑆𝐾𝑉 , . . . , 𝑆𝐾𝑉 }.

Algorithm 6 KV-Evict

Step 5 - Transfer or Local Reuse: For each solver branch, the scheduler chooses local reuse, transfer, or recomputation:

Require: KV-state catalog C, memory budget 𝐵 max Ensure: Updated catalog and freed memory 1: Compute priority score for each state:

  𝑟𝑒𝑢𝑠𝑒, 𝑆 is local and compatible,    𝜋 (𝑆) = 𝑡𝑟𝑎𝑛𝑠 𝑓 𝑒𝑟, 𝑇transfer < 𝑇prefill,   𝑟𝑒𝑐𝑜𝑚𝑝𝑢𝑡𝑒, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. 

𝑠𝑐𝑜𝑟𝑒 (𝑆) = 𝛼 · 𝑟𝑒𝑢𝑠𝑒 (𝑆) − 𝛽 · 𝑠𝑖𝑧𝑒 (𝑆) − 𝛾 · 𝑎𝑔𝑒 (𝑆). 2: while allocated memory > 𝐵 max do

Select state 𝑆 ∗ with lowest score. 4: if 𝑟𝑒 𝑓 𝑐𝑜𝑢𝑛𝑡 (𝑆 ∗ ) = 0 then 5: Free local tensor blocks and update catalog. 6: else 7: Decrement aliases or evict only unshared continuation blocks. 8: end if 9: end while 10: Return updated catalog. 3:

Step 6 - Solver Execution: Each solver resumes from the shared KV state and generates a branch-specific answer: (𝑗) 𝐴𝑔𝑒𝑛𝑡 𝑗 (𝑆𝐾𝑉 ) → 𝐴𝑛𝑠𝑤𝑒𝑟 𝑗 .

Step 7 — Safe Merge: The final synthesis stage merges branch outputs through a text-level reduction or prefix-compatible restricted composition: {𝐴𝑛𝑠𝑤𝑒𝑟 1, . . . , 𝐴𝑛𝑠𝑤𝑒𝑟𝑘 } → 𝑂𝑝𝑚𝑒𝑟𝑔𝑒 → 𝐹𝑖𝑛𝑎𝑙𝐴𝑛𝑠𝑤𝑒𝑟 . This call flow shows why AAFLOW+ is most effective for sharedprefix multi-agent workloads. The expensive evidence context is materialized once, while each branch pays only transfer/resume and continuation costs.

Eviction is aware of forked state. Shared prefix blocks are retained when multiple branches still reference them, while low-value continuation blocks can be released earlier. 17

F

EVALUATION DETAILS

prefix and scale mainly with branch continuation work. The implementation uses a finite parallel-wave model, so small agent counts can fit into one stateful wave while larger agent counts require additional waves. The multi-agent scaling results show (Table 9) that AAFLOW+ changes how latency grows with agent count. On the HF backend, the strongest comparison appears at 16 agents: for Mistral, AAFLOW+ reaches 224.39 s, while the nearest non-AAFLOW+ competitor, SGLang prefix, is 1704.86 s, giving a 7.60× speedup; dense prefill is even worse at 2716.11 s. For Llama3 at 16 agents, AAFLOW+ reaches 220.05 s, while SGLang prefix is 1679.81 s, again a 7.63× gap. The same trend holds on the live serving backends. On vLLM at 16 agents, Mistral AAFLOW+ is 465.45 s versus 3532.56 s for SGLang prefix; on SGLang, Mistral AAFLOW+ is 701.29 s versus 5363.03 s for SGLang prefix. Llama3 follows the same pattern, reaching 253.53 s versus 1926.11 s on vLLM and 329.07 s versus 2514.49 s on SGLang. AAFLOW+ outperforms other methods by materializing the shared prefix once and forking reusable state, causing memory growth in execution waves as parallel capacity is reached, rather than through repeated full-prefill operations. In contrast, nonAAFLOW+ baselines like dense prefill and AAFLOW-text allocate a complete prefill for each agent, while vLLM-prefix and SGLangprefix offer only local prefix reuse within the engine. Although local reuse reduces some duplication, it does not eliminate workflowlevel branch duplication when multiple agents share a prefix, leading to a growing scaling gap as agent count increases.

We utilize a trace-driven model parameterized by hardware microbenchmarks to rigorously isolate the fundamental systems-level KV-transfer limits from the inherently high variance of Pythonbased agentic frameworks. We show full experimental results with the Hugging Face (HF) backend and the Mistral model in Section 6 (Evaluation) of this paper. In the appendix section, we will cover all experiments with HF, vLLM, and SGLang backends with the Llama3 model and vLLM, and SGLang backends with the Mistral model. All source code and experimental results are committed to the following repository: https://github.com/arupcsedu/AAFLO W, and all environmental setup and dependencies are written on stateful_agentic_algebra/Readme.md file.

F.1

Experiment 1 extension: TTFT Reduction

The first experiment measures time to first token (TTFT) as context length increases. Text-passing systems repeatedly rebuild the same prefix and therefore pay a TTFT cost that scales with context length: 𝑇𝑇 𝐹𝑇text ∝ 𝐿. AAFLOW+ pays the prefill cost once and then resumes from the distributed state: 𝑇𝑇 𝐹𝑇state ≈ 𝑇transfer + 𝑇resume . Across all three backends shown in Table 8, AAFLOW+ achieves the lowest workflow-level TTFT because it replaces repeated prompt replay with explicit state resume. On the HF backend, the clearest gap appears on Mistral: AAFLOW+ reaches 0.041 s mean TTFT, compared with 2.057 s for AAFLOW-text, 2.017 s for dense prefill, and 0.704 s for KVCOMM. Even the strongest non-AAFLOW+ workflow baseline, DistServe style, remains at 0.123 s, still 3.0× slower. On Llama3, the same ordering holds: AAFLOW+ reaches 0.030 s, while DistServe style is 0.052 s and SGLang prefix is 0.086 s. The largest-context results show the same effect more sharply. On vLLM at maximum context, Mistral AAFLOW+ is 0.171 s, whereas dense prefill is 38.880 s, AAFLOW-text is 39.655 s, KVCOMM is 13.315 s, and DistServe style is 0.316 s. On SGLang at maximum context, Mistral AAFLOW+ is 0.876 s, while dense prefill is 13.317 s, AAFLOW-text is 13.566 s, KVCOMM is 5.089 s, and DistServe style is 1.021 s. The reason is consistent across backends. Dense prefill and AAFLOW-text always reconstruct the prompt prefix, so their TTFT grows with context length. Local-prefix baselines such as vLLMprefix and SGLang-prefix improve over dense replay, but their reuse is confined to one serving engine and does not expose explicit transfer, placement, or branch lineage across workflow agents. KVCOMM communicates KV-like state, but in these experiments it pays more modeled communication overhead and achieves lower effective reuse than AAFLOW+. DistServe style remains the closest competitor because it also separates prefill from decode, but it still lacks the explicit fork/transfer/restricted-merge workflow abstraction of AAFLOW+, so it cannot fully eliminate branch-level prefix replay.

F.2

F.3

Experiment 3: Transfer vs. Recomputation

The goal is to evaluate the scheduler decision: 𝑇transfer =

KV bytes + latency, bandwidth

𝑇recompute = 𝑇prefill .

Experiment 3 (Table 10) shows that AAFLOW+ needs both explicit state operators and a cost model. The strongest concrete result comes from the HF sweep: at 10 Gbps, transfer is beneficial for only 5 of 8 tested Mistral contexts, but at 25 Gbps and above it is beneficial for all tested contexts. The maximum transfer-over-recompute speedup rises from 2.86× at 10 Gbps to 7.14× at 25 Gbps and up to 113.88× at 400 Gbps. This gives a clear scheduling rule: transfer is not always profitable on slow links, but it becomes decisively better on RDMA-class GPU-cluster networks. The baseline systems are structurally limited: dense and textpassing approaches cannot choose between transfer and recompute since KV state is not a transferable object. Local-prefix systems only reuse cache within a server, lacking general policies for crossagent or cross-node transfer. As shown in Table 10, both vLLM and SGLang manage KV internally but do not support stable public KV export/import in this evaluation. In contrast, AAFLOW+ enables state transfer as a core workflow operation, selecting transfer only when it is more bandwidth-efficient than prompt replay, thus achieving superior performance.

Experiment 2: Multi-Agent Scaling

F.4

The goal is to evaluate how total latency changes as the number of agents increases. Text-centric systems repeat prefill and context construction for each agent. AAFLOW+ should amortize the shared

Experiment 4: Memory Efficiency

The goal is to compare peak KV memory as branch factor grows. A stateful workflow should avoid duplicating the full shared prefix for every branch. 18

Table 8: Experiment 1: Mean TTFT by baseline with HF, vLLM, and SGLang backends and 1024 - 32768 context size. HF Backend Model

Baseline

Mistral Mistral Mistral Mistral Mistral Mistral Mistral Mistral Mistral Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

vLLM Backend

SGLang Backend

Mean TTFT(s)

Speedup (slower)

Mean TTFT(s)

Speedup (slower)

Mean TTFT(s)

Speedup (slower)

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style live vLLM serve live SGLang serve

0.041 2.017 2.057 0.437 0.280 0.704 0.123 n/a n/a

1.00× 49.2× 50.2× 10.7× 6.8× 17.2× 3.0× n/a n/a

0.116 20.026 20.425 4.211 2.629 6.876 0.197 19.925 n/a

1.00× 172.6× 176.1× 36.3× 22.7× 59.3× 1.7× 171.8× n/a

0.629 7.396 7.531 2.013 1.475 2.920 0.710 n/a 6.781

1.00× 11.8× 12.0× 3.2× 2.3× 4.6× 1.1× n/a 10.8×

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style live vLLM serve live SGLang serve

0.030 0.499 0.509 0.124 0.086 0.187 0.052 n/a n/a

1.00× 16.6× 17.0× 4.1× 2.9× 6.2× 1.7× n/a n/a

0.061 3.958 4.036 0.862 0.552 1.383 0.082 3.901 n/a

1.00× 64.9× 66.2× 14.1× 9.0× 22.7× 1.3× 64.0× n/a

0.139 1.810 1.843 0.481 0.348 0.704 0.161 n/a 1.674

1.00× 13.0× 13.3× 3.5× 2.5× 5.1× 1.2× n/a 12.0×

Table 9: Experiment 2: Scaling benchmark of AAFLOW+ versus nearest non-AAFLOW+ competitor, SGLang prefix with 32768 context size AAF

HF Backend

AAF

vLLM Backend

AAF

SGLang Backend

Model

Age LOW+ SGLang Speed LOW+ SGLang Speed LOW+ SGLang Speed nts total(s) total(s) up total(s) total(s) up total(s) total(s) up

Mistral Mistral Mistral Mistral Mistral

1 2 4 8 16

30.70 30.70 58.37 113.71 224.39

119.13 224.85 436.28 859.14 1704.86

3.88× 7.32× 7.47× 7.56× 7.60×

67.10 67.71 121.86 238.75 465.45

263.40 484.22 899.15 1796.07 3532.56

3.93× 7.15× 7.38× 7.52× 7.59×

91.62 90.49 176.53 348.13 701.29

353.92 679.06 1338.65 2654.54 5363.03

3.86× 7.50× 7.58× 7.63× 7.65×

Llama3 Llama3 Llama3 Llama3 Llama3

1 2 4 8 16

29.06 29.06 56.35 110.91 220.05

112.39 216.89 425.88 843.85 1679.81

3.87× 7.46× 7.56× 7.61× 7.63×

36.02 36.01 66.26 128.62 253.53

141.03 259.80 492.36 968.75 1926.11

3.91× 7.21× 7.43× 7.53× 7.60×

43.40 44.80 84.45 159.17 329.07

167.96 335.59 638.82 1211.65 2514.49

3.87× 7.49× 7.56× 7.61× 7.64×

The memory results(Table 11) show that AAFLOW+ achieves the lowest peak KV footprint across all three backends because it represents forked state through explicit shared ownership and lineage. For Mistral, the strongest gap appears against the fully duplicating baselines: AAFLOW+ uses 8.355 GiB on HF, while AAFLOW-text uses 53.986 GiB and DistServe style uses 50.995 GiB. Even the best local-prefix baselines remain significantly higher, with vLLM-prefix and SGLang-prefix both at 14.351 GiB, 14.591 GiB and KVCOMM at 15.702 GiB. On Llama3, AAFLOW+ uses 4.210 GiB on HF, while AAFLOW-text rises to 27.202 GiB and dense prefill to 25.188 GiB; the nearest local-prefix baselines remain around 7.2 GiB.

The same ranking is preserved on vLLM and SGLang. On vLLM, Mistral AAFLOW+ uses 8.290 GiB, while AAFLOW-text is 53.568 GiB and KVCOMM is 15.580 GiB. On SGLang, Mistral AAFLOW+ again uses 8.290 GiB, while AAFLOW-text is 53.268 GiB and KVCOMM is 15.153 GiB. The advantage of AAFLOW+ lies in storing a single shared prefix state and allocating only branch-specific continuation states, while dense prefill and AAFLOW-text duplicate the entire prefix. DistServe methods add extra branch and staging state, resulting in memory usage similar to dense prefill. KVCOMM and

19

Table 10: Experiment 3: Comparison of KV transfer cost to recomputing the prompt prefil. Model

Bandwidth

Beneficial contexts

HF Backend Speedup vs Recompute

vLLM Backend Speedup vs Recompute

SGLang Backend Speedup vs Recompute

Mistral Mistral Mistral Mistral Mistral

10 Gbps 25 Gbps 100 Gbps 200 Gbps 400 Gbps

5/8 8/8 8/8 8/8 8/8

2.46× 6.14× 24.54× 49.04× 97.90×

2.54× 6.35× 25.39× 50.73× 101.29×

2.86× 7.14× 28.55× 57.04× 113.88×

Llama3 Llama3 Llama3 Llama3 Llama3

10 Gbps 25 Gbps 100 Gbps 200 Gbps 400 Gbps

1/6 6/6 6/6 6/6 6/6

2.58× 6.44× 25.76× 51.48× 102.77×

2.57× 6.43× 25.72× 51.39× 102.61×

2.37× 5.92× 23.67× 47.30× 94.44×

Table 11: Experiment 4: Mean peak KV memory. HF Backend

vLLM Backend

SGLang Backend

Mean peak KV (GiB)

Mean peak KV (GiB)

Mean peak KV (GiB)

Model

Baseline

Mistral Mistral Mistral Mistral Mistral Mistral Mistral

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

8.355 49.987 53.986 14.351 14.591 15.702 50.995

8.290 49.600 53.568 14.240 14.338 15.580 50.600

8.290 49.510 53.268 14.113 14.142 15.153 50.312

Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style

4.210 25.188 27.202 7.231 7.311 7.912 25.695

4.145 24.800 26.784 7.120 7.239 7.790 25.300

4.327 24.212 26.319 6.851 6.912 7.687 25.128

local-prefix methods reduce duplication but lack explicit sharedstate ownership, preventing them from achieving AAFLOW+’s peak memory efficiency.

F.5

dense and AAFLOW-text. On vLLM, Mistral AAFLOW+ reaches 280.30 tok/s, while the strongest workflow competitor is 36.89 tok/s. On SGLang, Mistral AAFLOW+ reaches 207.09 tok/s, while the strongest workflow competitor is 27.10 tok/s. Llama3 shows the same pattern: AAFLOW+ reaches 300.57/314.26/256.84 tok/s across HF/vLLM/SGLang, whereas the strongest workflow competitor stays near 39.27/41.35/33.61 tok/s. The overhead values explain why AAFLOW+ keeps mean modeled framework overhead fixed at 0.0075 s across both models and all three backends. By contrast, AAFLOW-text and KVCOMM pay much larger overheads: for Mistral, AAFLOW-text incurs 11.1659 s on HF, 89.5948 s on vLLM, and 40.5748 s on SGLang, while KVCOMM incurs 21.6770 s, 174.3231 s, and 78.9242 s. Dense prefill incurs minimal framework overhead but suffers from low throughput due to repeated full-prefix computation. Local-prefix and DistServestyle baselines also have modest overhead but fall short of AAFLOW+ because they retain workflow-level duplication of shared context.

Experiment 5: Throughput and Framework Overhead

The goal is to measure effective generated-token throughput and modeled framework overhead Ω. AAFLOW+ should improve workflow throughput by removing repeated prefill and reducing text orchestration. The throughput and overhead results (Table 12) show the clearest split between workflow execution and raw serving. Among the workflow rows, AAFLOW+ is best on every backend. For Mistral, AAFLOW+ reaches 302.61 tok/s on HF, compared with 39.62 tok/s for SGLang prefix, 38.81 tok/s for vLLM local prefix, 37.62 tok/s for DistServe style, 35.59 tok/s for KVCOMM, and about 32 tok/s for 20

Table 12: Experiment 5: Mean throughput and overhead. n/a = not applicable, as that specific baseline is tightly coupled with that specific backend. HF Backend Model

Baseline

Mistral Mistral Mistral Mistral Mistral Mistral Mistral Mistral Mistral Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3 Llama3

vLLM Backend

SGLang Backend

Throughput (tok/s)

Mean Ω (s)

Throughput (tok/s)

Mean Ω (s)

Throughput (tok/s)

Mean Ω (s)

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style live SGLang serve live vLLM serve

302.61 31.94 31.84 38.81 39.62 35.59 37.62 n/a n/a

0.0075 0.0019 11.1659 0.0627 0.0657 21.6770 0.0597 n/a n/a

280.30 22.78 22.62 36.11 36.89 29.67 35.23 n/a 704.08

0.0075 0.0019 89.5948 0.0627 0.0657 174.3231 0.0597 n/a 0.0

207.09 20.99 20.91 26.54 27.10 24.03 25.93 591.14 n/a

0.0075 0.0019 40.5748 0.0627 0.0657 78.9242 0.0597 0.0 n/a

AAFLOW+ dense prefill AAFLOW-text vLLM local prefix SGLang prefix KVCOMM DistServe style live vLLM serve live SGLang serve

300.57 33.61 33.54 38.46 39.27 36.16 37.46 n/a n/a

0.0075 0.0019 6.7272 0.0627 0.0657 13.0375 0.0597 n/a n/a

314.26 25.83 25.65 40.48 41.35 33.45 39.49 801.55 n/a

0.0075 0.0019 47.4580 0.0627 0.0657 92.3380 0.0597 0.0 n/a

256.84 25.65 25.54 32.92 33.61 29.63 32.15 n/a 673.26

0.0075 0.0019 23.4720 0.0627 0.0657 45.6145 0.0597 n/a 0.0

Although live vLLM and SGLang show higher raw throughput, these reflect optimized server performance, not explicit distributed state transfer. Thus, AAFLOW+ leads among workflow systems in

this evaluation, with its remaining gap to live serving attributable to explicit accounting of distributed state-transfer costs, rather than concealing them within local engines.

21