Tiara: A Programmable Line-Rate ISA for Remote Memory Access Bojie Li Pine AI
arXiv:2606.13708v1 [cs.AR] 10 Jun 2026
Abstract
(2) Multi-level translation. Page tables, storage indirection layers, and block-table lookups resolve addresses through 𝑘 levels of indirection, each requiring a dependent RTT. This pattern is increasingly critical for disaggregated LLM inference: vLLM’s PagedAttention [21] scatters KV caches into non-contiguous blocks, and the client must read a Block Table to discover physical addresses before fetching KV data. (3) Conditional multi-host coordination. Distributed locks and log replication use atomic compare-and-swap (CAS) to acquire state on one node, then conditionally propagate to replicas, forming a sequence of dependent operations spanning multiple hosts. The cost is severe: latency grows as Depth × RTT, link utilization collapses between dependent accesses, and the CPU must orchestrate every intermediate step, precisely the “killer microseconds” regime [4] where hardware latency dominates software optimization. Yue et al. [48] quantified this for disaggregated PagedAttention: fetching KV data for a single LLaMA3-70B request incurs 160 sequential RTTs, leaving a 200 Gbps link idle 83% of the time. Existing approaches each fall short: chaining RDMA verbs on the memory-side NIC [41] resolves indirection in 1 RTT but is throughput-limited; two-sided RPC consumes CPU cores disaggregated memory nodes may lack; off-path SmartNIC offloading can increase latency (§2.2). We address this with Tiara, a compact instruction set for the memory-side NIC. A Tiara operator is a pre-registered program, analogous to an eBPF program in the kernel [14]. Its core mechanism is simple: a Load returns a value into a register that can immediately serve as the address for the next Load, chaining dependent dereferences with no round-trips. Around this core sit control-flow instructions (forward-only Jump, bounded Loop, async Wait) and integer Compute for address arithmetic. Static verification at registration guarantees bounded execution and sandboxed access. Tiara attempts no general computation, only the minimal address resolution that turns an indirect access into a direct one. We make two contributions: (1) Tiara, a minimal, statically verifiable NIC-side ISA. The contribution is not NIC-side programmability itself (RedN, RMC, NAAM, and BF-3 DPA all offer that) but the specific subset that simultaneously (i) collapses dependent dereferences in a hardware-native sub-𝜇s path that software-dispatched ARM/RISCV cores cannot reach (Fig. 3), and (ii) admits eBPF-style termination and memory-region verification at registration time, so operators are safe for multi-tenant sharing without runtime checks. (2) An FPGA prototype on Alveo U50, evaluated on five workloads. Compared to one-sided RDMA: 2.85× lower graph-traversal latency at depth 10 and 3.4× higher throughput at depth 3, 62%
RDMA one-sided verbs are the natural primitive for memory disaggregation, but they require the client to supply the exact remote address. The 1-RTT performance breaks down when the target address depends on data that must first be read from remote memory, a pattern we call the Indirection Wall. Indirection is pervasive: graph traversals follow pointers hop by hop, address translation walks multi-level page tables, distributed coordination requires conditional multi-host logic, and disaggregated LLM inference must resolve paged KV caches through block-table lookups. Each level of indirection costs one sequentially dependent network round-trip, yet offloading to existing RDMA NICs either consumes remote CPU cycles or has limited throughput. We present Tiara, a compact, statically verifiable instruction set that executes on the memory-side NIC. Tiara operators are pre-registered programs, analogous to eBPF programs in the kernel, that resolve indirection locally, collapsing multi-RTT dependent chains into a single roundtrip. On an FPGA-based prototype, Tiara reduces 10-hop graphtraversal latency by 2.85× over one-sided RDMA while sustaining 3.4× higher throughput, cuts page-table walk latency by 62%, reduces uncontended distributed-lock latency by 2.9×, achieves 2.8× throughput for disaggregated PagedAttention at 8 KB blocks, and 1.88× MoE expert-gather latency at 32 experts.
Keywords RDMA, remote memory access, SmartNIC, memory disaggregation, programmable networks, instruction set architecture
1
Introduction
Datacenter workloads increasingly demand memory beyond a single server’s capacity, separating compute and memory into networkattached pools [1, 42]. One-sided RDMA verbs are the natural primitive for this disaggregated setting: the requester specifies a remote address and the remote NIC completes the access directly, without notifying the remote CPU, a model a decade of systems has meticulously exploited [10, 19, 20, 30, 46]. But every verb requires the client to supply the exact remote address at issue time. This works when the address is computable locally (e.g., fixed-size HPC matrices), but a growing class of workloads derives the address from remote data that must be read first. The client cannot know the next address until the current read returns, so each dependent lookup costs a full round-trip that no batching or prefetching can hide. We call this sequential bottleneck the Indirection Wall, with three recurring patterns: (1) Pointer chasing. Graph traversals and linked data structures store each successor’s address in the current node. Every hop requires one sequential network round-trip. This pattern arises in social-network analysis, knowledge-graph queries, and agentdriven graph-RAG pipelines. 1
Bojie Li
(a) One-sided RDMA
Table 1: RTT cost of indirection across workloads. Workload
Pattern
RDMA
Tiara
Graph traversal (depth 𝑑) Page-table walk (3-level) Dist. lock + replication PagedAttention MoE expert loading
Pointer chase Multi-level translation CAS + cond. writes Table lookup + gather Paged translation
𝑑 RTTs 3+1 RTTs 5 RTTs 160† / 2‡ 2 RTTs
1 RTT 1 RTT 2 RTTs 1 RTT 1 RTT
Sparse attention (NSA)
Score-then-select
2 RTTs
1 RTT
Client NIC
(b) CPU-based RPC
Read L1 L2 ptr
† Unoptimized stop-and-wait, as deployed today [48]. ‡ Optimally batched: 1 RTT to read the block table, then 160 block reads in a 2nd RTT (requires constructing 160 work requests on the client).
Client NIC
RPC(vAddr)
Invoke(vAddr)
dispatch
idle
Read L2 L3 ptr
idle
Read L3 pAddr
idle
Read data Data
Load L1 Load L2 no CPU Load L3 1 RTT Memcpy
Load L1 Load L2 Load L3 1 RTT
4 RTTs
Memcpy
Data
1 RTT, but burns CPU cores
NIC resolves locally 1 RTT no CPU involvement
Data
1 RTT per level 4 sequential RTTs
lower page-table walk latency, 3.1× lower distributed-lock latency at 16 clients, 2.78× PagedAttention throughput at 8 KB blocks, and 1.88× MoE expert-gather latency at 32 experts. Tiara is open source at https://github.com/bojieli/Tiara.
(c) Tiara
Client NIC CPU
Figure 1: Three approaches to data-dependent remote access (e.g., a 3-level page-table walk). (a) One-sided RDMA incurs one RTT per indirection level plus one for the final data fetch. (b) CPU-based RPC resolves all levels in 1 RTT but consumes CPU cores on the memory node. (c) Tiara resolves all levels via host DRAM on the NIC and returns data in 1 RTT, without CPU involvement.
2 Motivation 2.1 The Cost of Indirection
RDMA BlueField−2
Latency (µs)
One-sided RDMA pays one RTT per indirection level; these RTTs are sequentially dependent: the client cannot issue level-𝑖+1 until level-𝑖 completes, and no batching or pipelining can help. Table 1 quantifies this cost across the five workloads we evaluate, plus sparse attention (NSA), an emerging AI pattern. Indirection across the AI stack. The Indirection Wall extends beyond classical systems. In MoE serving with disaggregated expert weights, the mapping from expert ID to physical location goes through a translation table, structurally identical to PagedAttention’s Block Table. DeepSeek’s Native Sparse Attention (NSA) [8] scores compressed keys in remote memory to select which full blocks to fetch, a “score-then-select” pattern where the decision of what to read depends on remote data. As AI systems adopt disaggregated memory [27, 37, 40, 50], data-dependent address resolution becomes a systematic bottleneck.
10 5 0
Ato
A H P D L mic tomic T GE T Wa ist.Lo ogRe pl Rd Wr T lk ck
End-to-end latency (μs)
Figure 2: BlueField-2 vs. one-sided RDMA latency.
Indirection vs. scatter-gather. Not all irregular access patterns are indirection. In embedding table lookups [15], all reads are independent and parallelizable via RDMA scatter-gather lists. The Indirection Wall arises specifically when the address depends on remote data that must be read first.
crossover (= RTT)
Memory-side offload
8
One-sided RDMA
6 Tiara FPGA PCIe (0.75 µs)
4
BF-3 DPA (~0.85 µs)
2 0 0.0
0.5
1.0
1.5
2.0
2.5
3.0
Host-memory access latency (μs)
Case Study: Graph Traversal. Social-network graphs stored in disaggregated memory exhibit pure pointer chasing. A 𝑘-hop query (friend-of-friend, influence propagation, community detection) reads a node, follows an edge to a neighbor, and repeats; each hop’s address depends on the previous hop’s data, so no prefetching is possible. Knowledge graphs and graph-based retrieval-augmented generation (graph-RAG) pipelines face the same bottleneck: multihop reasoning requires sequential node fetches whose addresses are discovered only at runtime. With one-sided RDMA, latency grows as 𝑘 × RTT; Tiara resolves all hops via local DRAM reads, keeping latency near 1 RTT regardless of depth (§4.2).
Figure 3: Offloading beats RDMA when host-memory latency < RTT. reads (level-1 → level-2 → level-3 entry → physical address): onesided RDMA costs 4 RTTs, while Tiara resolves all three levels locally in 1 RTT. This page table is not the OS page table but the blockindirection table disaggregated runtimes build over remote pools for dynamic reallocation, of which vLLM’s Block Table and MoE’s expert table are instances. ConnectX MTTs translate one flat region but cannot represent multi-level or application-managed indirection, and IOMMUs operate only on host-side IO mappings. This is the on-demand remote-paging path of systems such as ODRP [44]: a node faulting on a remote page must walk a memory-node-resident translation structure before the fetch, a walk Tiara collapses into the same round-trip as the fetch.
Case Study: Page-Table Walk. Consider a compute node accessing remote memory through a 3-level page table on the memory node. Translating a virtual address chains three strictly dependent 2
Tiara: A Programmable Line-Rate ISA for Remote Memory Access
Remote Nodes
hard to realize in fixed logic. Tiara trades that generality for two properties these designs cannot match. Latency: BF-3 DPA’s 0.85 𝜇s host access nearly matches Tiara’s 0.75 𝜇s, but Tiara’s hardware MPs commit a load one cycle after writeback with no dispatch, while off-path BF-2 ARM cores regress (Fig. 2). BF-3 DPA’s 0.85 𝜇s stays below the RTT crossover (Fig. 3), a viable target that narrows the gap. One could instead restrict the source language and target an existing ISA (Tiara itself compiles from a restricted OpenCL C subset, §3.3), but that buys static safety on any target, not the sub-𝜇s per-hop cost. The minimal ISA is what makes this cheap: a 2.95 K-LUT MP (§4.1) replicates 8× in ∼3% of a ConnectX die, whereas a RISC-V/ARM core is one-to-two orders larger and cannot be replicated at line rate; the contribution is this hardware/ISA co-design, not verification alone. Safety: NAAM leans on eBPF’s runtime verifier, while Tiara proves termination and region isolation statically with no runtime guard. Stateless data planes (P4 [5], NPUs such as Netronome) cannot express data-dependent loops or dependent loads against host memory, so none of the workloads in Table 1 fit.
RDMA
Tiara NIC
RDMA Engine Memcpy
Task Dispatcher
MP0 Regs, ALU IStore
MP1 Regs, ALU IStore
...
MP7 Regs, ALU IStore
PCIe DMA Engine PCIe
Host
Host DRAM (Memory Regions)
Host CPU
register operators
Compiler & Verifier
Figure 4: Tiara NIC architecture.
2.2
Existing Approaches
Figure 1 contrasts three approaches. CPU-based RPC (b) resolves all levels on the memory node in 1 RTT, but dedicates CPU cores to address arithmetic. NIC-side approaches (c) avoid CPU involvement but, as we show below, each introduces its own bottleneck.
3
Verb chaining. RedN [41] proved RDMA Turing-complete by chaining self-modifying work requests (WRs, the descriptors that tell the NIC what operation to perform). RedN executes on the memoryside NIC, resolving indirection in a single RTT. However, it relies on doorbell ordering: the CPU must signal (“ring a doorbell” via a PCIe write) after posting each WR, and the NIC processes them one at a time. This forces the NIC to fetch each WR individually from host memory (∼0.54 𝜇s per WR [41]), limiting throughput to ∼1 Mops, 65× below raw RDMA Read rates (§4.2). CPU-based RPC.. Two-sided RPC resolves addresses in 1 RTT, but software RPC consumes 22 CPU cores to saturate a 25 Gbps NIC’s message rate [18]. Disaggregated memory nodes increasingly lack general-purpose CPUs entirely (CXL memory pools, memory blades) [1]. Even when a CPU is available, RPC dispatch overhead adds 1 to 3 𝜇s [18, 36], often exceeding the useful work.
Tiara Design
Figure 4 shows the overall architecture. The Tiara NIC contains an RDMA engine, a task dispatcher, and 8 lightweight memory processors (MPs), each with a register file (16×64 b), an integer ALU, a depth-8 loop stack, a 32-entry in-flight async counter, and a BRAM instruction store for pre-registered operators. At registration time (dashed path), the host-side compiler and static verifier check operators before loading them into the NIC’s instruction stores via PCIe. At execution time (solid paths), incoming RDMA requests are dispatched to an MP, which accesses host DRAM via PCIe DMA. We detail the programming model (§3.1), instruction set (§3.2), and compiler and verifier (§3.3) below. Execution and multi-tenant operation. An MP is a sequential scalar core (11-state FSM, no cache, no branch prediction, no out-of-order): register-chained loads are made correct by stalling fetch until writeback. A 256-entry op_id → start_pc table in front of the dispatcher routes incoming requests to any registered operator in O(1), so one NIC hosts many tenants’ operators concurrently, an opaque per-task tag routing responses to the right caller. Isolation is static: every operator is verified at registration to access only its declared regions, so the runtime needs no per-access check and one tenant’s operator cannot reach another tenant’s memory regardless of how it executes. The memory subsystem is uncached and regionpartitioned (a device-id router sends local accesses to PCIe DMA, remote ones to the RDMA engine); we add no MP-side cache, since indirection’s poor locality offers nothing to cache. The shared resources are the 8 MPs and the host PCIe channel; the current dispatcher is work-conserving but not weighted, and per-tenant tokenbucket scheduling is a one-state extension left to future work. Higher line rates scale by instantiating more MPs, each only ∼3 K LUT (§4.1).
SmartNIC offloading. Running C/C++ on off-path SmartNIC ARM cores BlueField DPU [9] and FlexIO DPA [34] avoid the host CPU but offer no termination guarantee or memory sandboxing. Worse, it is not even faster. To validate this, we implement Tiara operators on NVIDIA BlueField-2 [29, 45], where the ARM cores access host memory via internal RDMA. Two servers are connected back-to-back via a DAC cable (RTT ∼1.9 𝜇s), giving the SmartNIC the shortest possible network RTT. Figure 2 shows that offloading to BlueField-2 increases latency for every operator: an atomic read regresses by 38%. The root cause is that each host-memory access costs 1.7 𝜇s via internal RDMA, close to the 1.9 𝜇s cable RTT. Figure 3 sweeps host-memory latency analytically: the crossover where offloading becomes worthwhile occurs at the network RTT. Tiara’s PCIe DMA at 0.75 𝜇s places it well below the crossover. Active messaging and on-path DPA.. NAAM [3] runs pre-registered eBPF on host or SmartNIC ARM cores; BF-3 DPA [35] adds 16 onpath RISC-V cores with ∼0.85 𝜇s host-memory access. Both share Tiara’s pre-registration vision and would collapse multi-RTT chains, and NAAM’s eBPF offers richer programmability (maps, helpers)
3.1
Programming Model
An operator is a small program of Tiara instructions, registered on a NIC before use (analogous to loading an eBPF program into 3
Bojie Li
Table 2: Tiara instruction set. Instruction
Description
Load/Store
CAS/CAA
Register ↔ local memory. A loaded value can be the next address. Bulk transfer with unified (device, addr) addressing; subsumes RDMA Read/Write. Atomic compare-and-swap / compare-and-add.
Jump Loop(M,N) Wait Ret
Forward-only conditional branch. Execute next 𝑁 ops for 𝑀 iterations. Block until in-flight async ops ≤ threshold. Return result to caller.
ComputeOp
Integer arithmetic, logical, shift for address computation.
Memcpy
Operator DistLock(latch, state, newVal, r1Dev, r1Addr, r2Dev, r2Addr): Loop(MAX_RETRIES, 2) // bounded CAS retry ok = CAS(latch, 0, 1) // local memory Jump(ok == 0, acquired) Ret(FAIL) acquired: old = Load(state) Store(state, newVal) Memcpy((r1Dev,r1Addr), state, 8) // async Memcpy((r2Dev,r2Addr), state, 8) // async Wait(0) // both replicas ACK Store(latch, 0) // release Ret(old)
Figure 5: Distributed lock operator. the operator can test via conditional Jump to execute a fallback path (e.g., skip the failed replica) or return an error to the caller.
the kernel); registration triggers compilation and static verification (§3.3). A client invokes it by sending a single message with the operator ID, parameters (up to 8 registers), and optional inline data. The NIC creates a task backed by a register file (16×64 b) and executes the operator without involving the host CPU. Pre-registration is key: only the operator ID and parameters travel on the wire: one message per invocation. In contrast, PRISM [6] and RedN [41] transmit multiple work requests per task on every call, consuming PCIe and network bandwidth proportional to task complexity. An operator accesses local memory via DMA and remote memory on any node via Memcpy with unified addressing, enabling multihost orchestration in a single invocation, e.g., a distributed lock operator CAS-acquires a latch, replicates state to two replicas via parallel async Memcpy, waits for acknowledgments, and returns, all without CPU involvement on any node. PRISM’s chaining primitives are scoped to a single client-server pair; Tiara’s unified addressing removes this limitation.
3.2
3.3
Compiler and Verifier
Compiler. Operators are written in a restricted subset of OpenCL C [32] so that every operator’s control flow is a Static Control Part (SCoP) [13], making termination and resource bounds decidable at compile time. An LLVM-based [22] toolchain lowers OpenCL C to IR (standard passes run unmodified); a custom Tiara backend does linear-scan register allocation [39], flattens loops into bounded Loop(M,N), selects opcodes, and inlines write data with the request to fold a 2-RTT read-then-write into 1 RTT. The resulting binaries are compact (typically 10 to 50 instructions). Static verification. Operators are verified at registration time, before they ever execute on the data path. This is the critical differentiator from SmartNIC C programming: (1) Termination guarantee. Forward-only jumps (no backward branches) and bounded loop iterations give every operator a statically computable upper bound on execution steps. The verifier computes this bound and rejects operators exceeding a configurable limit. (2) Fine-grained access control. One-sided RDMA exposes entire memory regions to any client with the region key. Tiara’s verifier checks at registration that every memory access falls within server-configured regions. Server-registered operators can encapsulate restrictive logic (e.g., return usernames but never passwords), and the client never sees raw memory, making Tiara both more secure and more flexible than one-sided RDMA.
Instruction Set
Table 2 lists Tiara’s instructions. The design principle is minimal but sufficient: enough to express all indirection patterns in Table 1, simple enough to verify statically and implement in hardware. We derived this set by decomposing all workloads in Table 1 into elemental operations: every instruction is required by at least one workload, and removing any breaks that workload. Three design choices merit emphasis. The key enabler is registerchained loads: a Load writes its result into a register that a subsequent Load can use as its address operand the very next cycle, turning a multi-RTT pointer chase into local memory accesses, each ∼0.75 𝜇s on our FPGA prototype versus a full network RTT. Second, unified addressing: addresses are (host_id, region_id, offset) tuples. A Memcpy with remote source and local destination is an RDMA Read; with remote destination, an RDMA Write. This eliminates separate verb types and simplifies multi-host operators. Third, async + wait: Memcpy instructions execute asynchronously; Wait(threshold) synchronizes. Setting the threshold to 0 waits for all in-flight operations; a threshold > 0 enables quorum-style synchronization (e.g., proceed when all but one replica acknowledges). It also enables pipelining: issuing KV block reads as blocktable entries are resolved, overlapping address resolution with transfer. An async op to a failed node times out and sets an error flag
3.4
Example Operators
Distributed lock. Figure 5 shows a lock operator spanning three hosts: the NIC acquires a latch via local CAS (retried in a bounded Loop), updates state, replicates to two backups via parallel async Memcpy, and releases, collapsing RDMA’s 5 sequential RTTs into 2 (client→primary, then primary→replicas in parallel) with no CPU involved. Page-table walk. Three chained Loads (each using the previous result as the next address) resolve a 3-level page table; a final Memcpy transfers the data page. RDMA: 4 RTTs; Tiara: 1 RTT. The same pattern extends to PagedAttention: a Loop iterates over block IDs, resolving each via Load and issuing async Memcpy for the KV block. 4
Tiara: A Programmable Line-Rate ISA for Remote Memory Access
4 Evaluation 4.1 Setup
25
Latency (μs)
Testbed. Two dual-socket Intel Xeon Gold 6326 servers are connected via 100 GbE QSFP28 through a single TOR switch (RDMA Read RTT ∼2.5 𝜇s). Baselines run on NVIDIA BlueField-2 NICs; for Tiara the memory-side NIC is an AMD Alveo U50 FPGA running the Corundum [12] NIC stack, accessing host DRAM via PCIe.
RDMA
20
RPC RedN
15
PRISM
10 5
Implementation. We implement Tiara’s execution engine as an extension to the Corundum NIC pipeline on the Alveo U50: 8 memory processors at 200 MHz, each with a 16-register file and a 1024entry BRAM instruction store. Each MP is only 2.95 K LUT (+1.69 K FF); the single-MP build meets timing at 200 MHz post-route with +0.184 ns slack (Fmax ∼208 MHz), matching the Corundum app clock. The remaining ∼24 K LUT is the OOC memory stub, replaced by Corundum DMA in deployment.
1
2
3
4
5
6
7
8
9
10
Traversal depth (hops)
Throughput (Mops, log)
Figure 6: Graph traversal latency vs. depth.
Measurement methodology. Tiara latencies are cycle-accurate on the Verilator model (5 ns clock, 150-cycle PCIe DMA, 500-cycle RDMA RTT, calibrated to the U50 build); saturated throughput is derived from measured latency for 8 MPs × 12 outstanding tasks (96 dispatcher slots). Non-Tiara baselines are analytical models, parameterized identically and printed alongside the measured data in each .dat file: RDMA [28] at 2.5 𝜇s RTT; RPC (eRPC-style [18]) at 1.5 𝜇s dispatch + 0.17 𝜇s/cached-DRAM hop, at 16 and 22 (saturation) cores; RedN [41] at 1.1 𝜇s/WR; PRISM [6] at 0.5 𝜇s/hop (graph only; it lacks arithmetic, loops, multi-host coordination). BF2 points (Fig. 2) are measured on a physical NIC; the BF-3 DPA marker (Fig. 3) is from NVIDIA’s datasheet [35].
4.2
Tiara
101
100
1
Tiara
RPC (22c)
RDMA
RedN
RPC (16c)
PRISM
2
3
4
5
6
7
8
9
10
Traversal depth (hops)
Figure 7: Graph traversal throughput vs. depth. ∼1 Mops, ∼26× below RDMA at depth 1, due to doorbell-ordering overhead (§2), serializing execution across only 8 processing units. Tiara avoids this: operators reside in FPGA fabric with no WR fetching.
Graph Traversal
Setup. A social network graph in remote RDMA-registered memory; 64-byte nodes with adjacency-list pointers. The Tiara operator performs a depth-limited walk.
4.3
Results. Figure 6 shows latency vs. depth. RDMA grows linearly at 𝑑×RTT. RPC is nearly flat (all hops resolve via node-local DRAM, ∼0.17 𝜇s/hop) but dispatch overhead dominates and it consumes a CPU core per traversal. Tiara scales at 1 RTT + 𝑑 × 0.79 𝜇s, 2.85× faster than RDMA at depth 10 (8.78 vs. 25.0 𝜇s). RedN and PRISM also run on the memory-side NIC; PRISM scales at ∼0.5 𝜇s/hop, RedN higher from doorbell ordering (§2). In a production ASIC, Tiara’s per-hop cost would match PRISM’s with a richer ISA; the current gap is inherent to FPGA PCIe latency [23].
Page-Table Walk
Setup. A 3-level page table in remote RDMA-registered memory (8-byte entries, table sizes representative of a 256 GB disaggregated memory pool). Results. Figure 8 shows latency. RDMA needs 4 RTTs (10.0 𝜇s); Tiara resolves all three levels via PCIe in 1 RTT (3.75 𝜇s), a 62% reduction (2.7×). RedN needs extra WRs for shift/mask arithmetic per level, amplifying its doorbell overhead. On throughput Tiara sustains ∼25 Mops vs. RDMA’s 0.1 Mops, since each translation is one network message, not four.
Latency vs. RPC.. Tiara’s per-hop cost is 0.79 𝜇s (PCIe to host DRAM) vs. 0.17 𝜇s for cached-DRAM RPC, so RPC overtakes Tiara on latency beyond depth 5. Tiara still wins where it matters: saturated throughput (below), preserving (or eliminating) the memory node’s CPU (the CXL/memory-blade case RPC cannot serve), and footprint (8 MPs in ∼3% of a ConnectX die vs. ≥22 cores per 25 GbE NIC for saturated RPC).
4.4
Distributed Lock
Setup. A read-write lock replicated on one primary and two replicas. Lock acquisition: CAS to acquire latch → read/update lock state → replicate to 2 replicas → release latch.
Throughput. Figure 7 shows saturated throughput (log scale). Tiara reaches 29.5 Mops at 𝑑=3, 6.1× higher than RPC even at its 22core saturation point (4.88 Mops), and 8.3× higher than RPC at the 16-core paper baseline (3.55 Mops). PRISM tracks close to RDMA (NIC-native primitives, no doorbell ordering). RedN sustains only
Results. Figure 9 shows lock-acquire latency under 1 to 16 contending clients. Without contention, RDMA requires 5 sequential RTTs (CAS + read + 2 replica writes + release); Tiara collapses this to 2 RTTs: the first RTT delivers the request to the primary NIC, which executes the CAS locally and issues parallel replica writes 5
Bojie Li
4.5
3.8
12
Tiara
30
RDMA
Throughput (GB/s)
10 8 6 4 2 0
35
10.0
Latency (μs)
Latency ( s)
3-level page-table walk
RPC
25
RedN
20 15 10
1
Tiara
RDMA
RPC
2
4
RedN
6 4 2
8
16
211
213
215
217
KV block size (bytes, log)
Figure 9: Distributed lock latency vs. con-Figure 10: PagedAttention throughput vs. tention. KV block size. PRISM [6] adds NIC-native indirection and chaining; Tiara stores operators in NIC BRAM, avoiding per-WR PCIe fetches. RMC [2] and NAAM [3] share Tiara’s pre-registration model (software cores running C or eBPF); the differentiator is the hardware MP path that drops per-hop cost below the software-dispatch floor (Fig. 3). ODRP [44], Storm [33], and KV-Direct [23] achieve 1-RTT with data-structure-specific paths; Tiara supports many patterns from one design.
MoE Expert Gather
A MoE serving layer fetches 𝑘 expert-weight slabs (8 KB each) through a translation table indexed by selected expert IDs, structurally the same block-table indirection as PagedAttention. Tiara is cycle-accurate on the prototype; RDMA/RPC use the §4.1 baselines. At 32 experts: Tiara 14.2 𝜇s, RDMA 26.7 𝜇s (1.88×), RPC 41.7 𝜇s (2.93×); the gap grows with 𝑘 as RPC’s per-expert dispatch dominates while Tiara stays pipelined via async Memcpy.
Disaggregated PagedAttention
Setup. A memory node holds a vLLM-style paged KV cache [21]. A Block Table maps logical block IDs to physical addresses; task: fetch 8 MB of KV data (one layer’s KV cache for 2048 tokens, LLaMA370B) over 100 GbE. We vary the KV block size and measure effective throughput (GB/s), which captures both per-block overhead and data transfer cost. The Tiara operator resolves each block via Load and pipelines resolution with transfer via async Memcpy. We compare against optimally batched RDMA (2 RTTs), RPC, and RedN.
SmartNIC, disaggregated memory, and in-network computing. FlexIO DPA [34] and BlueField DPU [9, 29] offer general-purpose C; Tiara trades generality for static guarantees. ClickNP [24], iPipe [26], Floem [38], AccelTCP [31], and StRoM [43] process at the packet and transport level. FaRM [11], DrTM+R [7], FORD [49], Pilaf [30], and AIFM [42] minimize indirection via RDMA-friendly layouts; Tiara handles the residual at the NIC. NetLock [47], NetCache [17], 1Pipe [25], P4 [5], and nanoPU [16] target switch-side offload; Tiara is complementary on the memory-side NIC.
6
Conclusion
The Indirection Wall, where the remote address depends on remote data, is a fundamental obstacle for one-sided RDMA across disaggregated memory, graph analytics, and LLM inference, and CXL pooling that removes host CPUs will only widen it. Tiara shows that a compact, statically verifiable NIC-side ISA collapses multiRTT indirection into a single round-trip. Extending the integeronly ISA to floating-point workloads without breaking static verification is future work.
7
Acknowledgments
This paper was initially submitted to APNet 2026 and accepted for publication. We thank the anonymous reviewers and our shepherd for their valuable feedback. We used Cursor and Claude Code extensively for code generation and paper writing. Because ACM and APNet 2026 policies do not accept papers produced substantially by generative AI, we decided to withdraw the paper from APNet 2026 and publish it on arXiv instead. This work was initially a submission to SIGCOMM 2023 during the author’s time at Huawei. The current implementation is cleanslate and does not use any proprietary materials from Huawei. We thank the collaborators at Huawei who gave rise to the initial idea and the initial experiments for this paper. The author list of that earlier version was: Chengjun Jia (Tsinghua University), Bojie Li, Lijun Li, Xiaoping Fan, Changhu Chen, Haifeng Lin, Haonan Chen,
Results. Figure 10 shows throughput vs. block size. At small blocks (1 to 4 KB) per-block overhead dominates: Tiara reaches 8.7 GB/s at 4 KB while batched RDMA achieves only 2.7 GB/s (client-side WR construction). Tiara saturates effective line rate (∼12 GB/s) at just 8 KB blocks (2.8× batched RDMA), because its pipelined resolvethen-transfer hides resolution behind transfer; other systems converge only at ≥256 KB. RedN tracks RPC at large blocks but lags at small ones (per-block doorbell ordering).
5
RDMA RPC
8
Contending clients
via async Memcpy; the second RTT covers the replica acknowledgments (Wait). No host CPU is involved on any node. RedN reduces RDMA’s 5 RTTs to 1 RTT but pays doorbell-ordering overhead for the replica write WR chains. Under contention (16 clients), RPC degrades the least (∼1.2×) because CAS retries are nanosecond-scale CPU-local operations, overtaking Tiara at ∼4 clients, the same axis as the graph-traversal latency tradeoff above: RPC wins on perclient latency when a CPU core is available, Tiara wins when the memory node has no CPU to dedicate or when concurrent throughput matters more. RedN degrades ∼4.9× from 1 to 16 clients (doorbell overhead per retry), and RDMA 2.5× (each failed CAS incurs an RTT).
4.6
Tiara
5
Figure 8: Page-table walk latency.
4.5
10
Related Work
Programmable RDMA and active messaging. RedN [41] chains self-modifying work requests in 1 RTT but at limited throughput (§2); 6
Tiara: A Programmable Line-Rate ISA for Remote Memory Access
Yong Chao, Uri Hasson, Eti Siminchi, Zvika Rubin, Shamir Rabinovitch, Mingxiang Li, Han Ruan, Kai Zheng, Jingbin Zhou, and Kun Tan (all Huawei).
[20] Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (Savannah, GA, USA) (OSDI’16). USENIX Association, USA, 185–201. [21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP. 611–626. [22] Chris Lattner. 2008. LLVM and Clang: Next generation compiler technology. In The BSD conference, Vol. 5. 1–20. [23] Bojie Li, Zhenyuan Ruan, Wencong Xiao, Yuanwei Lu, Yongqiang Xiong, Andrew Putnam, Enhong Chen, and Lintao Zhang. 2017. KV-Direct: HighPerformance In-Memory Key-Value Store with Programmable NIC (SOSP). 16 pages. doi:10.1145/3132747.3132756 [24] Bojie Li, Kun Tan, Layong (Larry) Luo, Yanqing Peng, Renqian Luo, Ningyi Xu, Yongqiang Xiong, Peng Cheng, and Enhong Chen. 2016. ClickNP: Highly Flexible and High Performance Network Processing with Reconfigurable Hardware (SIGCOMM). 14 pages. doi:10.1145/2934872.2934897 [25] Bojie Li, Gefei Zuo, Wei Bai, and Lintao Zhang. 2021. 1Pipe: Scalable total order communication in data center networks. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference. 78–92. [26] Ming Liu, Tianyi Cui, Henry Schuh, Arvind Krishnamurthy, Simon Peter, and Karan Gupta. 2019. Offloading Distributed Applications onto SmartNICs Using IPipe (SIGCOMM). 16 pages. doi:10.1145/3341302.3342079 [27] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In SIGCOMM. [28] Mellanox. 2015. RDMA Aware Networks Programming User Manual Rev 1.7. https://network.nvidia.com/related-docs/prod_software/RDMA_Aware_ Programming_user_manual.pdf Accessed: 2021-11-22. [29] Mellanox. 2021. NVIDIA BLUEFIELD-2 DPU. https://www.nvidia.com/content/ dam/en-zz/Solutions/Data-Center/documents/datasheet-nvidia-bluefield-2dpu.pdf Accessed: 2022-04-20. [30] Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using { OneSided } { RDMA } Reads to Build a Fast, { CPU-Efficient } { Key-Value } Store. In 2013 USENIX Annual Technical Conference (USENIX ATC 13). 103–114. [31] YoungGyoun Moon, SeungEon Lee, Muhammad Asim Jamshed, and KyoungSoo Park. 2020. AccelTCP: Accelerating Network Applications with Stateful TCP Offloading. In Proceedings of the 17th Usenix Conference on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI’20). USENIX Association, USA, 77–92. [32] Aaftab Munshi. 2009. The opencl specification. In 2009 IEEE Hot Chips 21 Symposium (HCS). IEEE, 1–314. [33] Stanko Novakovic, Yizhou Shan, Aasheesh Kolli, Michael Cui, Yiying Zhang, Haggai Eran, Boris Pismenny, Liran Liss, Michael Wei, Dan Tsafrir, and Marcos Aguilera. 2019. Storm: A Fast Transactional Dataplane for Remote Data Structures. In Proceedings of the 12th ACM International Conference on Systems and Storage (SYSTOR). doi:10.1145/3319647.3325827 [34] NVIDIA. 2023. NVIDIA DOCA FlexIO SDK. https://developer.nvidia.com/ networking/flexio [35] NVIDIA. 2024. NVIDIA BlueField-3 DPU Datasheet – Datapath Accelerator. https://www.nvidia.com/en-us/networking/products/data-processing-unit/. [36] Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. 2019. Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads.. In NSDI, Vol. 19. 361–378. [37] Pratyush Patel, Esha Choukse, Chaojie Zhang, Í nigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In ISCA. [38] Phitchaya Mangpo Phothilimthana, Ming Liu, Antoine Kaufmann, Simon Peter, Rastislav Bodik, and Thomas Anderson. 2018. Floem: A Programming System for NIC-Accelerated Network Applications. In OSDI. https://www.usenix.org/ conference/osdi18/presentation/phothilimthana [39] Massimiliano Poletto and Vivek Sarkar. 1999. Linear scan register allocation. ACM Transactions on Programming Languages and Systems (TOPLAS) 21, 5 (1999), 895–913. [40] Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. In FAST. [41] Waleed Reda, Marco Canini, Dejan Kostić, and Simon Peter. 2022. RDMA is Turing complete, we just did not know it yet! (NSDI). [42] Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. 2020. AIFM: High-Performance, Application-Integrated Far Memory. In OSDI. [43] David Sidler, Zeke Wang, Monica Chiosa, Amit Kulkarni, and Gustavo Alonso. 2020. StRoM: Smart Remote Memory (EuroSys). Article 29, 16 pages. doi:10. 1145/3342195.3387519
References [1] Marcos K Aguilera, Kimberly Keeton, Stanko Novakovic, and Sharad Singhal. 2019. Designing far memory data structures: Think outside the box. In Proceedings of the Workshop on Hot Topics in Operating Systems. 120–126. [2] Emmanuel Amaro, Zhihong Luo, Amy Ousterhout, Arvind Krishnamurthy, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. 2020. Remote Memory Calls. In Proceedings of the 19th ACM Workshop on Hot Topics in Networks (Virtual Event, USA) (HotNets ’20). Association for Computing Machinery, New York, NY, USA, 38–44. doi:10.1145/3422604.3425923 [3] Anonymous. 2025. NAAM: NIC-Accelerated Active Messaging for Disaggregated Memory. arXiv preprint arXiv:2509.07431. [4] Luiz Barroso, Mike Marty, David Patterson, and Parthasarathy Ranganathan. 2017. Attack of the Killer Microseconds. Commun. ACM 60, 4 (mar 2017), 48–54. doi:10.1145/3015146 [5] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, and David Walker. 2014. P4: Programming Protocol-Independent Packet Processors. SIGCOMM Comput. Commun. Rev. 44, 3 (jul 2014), 87–95. doi:10.1145/2656877. 2656890 [6] Matthew Burke, Sowmya Dharanipragada, Shannon Joyner, Adriana Szekeres, Jacob Nelson, Irene Zhang, and Dan R. K. Ports. 2021. PRISM: Rethinking the RDMA Interface for Distributed Systems (SOSP). doi:10.1145/3477132.3483587 [7] Yanzhe Chen, Xingda Wei, Jiaxin Shi, Rong Chen, and Haibo Chen. 2016. Fast and General Distributed Transactions Using RDMA and HTM. In Proceedings of the Eleventh European Conference on Computer Systems (London, United Kingdom) (EuroSys ’16). Association for Computing Machinery, New York, NY, USA, Article 26, 17 pages. doi:10.1145/2901318.2901349 [8] DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024). [9] KEVIN DEIERLING. 2020. What Is a DPU. https://blogs.nvidia.com/blog/2020/ 05/20/whats-a-dpu-data-processing-unit Accessed: 2021-11-22. [10] Aleksandar Dragojević, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: Fast Remote Memory. In NSDI. https://www.usenix.org/ conference/nsdi14/technical-sessions/dragojevic [11] Aleksandar Dragojević, Dushyanth Narayanan, Edmund B Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, and Miguel Castro. 2015. No compromises: distributed transactions with consistency, availability, and performance. In Proceedings of the 25th symposium on operating systems principles. 54–70. [12] Alex Forencich, Alex C. Snoeren, George Porter, and George Papen. 2020. Corundum: An Open-Source 100-Gbps NIC. In Proceedings of the 28th IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM). 38–46. doi:10.1109/FCCM48280.2020.00015 [13] Sylvain Girbal, Nicolas Vasilache, Cédric Bastoul, Albert Cohen, David Parello, Marc Sigler, and Olivier Temam. 2006. Semi-automatic composition of loop transformations for deep parallelism and memory hierarchies. In International Journal of Parallel Programming, Vol. 34. Springer, 261–317. [14] Toke Höiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. 2018. The eXpress Data Path: Fast Programmable Packet Processing in the Operating System Kernel. In CoNEXT. 54–66. [15] Yibo Huang, Zhenning Yang, Jiarong Xing, Yi Dai, Yiming Qiu, Dingming Wu, Fan Lai, and Ang Chen. 2024. Disaggregating Embedding Recommendation Systems with FlexEMR. arXiv preprint arXiv:2410.12794 (2024). [16] Stephen Ibanez, Alex Mallery, Serhat Arslan, Theo Jepsen, Muhammad Shahbaz, Changhoon Kim, and Nick McKeown. 2021. The nanoPU: A Nanosecond Network Stack for Datacenters. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 239–256. https://www.usenix.org/conference/osdi21/presentation/ibanez [17] Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate Foster, Changhoon Kim, and Ion Stoica. 2017. NetCache: Balancing Key-Value Stores with Fast In-Network Caching. In Proceedings of the 26th Symposium on Operating Systems Principles (Shanghai, China) (SOSP ’17). Association for Computing Machinery, New York, NY, USA, 121–136. doi:10.1145/3132747.3132764 [18] Anuj Kalia, Michael Kaminsky, and David Andersen. 2019. Datacenter RPCs can be General and Fast. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 1–16. https: //www.usenix.org/conference/nsdi19/presentation/kalia [19] Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA Efficiently for Key-Value Services. In Proceedings of the 2014 ACM Conference on SIGCOMM (Chicago, Illinois, USA) (SIGCOMM ’14). Association for Computing Machinery, New York, NY, USA, 295–306. doi:10.1145/2619239.2626299 7
Bojie Li
[44] Zixuan Wang, Xingda Wei, Jinyu Gu, Hongrui Xie, Rong Chen, and Haibo Chen. 2025. ODRP: On-Demand Remote Paging with Programmable RDMA. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, 1101–1115. [45] Xingda Wei, Rongxin Cheng, Yuhan Yang, Rong Chen, and Haibo Chen. 2023. Characterizing Off-path SmartNIC for Accelerating Distributed Systems. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 987–1004. https://www.usenix.org/ conference/osdi23/presentation/wei-smartnic [46] Xingda Wei, Zhiyuan Dong, Rong Chen, and Haibo Chen. 2018. Deconstructing RDMA-Enabled Distributed Transactions: Hybrid is Better (OSDI). [47] Zhuolong Yu, Yiwen Zhang, Vladimir Braverman, Mosharaf Chowdhury, and Xin Jin. 2020. NetLock: Fast, Centralized Lock Management Using Programmable Switches. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies,
Architectures, and Protocols for Computer Communication (Virtual Event, USA) (SIGCOMM ’20). Association for Computing Machinery, New York, NY, USA, 126–138. doi:10.1145/3387514.3405857 [48] Shengnan Yue, Mowei Wang, Yu Yan, Weiqiang Cheng, Zihan Jiang, and Zhenhui Zhang. 2025. RTT-or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving. In Proceedings of the 2nd Workshop on Networks for AI Computing. 5–7. [49] Ming Zhang, Yu Hua, Pengfei Zuo, and Lurong Liu. 2022. FORD: Fast One-sided RDMA-based Distributed Transactions for Disaggregated Persistent Memory. In 20th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association. [50] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In OSDI.
8