LatencyLab: A DPDK-Based P4 Pipeline Latency Measurement Framework for FPGA SmartNICs

LatencyLab: A DPDK-Based P4 Pipeline Latency Measurement Framework for FPGA SmartNICs P. Kuppili∗ , Z. Han‡ , Y. Qian‡ , S. Handagala‡ , M. Zink† , M. Leeser‡ , R. Ricci∗

arXiv:2609.39978v1 [cs.DC] 30 Sep 2026

∗ University of Utah

† University of Massachusetts Amherst

Abstract—P4-programmable FPGA SmartNICs place packet processing directly on the wire, but open FPGA P4 toolflows do not expose timestamping at the pipeline boundary, so the latency a P4 program adds on the target FPGA is rarely measured. This paper presents LatencyLab, a DPDK-based measurement framework for FPGA P4 pipeline latency that needs neither PHC/PTP support on the datapath nor clock synchronization. The FPGA’s two ports share a network segment, so the switch multicasts a copy of each probe packet to both: one copy passes through the VitisNetP4 pipeline, the other through a matched bypass path. A kernel-bypass DPDK receiver busy-polls both ports and timestamps every packet with the CPU timestamp counter (TSC) as it is retrieved from the NIC’s receive circular buffer. The arrival-time difference of the two copies isolates the pipeline latency after calibration against a null bitstream carrying the same traffic; transmit time cancel in the subtraction. We evaluate four VitisNetP4 programs on an AMD Alveo U280, probing each with a 20,000-packet trace measured ten times per session over five independent sessions, all TSC-timestamped and reflected for hardware timestamping. The measured latency distributions are tight and reproducible: 99% of packets fall within 20 ns of the median, session medians repeating within 1–2 ns (FiveTuple 107/137 ns, Forward 149 ns, RemoveHeader 177 ns, Checksum 364 ns at 250 MHz). Two independent checks agree with the framework: a kernel-free reflector returns every probe pair to a ConnectX-5 NIC whose adapter clock reproduces the measured distributions within a few nanoseconds, quantile by quantile, and every measured packet falls 18 to 22 clock cycles below the vendor’s worst-case latency bound. The framework also reveals behavior that averages hide: FiveTuple does not have a single latency; at each bring-up it settles into one of two values 31 ns apart, an effect traced to the clock of its lookup memory. Index Terms—Data Plane Development Kit (DPDK), differential measurement, FPGA SmartNICs, hardware timestamping, latency measurement, Open-NIC-Shell, P4, VitisNetP4

I. I NTRODUCTION Modern datacenters offload packet processing from hosts into programmable network interface cards (NICs) [1], and FPGA SmartNICs programmed in P4 bring that programmability all the way to the wire [2]–[4]. On the open side of this ecosystem, Open-NIC-Shell [5] with AMD VitisNetP4 [6] turns an Alveo U280 into a P4-programmable 100 GbE NIC through a standard toolflow [7], and a growing class of applications now runs computation inside that pipeline: sketchbased traffic monitoring maintains count-min counters in the data plane at 100 Gbps [8], and stateful network functions built from P4 and high-level synthesis (HLS) extract and aggregate flow information at line rate [9]. Both systems run on the platform measured in this paper, and both were validated for throughput, but the per-packet latency these pipelines add —

‡ Northeastern University

the other half of a real-time processing budget — could not be measured with existing tools. That number is rarely measured, because the measurement points do not exist. Direct one-way timing needs timestamps at the pipeline boundary and a synchronized clock, typically a disciplined PTP Hardware Clock (PHC) [10]. Datacenter switches and commodity NICs (cNICs) provide both [11], [12]. Open FPGA P4 datapaths provide neither. Previous studies highlight this gap through the compensatory structures they had to construct around it: OP4T needed a custom timestamping overlay because P4-to-FPGA toolflows expose no timestamping primitive at the pipeline boundary [13], and P4STA inserted a separate stamper FPGA into the path [14]. Passive FPGA measurement falls back to free-running local counters that no host clock can read [15], and NanoPU reached nanosecond timing only through custom CPU–NIC integration unavailable on off-the-shelf FPGA SmartNICs [16]. Timing signals are generally unavailable inside FPGA P4 data planes [17]–[19], and the platforms that do expose PHCs do so through proprietary stacks [20], [21] or without open P4 support [22]. The other alternative is to timestamp packets at the destination host, but existing measurement literature cautions against doing so. MoonGen measures latency only with NIC hardware timestamps because CPU-clock host stamps carry PCIe and OS-scheduling variance [23]. Where Has My Time Gone [24] and KV-Direct [25] show that host and PCIe interactions can outweigh the device latency under study. P4STA [14] and OSNT [26] exist because software timestamping cannot deliver reliable nanosecond accuracy. Our preliminary measurements on this U280/Open-NIC-Shell platform, reported in II-D, agree on both counts: return paths that traverse the receiver host bury the pipeline latency under PCIe and kernel effects, which are orders of magnitude larger (Sec. II-D), and kernel receive timestamps carry quantization artifacts of tens of nanoseconds, the same scale as the quantity being measured. This paper presents LatencyLab, a DPDK-based framework that measures the latency distribution of a P4 program on an FPGA-based SmartNIC with packet-level resolution, using only the open toolflow and commodity hardware. Let Lp4 denote the transit latency of the VitisNetP4 pipeline. LatencyLab estimates Lp4 differentially. The FPGA’s two ports sit on the same network segment, so the switch multicasts a copy of each probe packet to both; one copy is processed by the P4 pipeline while the other crosses a matched bypass path. A kernel-free DPDK receiver busy-polls both ports and timestamps each

packet with the CPU timestamp counter (TSC) at the NIC’s receive circular buffer, so the per-probe arrival-time difference contains Lp4 plus a constant path asymmetry. The transmit time and shared network path cancel in the subtraction, and no clocks on different devices need synchronization. The remaining asymmetry is measured using a null bitstream — the plain shell with no P4 pipeline on either path — and subtracted from the differential (Sec. III-E). Because LatencyLab uses host TSC timestamps rather than NIC hardware timestamps, we verify it against the reference the literature does trust, NIC hardware receive timestamping [23], [26]: every probe pair is reflected, without entering a kernel network stack, to a cNIC that timestamps arrivals in hardware, and its distributions are compared with LatencyLab’s. We also check every estimate against the vendor’s worst-case latency model [27]. Both agree with the framework (Sec. V-A). The verification is needed once. A deployed LatencyLab requires no PHC, no PTP, and no hardwaretimestamping NIC, so it applies where such support is missing. The contributions of this paper are: • LatencyLab, a DPDK-based differential framework for FPGA P4 pipeline latency requiring no datapath timing support (Secs. III, IV). • Characterization of the latency distributions of four VitisNetP4 programs — reproducible to 1–2 ns across independent hardware bring-ups (Sec. V). • Verification of the framework against an independent NIC hardware clock and the vendor worst-case model (Sec. V-A). • A case study enabled by LatencyLab’s packet-level resolution: FiveTuple exhibits two observed session-level modes approximately 31 ns apart, consistent with its separately clocked content-addressable memory (CAM) (Sec. V-B). II. BACKGROUND This section gives the background for LatencyLab: the platform it targets, the P4 programs under test, and the preliminary measurements that informed its design. A. Testbed All measurements run on the Open Cloud Testbed (OCT) [28], which provides bare-metal servers whose FPGA NIC ports are directly connected to the datacenter switch fabric [29]. Our slice consists of two servers on a top-ofrack (TOR) switch as shown in Fig. 1a. The sender node has one 100 GbE NVIDIA ConnectX-5 cNIC connected to the TOR. The receiver node hosts an AMD Alveo U280 [29] with two 100 GbE connections directly to the TOR switch using QSFP28 transceivers and one 100 GbE NVIDIA ConnectX5 cNIC. The sender replays probe traffic from its cNIC; the receiver hosts the FPGA under test. The receiver uses a dual-socket Intel Xeon Gold 6226R (2.90 GHz, 16 cores per socket) host running Ubuntu 22.04 LTS (kernel 5.15.0-177); we use Open-NIC-Shell (commit

a33fb0b) [5], open-nic-dpdk with Xilinx dma ip drivers (commit 7859957) [30], DPDK 20.11, VitisNetP4 2023.2, and Vivado 2023.2. Both DPDK polling cores are pinned to cores 4 and 6 on the same processor socket (NUMA node 0). B. Why Open-NIC-Shell We build on Open-NIC-Shell [5] for two reasons. First, it is the de facto open-source NIC shell for AMD FPGAs and the standard path for deploying P4 on shared research testbeds such as OCT [7], which is where the pipeline-cost question arises in practice. Second, and essential to the framework, its QDMA subsystem has an open DPDK poll-mode driver, which allows LatencyLab to timestamp packets as they are retrieved from the NIC’s receive circular buffer with no kernel involvement (Sec. III-C). As shown in Fig. 2, Open-NIC-Shell exposes the two 100 GbE ports of the FPGA as two PCIe physical-function interfaces on the host. One port’s receive path carries the VitisNetP4 IP (interface P Fvitis ); the other carries a passthrough plugin (P Fbypass ). Everything else about the two paths is common: the same 250 MHz user-logic clock, the same QDMA subsystem, the same PCIe Gen3 x16 link. For measurement purposes this is the property that matters: the processing paths are architecturally identical apart from the VitisNetP4 pipeline itself. Consequently, any components common to both paths effectively cancel out in a differential comparison. C. P4 Programs Under Test We evaluate four programs derived from the AMD VitisNetP4 example designs [31], built with Vivado [32]. FiveTuple extracts the 5-tuple and matches it in a content-addressable memory (CAM) before forwarding; its lookup engine is generated with a 300 MHz CAM clock alongside the 250 MHz packet-pipeline clock, a detail that matters in Sec. V-B. Forward parses Ethernet/VLAN/IPv4/IPv6/TCP/UDP and forwards on an IPv4 destination match. RemoveHeader strips the VLAN header and forwards the shortened packet. Checksum parses the same headers as Forward and additionally computes and matches the IPv4 header checksum in the data plane; it is the deepest pipeline of the four. A fifth bitstream, the null, is the plain shell with passthrough/bypass plugins on both ports and no P4 pipeline anywhere; it serves as the calibration reference in Sec. III-E. D. What Host-Path Timing Shows: PCIe and Kernel Dominate Before building the framework, we measured what hostvisible timing can actually resolve, using an earlier arrangement of the same testbed in which the primary node hosts both the cNIC and the FPGA, as shown in Fig. 1b. Probe packets are replayed with tcpreplay from the primary node’s cNIC and looped back to the same node’s FPGA ports, where tcpdump records software timestamps in the host; the loop is closed in one of two ways. In the first, a secondary node connected over the TOR switch reflects each packet in its AMD Alveo U280 in hardware, so the packet never touches a host and the round trip includes two PCIe crossings, both on the primary node.

(a) The LatencyLab measurement framework.

(b) Preliminary host-timed loopback study.

Fig. 1: Measurement setups on the same OCT slice. the work presented in this paper. III. T HE L ATENCY L AB F RAMEWORK

Fig. 2: Open-NIC-Shell datapaths as used by LatencyLab.

Sec. II-D leaves two requirements: cancel everything the two paths share, and timestamp packets before the kernel can distort them. LatencyLab meets the first with a differential design, where the switch delivers a copy of each probe to both FPGA ports and the paired arrival times are subtracted. The second is addressed with a DPDK receiver that timestamps every packet with the CPU timestamp counter (TSC). One term survives the subtraction, the constant asymmetry between the two ports themselves; a null bitstream with no P4 pipeline on either path measures it directly, so it is measured and removed rather than assumed negligible. A. Overview

Fig. 3: Raw RTT of the two loopback paths; the sub-µs pipeline latency is invisible in both. In the second, the secondary node reflects packets in software through a VLAN bridge, which adds another traversal of its PCIe link and kernel network stack, for a total of four PCIe crossings. Fig. 3 compares the raw RTT distributions of the two paths. The result confirms the findings in [24], [25]: both RTT distributions operate at the scale of microseconds, the two added host crossings visibly inflate both the median and the spread, and the approximately 0.1 - 0.4 µs pipeline contribution is unresolved within either distribution. This result motivates

Fig. 1a shows the measurement framework design. The sender replays a trace of probe packets, each carrying a unique ID in its payload. Both FPGA ports are connected to the same network segment, so the switch multicasts a copy of each probe to each FPGA port. One copy is processed by the VitisNetP4 pipeline on its way to P Fvitis ; the other crosses the passthrough plugin to P Fbypass . A DPDK application on the receiver polls both ports and timestamps every arriving packet, and the ID ties the two copies of a probe back together. The copies share the transmit and upstream network path but enter distinct FPGA ports and QDMA queues, their arrival-time difference removes the shared terms and retains the pipeline latency plus a constant port-and-queue asymmetry. The transmit time and the shared path drop out in the subtraction, which is why no clocks on different devices need to be synchronized. The rest of this section fills in each stage. B. Sender and Probe Traffic Probe traffic for each program is taken directly from its AMD VitisNetP4 example design [31] and built into a 20,000packet trace, with a unique ID written into every payload. The sender replays the trace with tcpreplay at 500 packets per second, one probe every 2 ms, and this rate is a deliberate choice rather than a limitation. The quantity we want is transit latency. At any rate where probes can queue behind each other,

in the switch, in the pipeline, in the DMA engines, or at the poller, the measurement would also include queueing delay, which depends on load rather than on the program. Spacing probes 2 ms apart makes each probe an isolated event through the entire system. The slow rate also protects timestamp quality on the receiver: each copy arrives in its own DMA burst, so the poller handles it alone and stamps it immediately, in the same way on both ports. Higher offered loads would characterize platform queueing rather than isolated pipeline transit latency; we leave that distinct question to future work. C. Kernel-Free Receiver The receiver is a DPDK application on Open-NICShell’s [30] QDMA poll-mode driver, with the FPGA’s two functions bound to vfio-pci. One dedicated core polls each port in a busy loop; no interrupt, kernel stack, or system call touches the packet path. For each received packet the poller reads the TSC using rte_rdtsc_precise() immediately after rte_eth_rx_burst() dequeues the packet from the NIC receive ring, and then parses the embedded ID. Both pollers run on cores of the same processor socket and read hardware-synchronized invariant TSCs, so their timestamps require no cross-device clock conversion. TSC cycles are converted to nanoseconds using the TSC frequency reported by DPDK’s rte_get_tsc_hz(),(2.89 GHz on this host); the CPU’s constant_tsc/nonstop_tsc capability guarantees the TSC ticks at this fixed rate regardless of core frequency scaling. Any fixed intercore offset is included in the calibrated asymmetry δ. D. Reflect Framework for Verification Since LatencyLab’s TSC timestamps constitute a customdesigned timing mechanism, we validate them against NIC hardware receive timestamps [23], [26], which are used by prior measurement systems as a reference for nanosecondscale latency. Any NIC that can hardware-timestamp arbitrary received frames can serve as this reference; we used the ConnectX-5 already present in the sender. The challenge is delivering the packets to that NIC without them passing through any kernel network stack, since that would reintroduce the very timing noise of Sec. II-D; the reflect path shown in Fig. 1a solves this. Immediately after a packet is timestamped, the same DPDK process returns it through the receiver’s cNIC (attached through the bifurcated mlx5 driver) toward the sender’s cNIC, which timestamps every arriving frame against its own adapter clock (HWTSTAMP_FILTER_ALL with nanosecond precision). The reflect path must preserve two things about each pair: which copy is which, and their timing comparability. A one-byte tag in the payload handles the first, marking each frame as the VitisNetP4 or the bypass copy, since both arrive at the same NIC; a fixed TX queue per path handles the second, turning any queue-to-queue difference into a constant per-path offset that the analysis absorbs, rather than per-packet noise that would blur the distributions being compared. The adapter-clock arrival difference of a reflected pair re-measures the differential measured by LatencyLab, plus

noise from the reflection channel itself; Sec. IV-B turns this into a precise consistency test. The reflect framework exists only for verification. A deployed LatencyLab does not require this component, nor any hardware-timestamping NIC. E. Null Calibration One term survives the subtraction: the two ports are distinct physical ports with distinct QDMA queues, so their delivery paths differ by a small constant. Modeling this asymmetry would mean trusting assumptions about port internals, so LatencyLab measures it instead. The null bitstream (Sec. II-C) makes both paths passthrough while changing nothing else. Replaying the same probe trace using the null bitstream therefore yields the port-and-queue asymmetry for that trace, whose median is subtracted from the program differential. The measured offsets are small and stable, 14 to 28 ns depending on the trace, with a session-to-session spread under 7 ns. Without this step, every reported latency would carry an unknown bias of that magnitude. IV. M EASUREMENT M ODEL This section defines LatencyLab’s measurement protocol and the quantity it computes. Measurements are organized into sessions. A session is one complete hardware instantiation (bitstream flash, PCIe re-enumeration, driver and table initialization, DPDK start) followed by ten replay runs, and we collect five independent sessions per program, with each session observing P Fvitis and P Fbypass concurrently. Each instantiation puts the hardware into a fresh state: a property that is set at reset and then held until the next reset stays constant within a session but can differ across sessions. Collecting five independent sessions therefore separates resettime effects from per-packet behavior, a choice that is justified in Sec. V-B. In total, each program is measured with 20,000 probes × 10 runs × 5 sessions per program, with both paths observing the same probes. Every probe is TSC-timestamped on both ports and reflected for hardware timestamping, so both clocks observe the same packets. We repeat the same session protocol with the null bitstream for calibration. A. Differential Estimator Consider probe i, transmitted at time T (i), whose two copies arrive at the receiver. Writing S(i) for the delay shared by both copies (wire, switch, and everything else upstream of the split), the two TSC arrival stamps are: tP (i) = T (i) + S(i) + Lp4 (i) + dP + eP (i),

(1)

tB (i) = T (i) + S(i) + dB + eB (i),

(2)

where dP , dB are the constant per-path delivery components (port, queue, DMA engine) and eP , eB run-varying measurement error caused by polling phase and PCIe/DMA timing. Their difference, D(i) = tP (i) − tB (i) = Lp4 (i) + δ + ε(i),

(3)

retains only the pipeline latency, the constant asymmetry δ = dP − dB , and noise ε = eP − eB . The transmit time and the shared path components are eliminated. Eq. (3) still contains the port asymmetry δ, and the null bitstream supplies the equation that removes it. With no P4 pipeline on either path, the same measurement gives: D0 (i) = δ + ε0 (i).

(4)

Subtracting Eq. (4) from Eq. (3) cancels δ and leaves Lp4 plus noise. In practice the subtraction uses δ̂, the median nullbitstream differential for the corresponding traffic trace D0 , and the measurement error is mitigated through aggregation across ten runs per session: the latency of probe i repeats in every run while the noise does not, so the median preserves the recurring packet-dependent component while reducing the influence of run-specific timing variation. We use the median rather than the mean because one atypical polling, DMA, or PCIe delay can substantially shift the mean, whereas the median of ten repeats is robust to such timing outliers. The estimate is: Fig. 4: Pipeline latency of the four programs. L̂p4 (i) = med Dr (i) − δ̂. r=1..10

(5)

Sec. V reports, for each program, the distribution of L̂p4 (i) over all probes and sessions. B. Verifying the Framework We perform two independent evaluations to evaluate the differential estimator in Eq. (5) and present the results in Sec. V-A. 1) Hardware-clock check: The reflect framework (Sec. III-D) re-times each probe pair in the sender cNIC’s adapter clock, observing R(i) = D(i) + C(i) where C(i) is reflection-channel noise generated by devices and clocks disjoint from LatencyLab. If LatencyLab measures D correctly and C is independent of it, the distribution of R must equal the convolution of the measured D distribution with that of the paired residuals C(i) = R(i) − D(i); randomly re-pairing residuals with differentials synthesizes this convolution, and comparing its quantiles against the directly measured R tests the framework at packet-pair granularity with no shared clock or timestamping mechanism. 2) Vendor-model check: VitisNetP4 reports a worst-case calculated latency for each generated pipeline, extended by a traffic-dependent penalty only when the program inserts headers [6]; none of our four programs does, so the bound is the calculated latency itself: 53, 55, 66, and 109 cycles at 250 MHz for FiveTuple, Forward, RemoveHeader, and Checksum, respectively. Since this constitutes an upper bound rather than a deterministic prediction, consistency with the model requires every retained L̂p4 (i) to remain below the corresponding threshold, when the margin exhibits uniform behavior across programs — consistent with conservative bound properties — independent of program scale.

V. R ESULTS AND A NALYSIS A. Latency of the Four Programs Fig. 4 and Table I show the main results of the paper. For each program, the figure places three things side by side: the latency measured by LatencyLab, the same probes remeasured by the independent hardware clock, and the vendor worst-case bound. All numbers are per-probe estimates: each probe is measured ten times, and Eq. (5) takes the median to remove single-measurement noise. The three sources tell one consistent story. The LatencyLab distributions are narrow (p5–p99 spans of 31–36 ns, p99 within 22 ns of the median), which is what a feed-forward pipeline should produce; the residual spread is remaining measurement noise plus genuine parsing differences across packets in a trace. The hardware clock, which sees the same differential through the noisier reflection channel, results in very similar distributions but with a wider percentile span. Every distribution sits well below its vendor bound. The programs separate cleanly and order by pipeline depth, from FiveTuple (26.7 cycles in its low state) up to Checksum (91 cycles). The results are reproducible: across five full hardware re-initializations per program, raw session medians stay within a 1–2 ns band (165–167 ns for Forward, 386–390 ns for Checksum). The agreement between the two clocks in Fig. 4 is quantitative, not just visual. Fig. 5 shows the convolution check of Sec. IV-B for Forward: the measured and predicted distributions agree within 1 ns at p50 and p90 and within 4 ns at p25– p99, and across all four programs the agreement holds within 8 ns at every quantile from p25 through p95 (the HW column of Table I). Where the reflection channel preserves packet size (Forward), the centers also agree, to 9 ns; programs that alter packet length (RemoveHeader, FiveTuple) shift the reflected

TABLE I: Per-program P4 pipeline latency P4 Program FiveTuple (low) FiveTuple (high) Forward RemoveHeader Checksum

p5 92 124 138 157 349

p25 101 131 145 171 358

LatencyLab (ns) p50 p75 107 112 137 143 149 154 177 181 364 370

p95 120 152 163 188 377

p99 127 159 169 192 381

Fig. 5: Measured hardware-clock distribution vs. the convolution predicted from LatencyLab’s measurements (Forward, Sec. IV-B). copies’ serialization by a program-dependent constant, which is why the hardware-clock results in Fig. 4 and Table I are center-aligned. The hardware clock therefore validates the distribution shape for all programs, and the absolute center when the reflection path preserves packet size. Every per-probe estimate lies below its program’s bound, with one systematic exception: the first probe after an idle gap arrives late by tens of nanoseconds. This is not a pipeline effect. It affects only packet one of a run, appears identically for every program regardless of pipeline depth, and the feedforward pipeline holds no state that could warm up; the receive path performs its first DMA activity after the inter-run idle interval. After excluding these startup probes, all retained estimates remain below their bounds, and even p99 leaves 12– 18 cycles of margin. The headroom at the median is nearly constant, 18.7, 17.7, 21.8, and 18.0 cycles for FiveTuple’s high mode, Forward, RemoveHeader, and Checksum, across pipelines whose depths differ by 2.5×. One program does not have a single latency, and we examine it next. B. Special Case: FiveTuple’s Two Latency States FiveTuple’s five sessions do not result in similar latencies. Four sessions (2–5) sit at 105–107 ns while session 1 sits at 137 ns, a step of about 31 ns (7.7 cycles) with no intermediate

p5 75 111 114 128 332

Hardware validation (ns) p25 p50 p75 p95 94 107 116 133 125 137 156 182 134 149 162 185 160 177 199 224 351 364 381 399

p99 151 204 207 241 412

Vendor bound (ns) (cycles) 212 53 212 53 220 55 264 66 436 109

Fig. 6: FiveTuple raw session medians as measured by LatencyLab (right) and, by the hardware clock of the reflect framework (left). values and no within-session transition across the ten runs, as shown in Fig. 6. The hardware clock of the reflect framework saw the same split from entirely different hardware on the same day: with the reflection channel’s constant removed (Sec. V-A), its low sessions sit at 104–110 ns and session 1 at 126 ns. The state is real, and it is chosen at instantiation: the latency is decided once, when the FPGA comes up, and does not change again until the next flash and reset. The likely reason is the CAM’s separate clock. FiveTuple is the only program of the four whose lookup engine runs on its own clock: VitisNetP4 instantiates the CAM at 300 MHz alongside the 250 MHz packet-pipeline clock [6]. A signal crossing between two clock domains costs a number of cycles that depends on how the two clocks’ edges align [33], and on this platform both clocks derive from the same on-chip clock generator, so their alignment is set when the FPGA comes up and holds until the next reset. This mechanism predicts the qualitative behavior we observe: a latency chosen per bringup, quantized in cycles, constant within a session, and absent from the three single-clock programs. The P4 control flow also applies the table from two call sites, so the crossing penalty is paid twice per packet.

This case study shows what the framework buys. A hosttimed measurement would drown a 31 ns step in PCIe and kernel spread (Sec. II-D). A single-session measurement could observe only one mode and miss the reset dependence. A measurement averaged across sessions could report a value between the observed modes that the hardware never exhibits. Resolving the behavior took packet-level differentials, and a session-structured protocol. The practical takeaway is that a FiveTuple-class pipeline has at least two stable per-reset latencies in the measured sessions, which matters for latency budgeting. VI. C ONCLUSION This paper presented LatencyLab, a DPDK-based framework that measures the latency a P4 program adds on an FPGA SmartNIC, with packet-level resolution and without the timing support such platforms lack: no datapath PHC, no PTP synchronization between devices, and no kernel in the timestamp path. The design uses what the platform already provides. The shell’s two host-visible ports receive copies of the same probe, which turns the problem into an arrival-time differential in which everything shared cancels. The shell’s DPDK support lets one process timestamp both ports at the NIC’s receive circular buffer with a single invariant clock. A null bitstream measures and removes the constant port-andqueue offset that the differential cannot cancel. On an AMD Alveo U280, we evaluate four VitisNetP4 programs using 20,000 probes, each measured ten times per session across five independent sessions. For each reported program or FiveTuple latency mode, the 99th-percentile latency is at most 22 ns above the median—meaning that 99% of the per-probe latency estimates are no greater than the median plus 22 ns. Session medians are reproducible to 1–2,ns across bring-ups for the single-clock programs, while FiveTuple exhibits two reset-dependent modes. Two independent checks agree with the framework: a NIC hardware clock re-timed every probe pair through the kernel-free reflect framework and reproduced the measured distributions, and the vendor’s worstcase model bounds every estimate with similar headroom across all four programs. The FiveTuple case study shows what this resolution is worth. The program has two latency modes, 31 ns apart, consistent with the relative alignment of its CAM and packet-pipeline clock domains. Future work includes characterizing queueing onset at high offered load, probing the FiveTuple state ladder across many resets, and porting to other Open-NIC-Shell-class boards. Our bitstreams, measurement applications, and dataset are publicly available in the LatencyLab GitHub repository at https://github.com/OCT-FPGA/LatencyLab. VII. ACKNOWLEDGEMENT This work was funded by National Science Foundation (NSF) grants 2319962, 2130907, 2130891, 2027208, and 2431419. All opinions and statements are those of the authors and do not represent the position of the NSF. The authors

thank the Open Cloud Testbed team for maintaining the FPGA infrastructure on which all measurements were performed. R EFERENCES [1] M. E. Fiuczynski, R. P. Martin, T. Owa, and B. N. Bershad, “SPINE: a safe programmable and integrated network environment,” in Proceedings of the 8th ACM SIGOPS European Workshop on Support for Composing Distributed Applications, ser. EW 8. New York, NY, USA: Association for Computing Machinery, 1998, p. 7–12. [Online]. Available: https://doi.org/10.1145/319195.319197 [2] P. Benáček, V. Puš, and H. Kubátová, “P4-to-VHDL: automatic generation of 100 gbps packet parsers,” in 2016 IEEE 24th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2016, pp. 148–155. [3] H. Wang, R. Soulé, H. T. Dang, K. S. Lee, V. Shrivastav, N. Foster, and H. Weatherspoon, “P4FPGA: a rapid prototyping framework for P4,” in Proceedings of the Symposium on SDN Research, ser. SOSR ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 122–135. [Online]. Available: https://doi.org/10.1145/3050220.3050234 [4] S. Ibanez, G. Brebner, N. McKeown, and N. Zilberman, “The P4→NetFPGA workflow for line-rate packet processing,” in Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ser. FPGA ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 1–9. [Online]. Available: https://doi.org/10.1145/3289602.3293924 [5] AMD, “GitHub - Xilinx/open-nic-shell: AMD OpenNIC Shell includes the HDL source files — github.com,” https://github.com/{X}ilinx/ open-nic-shell, [Accessed 13-07-2026]. [6] AMD, “AMD Technical Information Portal — docs.amd.com,” https://docs.amd.com/r/en-{U}{S}/ug1308-vitis-p4-user-guide/{I} ntroduction, [Accessed 13-07-2026]. [7] Z. Han, S. Handagala, K. Patle, M. Zink, and M. Leeser, “A framework to enable runtime programmable P4-enabled FPGAs in the Open Cloud Testbed,” in IEEE INFOCOM 2023 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2023, pp. 1–6. [8] Z. Han, Y. Qian, M. Zink, and M. Leeser, “Memory-efficient sketch acceleration for handling large network flows on FPGAs,” in 2024 International Conference on Field Programmable Technology (ICFPT), 2024, pp. 1–9. [9] Z. Han, A. Briasco-Stewart, M. Zink, and M. Leeser, “Extracting TCPIP headers at high speed for the anonymized network traffic graph challenge,” in 2024 IEEE High Performance Extreme Computing Conference (HPEC), 2024, pp. 1–6. [10] T. Martı́nek, M. Campanella, F. Pederzolli, J. Hill, M. Dimolianis, and D. Parniewicz, “White Paper: Timestamping and Clock Synchronisation in P4-Programmable Platforms,” https://resources. geant.org/wp-content/uploads/2022/09/GN4-3 White-Paper Timestamping-and-Clock-Synchronisation-in-P4-Programmable-Platforms. pdf, 2022. [11] P. G. Kannan, R. Joshi, and M. C. Chan, “Precise time-synchronization in the data-plane using programmable switching ASICs,” in Proceedings of the 2019 ACM Symposium on SDN Research, ser. SOSR ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 8–20. [Online]. Available: https://doi.org/10.1145/3314148.3314353 [12] P. Kuppili, A. Maricq, B. E. Stephens, R. Stutsman, and R. Ricci, “Poster: Topocloud: Getting datacenter network experiments into the right shape,” in 2024 IEEE 32nd International Conference on Network Protocols (ICNP), 2024, pp. 1–2. [13] M. Hawari and T. Clausen, “OP4T: bringing advanced network packet timestamping into the field,” in 2021 International Conference on Information Networking (ICOIN), 2021, pp. 137–142. [14] R. Kundel, F. Siegmund, J. Blendin, A. Rizk, and B. Koldehofe, “P4STA: high performance packet timestamping with programmable packet processors,” in NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium, 2020, pp. 1–9. [15] G. Korcyl and Z. Duliński, “Passive latency measurement with nanosecond precision for time-critical applications,” Comput. Netw., vol. 279, no. C, Jun. 2026. [Online]. Available: https://doi.org/10.1016/ j.comnet.2026.112138 [16] S. Ibanez, A. Mallery, S. Arslan, T. Jepsen, M. Shahbaz, C. Kim, and N. McKeown, “The nanoPU: a nanosecond network stack for datacenters,” in 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, Jul. 2021,

pp. 239–256. [Online]. Available: https://www.usenix.org/conference/ osdi21/presentation/ibanez [17] S. Bal, Z. Han, S. Handagala, M. Cevik, M. Zink, and M. Leeser, “P4based in-network telemetry for FPGAs in the open cloud testbed and FABRIC,” in IEEE INFOCOM 2024 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2024, pp. 1–6. [18] “URGENT: Existing support for linuxptp on open-nic driver · Issue #56 · Xilinx/open-nic-driver — github.com,” https://github.com/{X}ilinx/ open-nic-driver/issues/56, [Accessed 13-07-2026]. [19] “AMD Technical Information Portal — docs.amd.com,” https://docs.amd.com/r/en-{U}{S}/ug1308-vitis-p4-user-guide/{S} tandard-{M}etadata, [Accessed 13-07-2026]. [20] “P4 - Silicom Denmark — silicom.dk,” https://www.silicom.dk/ product-category/functional-ip/p4/, [Accessed 13-07-2026]. [21] “OpenNIC FPGA SmartNIC Shell on iW-Fibre Platforms — iwave-global.com,” https://iwave-global.com/articles/ opennic-fpga-smartnic-shell-on-iw-fibre-platforms/, [Accessed 1307-2026]. [22] “Support for Alveo V80 (xcv80-lsva4737-2MHP-e-S) · Issue #72 · Xilinx/open-nic-shell — github.com,” https://github.com/{X}ilinx/ open-nic-shell/issues/72, [Accessed 13-07-2026]. [23] P. Emmerich, S. Gallenmüller, D. Raumer, F. Wohlfart, and G. Carle, “Moongen: A scriptable high-speed packet generator,” in Proceedings of the 2015 Internet Measurement Conference, ser. IMC ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 275–287. [Online]. Available: https://doi.org/10.1145/2815675.2815692 [24] N. Zilberman, M. Grosvenor, D. Popescu, N. Manihatty Bojan, G. Antichi, M. Wojcik, and A. Moore, “Where has my time gone?” in Passive and Active Measurement Conference (PAM). Springer, 2017. [Online]. Available: https://www.repository.cam.ac.uk/ handle/1810/263038 [25] B. Li, Z. Ruan, W. Xiao, Y. Lu, Y. Xiong, A. Putnam, E. Chen, and L. Zhang, “KV-Direct: high-performance in-memory key-value store with programmable NIC,” in Proceedings of the 26th Symposium on Operating Systems Principles, ser. SOSP ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 137–152. [Online]. Available: https://doi.org/10.1145/3132747.3132756 [26] G. Antichi, M. Shahbaz, Y. Geng, N. Zilberman, A. Covington, M. Bruyere, N. Mckeown, N. Feamster, B. Felderman, M. Blott, A. W. Moore, and P. Owezarski, “OSNT: open source network tester,” IEEE Network, vol. 28, no. 5, pp. 6–12, 2014. [27] AMD, “AMD Technical Information Portal — docs.amd.com,” https: //docs.amd.com/r/en-{U}{S}/ug1308-vitis-p4-user-guide/{L}atency, [Accessed 13-07-2026]. [28] M. Zink, D. Irwin, E. Cecchet, H. Saplakoglu, O. Krieger, M. Herbordt, M. Daitzman, P. Desnoyers, M. Leeser, and S. Handagala, “The Open Cloud Testbed (OCT): A platform for research into new cloud technologies,” in 2021 IEEE 10th International Conference on Cloud Networking (CloudNet), 2021, pp. 140–147. [29] M. Leeser, S. Handagala, and M. Zink, “FPGAs in the cloud,” Computing in Science & Engineering, vol. 23, no. 6, pp. 72–76, 2021. [30] AMD, “GitHub - Xilinx/open-nic-dpdk: DPDK Drivers for AMD OpenNIC — github.com,” https://github.com/{X}ilinx/open-nic-dpdk, [Accessed 13-07-2026]. [31] AMD, “AMD Technical Information Portal — docs.amd.com,” https://docs.amd.com/r/en-{U}{S}/ug1308-vitis-p4-user-guide/{E} xample-{D}esigns, [Accessed 13-07-2026]. [32] AMD, “AMD Technical Information Portal — docs.amd.com,” https://docs.amd.com/r/en-{U}{S}/ug910-vivado-getting-started/{V} ivado-{D}esign-{S}uite-{O}verview, [Accessed 13-07-2026]. [33] C. E. Cummings, “Clock domain crossing (CDC) design & verification techniques using systemverilog,” SNUG-2008, Boston, 2008.

Related documents

• LatencyLab: A DPDK-Based P4 Pipeline Latency Measurement Framework for FPGA SmartNICs#1145145
• Measuring Speed Using a Computer--Several Techniques.#333125
• Catscan: Visualizing Pipelines of CPU Performance Simulation#1157469
• Hardware Synthesis of Synchronous Data Flow Models#640219
• A Design Space Exploration of Async/Await#659237