arXiv CSopen access

Beyond Thread States: Diagnosing Performance Degradation with eBPF and Thread Dynamics

Beyond Thread States: Diagnosing Performance Degradation with eBPF and Thread Dynamics Diogo Landau ∗ , Jorge G. Barbosa † , Nishant Saurabh ∗ ∗ Department of Information and Computing Sciences, Utrecht University, NL † LIACC, Faculdade de Engenharia, Universidade do Porto, PT

I. I NTRODUCTION The modern internet is often powered by a vast network of interconnected applications (e.g., online data-intensive applications [1], [2] such as databases, message brokers, and machine learning solutions), providing data, services, and functionality to connecting processes [2]. Load variability and interference can degrade these applications’ Quality of Service (QoS), negatively affecting dependent applications and enduser experience [3]–[6]. Restoring QoS requires diagnosing the cause of degradation to apply suitable mitigation [7]. This necessitates a solution capable of observing an application’s performance by collecting appropriate metrics to understand its state. An established approach to performance investigation is Thread State Analysis (TSA) [8], which quantifies the time each application thread spends in six distinct states (Executing, Runnable, Swapping, Sleeping, Lock, and Idle). TSA provides valuable insight into the subsystems where application threads spend time and helps identify performance bottlenecks within those subsystems (e.g., CPU contention, I/O wait, or synchronization delays). However, TSA alone cannot reveal the interactions among threads that collectively characterize an application’s performance. This limitation arises because

TSA collects metrics at a per-thread granularity, making it opaque to the specific system resources accessed by threads and, consequently, to the thread dynamics that emerge from these interactions. These thread dynamics [9] are essential for uncovering common performance degradation patterns related to specific system resources. Figure 1a illustrates this through a database experiencing degraded performance caused by the serialization of read and write operations on a shared row. TSA results in Figure 1b indicate that the increased time spent by thread t4 waiting on locks strongly correlates with the observed performance degradation. However, the thread dynamics graph in Figure 1c further reveals that the only other thread interacting with t4 is t3, through a shared lock. This interaction highlights that the degraded performance originates from thread t3, which is processing a sudden surge of write operations. 2.0

Wait Time (s/s)

Abstract—Online Data-Intensive applications face performance degradation from load variability and resource interference. While Thread State Analysis (TSA) based approaches enable identifying constrained subsystems, they lack the granularity to reveal the inter-thread dependencies that propagate degradation. In this paper, we present an application-agnostic performance degradation analysis method that extends TSA by capturing fine-grained thread dynamics. We implemented 16 eBPF-based metrics across six kernel subsystems, including scheduling, VFS, networking, futex, multiplexing IO, and block IO which enables tracing thread interactions with specific resources like futexes, sockets, and disks. Our method leverages the fact that performance degradation propagates along inter-thread dependencies, and a subset of thread-resource interactions can enable capturing common degradation patterns. To this end, we employ a selective thread tracking algorithm that traces performance issues from entry-point threads to constrained resources. Experimentation with diverse applications under variable workloads and resource contention shows our method successfully diagnoses CPU, disk, lock, and external service contention with minimal overhead, while also revealing internal application constraints. Index Terms—eBPF Instrumentation, Performance Degradation, Performance Interference, Thread State Analysis

Response Time (ms)

arXiv:2605.25298v1 [cs.DC] 24 May 2026

[email protected]∗ , [email protected]† , [email protected]∗

1.5 1.0 0.5 60

80

100

Relative Time (s) (a)

120

1.00

t4 t3

0.75 0.50

t3

f1

0.25 0.00 60

80

100

120

Relative Time (s) (b)

t4

(c)

Fig. 1: Illustration of a database experiencing lock contention, leading to degraded performance: (a) 95th percentile response time; (b) Total time database threads spend waiting for locks; (c) Visualization of database thread dynamics between revealing threads t3 and t4 interacting via a lock f1.

Traditionally, performance analysis approaches have relied on system-level metrics exposed by the operating system (e.g., the Linux /proc filesystem) to infer resource constraints and performance bottlenecks [10]–[17]. While such metrics provide valuable insights, they often lack the granularity needed to capture an application’s dynamics. Recent advances in the Linux kernel, particularly the development of the extended Berkeley Packet Filter (eBPF) subsystem, offer new opportunities to overcome these limitations. Building on eBPF, emerging approaches [18], [19] have begun to collect detailed thread state metrics that provide richer context for diagnosing performance degradation. However, these methods still fall short in capturing the finegrained interactions necessary to fully uncover an application’s

thread dynamics. A notable effort in this direction is presented by [20], which defines a set of metrics for identifying lockrelated dynamics in NoSQL applications. While promising, this approach remains domain-specific and focuses solely on sychronization behaviour. In contrast, our goal is to generalize this capability to encompass a broader range of system resources and application types. This paper introduces a set of TSA metrics that, through carefully tuned granularity, also expose the thread dynamics that lead to an application’s experienced degraded performance. To achieve this, we implement 16 eBPF-based metrics across six kernel subsystems, including scheduling, virtual file system (VFS), networking, futex, multiplexing IO, and block IO. These metrics enhance observability by tracking a thread’s use of specific kernel resources, revealing common degradation scenarios like lock, disk, CPU, and external service dependency contention. Furthermore, we employ selective thread tracking to analyse application degradation under the lens of its thread dynamics, to reveal how request-facing threads are contended by resource-constrained threads within an application. To validate our approach, we evaluated our method on an extensive set of online data-intensive applications, offering variability in application domains, threading models, interprocess communication (IPC) methods, and user-space virtual machines. Our experiments considers scenarios where applications degrade under a combination of variable workload patterns and resource types. Our results show that the collected metrics successfully highlight the constrained resources and the thread dynamics that led to an application’s degraded state. In the presence of lock contention scenarios, the metrics enabled identifying the threads and futex resources hindering application response time. For disk contention, they revealed the application’s interthread communication model, linking server response time to disk performance. In CPU-constrained cases, the implemented metrics highlighted impaired threads, identifying runqueue activity as the primary cause of degraded performance. Finally, when degradation originated from an external service dependency, metrics revealed the external service impairing performance, indicating the issue was outside the application under analysis. We have so far analyzed how performance degradation manifests at the thread level; however, we aim to extend diagnosability by integrating system-level information to obtain a more comprehensive view of an application’s performance, which we plan to address in our future work. The complete source code and reproducibility artifact is released opensource in an online GitHub repository [21].

virtualized resources or the underlying hardware. One such method is sampling, which collects system-level metrics at periodic intervals to derive statistics encoding an application’s activity [22], [23]. Another method is binary instrumentation [20], [24], [25], which dynamically inserts probes in the application to collect custom performance statistics. SystemTap [26], LTTng [27] and Dtrace [28] are early solutions which dynamically instrument kernel-level code through dedicated kernel modules. However, their unrestricted kernel memory access frequently triggered kernel panics, rendering them unsuitable for production environments. Nevertheless, Extended Berkeley Packet Filter (eBPF) [29] addressed this issue by implementing and utilizing a kernel virtual machine which restricts ebpf program’s memory access through specialised helper functions, while interpreting probe instructions. Typically, eBPF probes are event-driven lightweight programs that are placed by specific system calls into kernel memory, and executed in a protected environment. Such programs are often difficult to develop, primarily due to its complex interactions with kernel data structures, events and its sandboxed execution. However, tools like BCC [30], bpftrace [31], libbpf [32] libbpfrs [33] abstract eBPF’s lowlevel details and provide a high-level interface to develop ebpf programs. In this paper, we use libbpf [31] to strategically trace particular kernel functions (see details in Section IV). Performance Degradation Analysis: Root-cause analysis (RCA) is central to diagnosing application performance degradation [34]. Statistical approaches such as BARO [35] and N-Sigma [36], [37] detect distribution shifts in timeseries metrics before and after anomalies to flag potential causes. Similarly, prior work [14]–[17] correlates hardware resource usage with performance anomalies, while systems like FIRM [38] apply learning-based models to infer metric relevance for performance variability within microservice dependency graphs. While the use of readily available hardware metrics makes these methods broadly applicable, it also limits diagnostic depth, omitting the fine-grained behavioral context necessary to accurately identify the causes of application-level performance degradation. With the emergence of eBPF subsystem, recent work has expanded observability beyond hardware resources to include detailed kernel-level interactions. Rezvani et al. [18] leverage eBPF to analyze the relationship between application load and the epoll subsystem, while Jha et al. [19] instrument the virtual file system and block I/O layers to detect anomalous application behavior. Other solutions, such as Coroot [39] and Pixie [40], employ eBPF to capture process-level metrics related to service communication and hardware resource usage. In contrast, APM solutions provided by Dynatrace [41] and New Relic [42] enhance application observability through language-specific agents that utilize bytecode instrumentation to capture transaction-level context and runtime specific metrics; however, they are limited to userspace behavior and do not observe thread interactions with kernel resources. Consequently, these tools remain opaque to important application thread dynamics, that reveal interactions between threads

II. R ELATED W ORK This section discusses state-of-the-art system instrumentation methods employed in performance degradation analysis related works. System-level Instrumentation: Application-agnostic performance monitoring methods collect system-level metrics to understand the application’s interaction with the kernel,

2

Target Application

Thread

IPC

Other Threads

collector

userspace kernel-space BPF TRACING / ITER

futex

net / vfs

epoll

sched

BPF Maps / Ringbufs

Thread

third-party applications that do not use them. Therefore, a generalized, application-agnostic method is needed to extend TSA with additional kernel-level thread–resource context in order to expose degradation propagation with low overhead. Our approach addresses this need by implementing targeted eBPF probes across six kernel subsystems. By collecting metrics at “thread→resource” granularity, it captures both the time a thread spends within a given subsystem and the specific resources on which this time is spent. Accordingly, this approach complements state-of-theart observability solutions rather than replacing them, as they continue to provide essential performance analysis capabilities, including distributed tracing and profiling.

Process

block IO

Fig. 2: Instrumentation architecture: Illustration of the required components to monitor a target application. Our instrumentation also accounts for IPC and therefore thread groups that have communicated with the target application will also be monitored.

III. D ESIGN AND A RCHITECTURE mediated by kernel resources such as locks, sockets, pipes, and other inter-process communication mechanisms, which are essential for diagnosing the propagation paths of performance degradation within a host. The importance of capturing such thread dynamics is underscored by [2], [9], who demonstrate that an application’s threading model plays a decisive role in its performance. KUTrace [9] exposes these dynamics and achieves full-system coverage by instrumenting kernel-to-userspace transitions. However, its kernel-boundary observation cannot identify specific coordination resources (e.g., locks) without injecting custom userspace C libraries, introducing both build-time and runtime dependencies. While prioritizing low overhead and full state coverage, it generates 2-20 MB/s of trace data to RAM, (upto 2.4 GB over 120 s). Our approach eliminates these dependencies and reduces data volume, enabling longerterm monitoring while preserving observability. To provide more granular observability, xCapture [43] measures the time each application thread spends within Linux kernel subsystems, enabling Thread State Analysis (TSA). However, its metrics lack resource-level context and therefore cannot capture the inter-thread interactions that underlie application performance behavior. Seo et al. [20] address this limitation for NoSQL databases by analyzing thread dynamics based on futex interactions. Building on this direction, our work generalizes this capability to a wider range of kernel resources and application types. Comparison Summary: Dynatrace [41] and NewRelic [42] provide application-layer observability, but require bytecodeagent instrumentation, which modifies runtime execution and can incur high overhead in terms of increased application response time, memory and CPU usage [44]–[47] in default mode, compared to kernel instrumentation. Coroot [39] and Pixie [40] are application-agnostic but operate at service/process granularity. KUTrace [9] and xCapture [43] provide thread-level tracing at finer granularity, but lack resource-level context needed to model inter-thread dependencies. KUTrace, however, provides custom instrumentation libraries that can be integrated into application software to expose additional context on lock usage. Since these libraries must be incorporated at build-/run- time, thread dynamics remain opaque for

Thread State Analysis (TSA) requires tracking the time each application thread spends in six distinct states [8]: Executing, Runnable, Swapping, Sleeping, Lock, and Idle. Some of these states can be further subdivided [48] to provide finer-grained insight into where threads spend their time. For example, Sleeping time may be partitioned into more specific categories such as waiting on storage, network, or pipe operations. However, our goal extends beyond simply measuring how long threads spend in these subsystems. We also aim to capture how threads interact with one another through shared systemlevel resources. These inter-thread dependencies often reveal how performance degradation originating from a constrained resource propagates across threads, thereby exposing the application’s critical path, i.e., the sequence of dependent threads and resources that ultimately drives performance degradation. To this end, given the defined set of Thread States, we identify and collect a comprehensive list of metrics required to perform TSA, as shown in Table I. We exclude the Swapping state from collection, since it typically arises under system-wide memory pressure rather than from thread-specific activity. Nevertheless, each metric corresponds to one of the six states and, where applicable, includes finer-grained subdivisions based on its Subsystem. To reveal an application’s thread dynamics, these metrics are collected at a resource-level granularity, enabling the attribution of thread time to specific system-level resources. For instance, “thread→futex” Granularity indicates, futex wait {time,count} metrics are collected per thread for a specific futex resource. This approach not only preserves the traditional TSA view of per-thread state distribution but also exposes inter-thread dependencies by capturing how threads wait on and interact through shared resources, often used as synchronization primitives. Table I summarizes the full set of collected metrics and corresponding thread states. Instrumentation Architecture: Figure 2 depicts our instrumentation architecture. When the target application executes probed kernel functions, custom eBPF programs capture raw statistics on thread interactions with the six kernel subsystems. These statistics are communicated to a userspace component, the collector, via maps and ring buffers, which then processes the raw data into the metrics summarized in Table I. To

3

1. Deploy Bootstrap process

2. Monitor A

Connect to metric database

Construct process dependency graph

Specify baseline and compare time ranges Discover and monitor new A processes that directly or indirectly communicate with B the boostrap process C

user input automated

3. Analyse

Select application for analysis

B

Run template or custom analysis queries

Construct thread dynamics graph

A

B

B

t2

t1

t3

C

f1 t4

Fig. 3: Diagnostic Workflow: overview of the diagnostic workflow for application performance degradation. Stages highlighted in red and green denote our method’s participation and include a summary of the associated operations, indicating whether they are automated or require user intervention. TABLE I: Enumeration of the instrumented metrics based on the Thread States enumerated by TSA [8], and the resource granularity required to monitor thread dynamics. State Subsystem Executing Runnable Scheduler Sleeping/ Idle

Metric Granularity Metric description runtime thread time spent on-CPU rq time thread time waiting on runqueue block time thread time in uninterruptible sleep iowait time thread time in uninterruptible sleep and in iowait sleep time thread time spent in interruptible sleep pipe wait time thread→pipe time thread waits for pipe VFS, thread→pipe thread pipe wait frequency Network, IO pipe wait count Sleeping/ socket wait time thread→socket time thread waits for socket Multiplexing, Idle socket wait count thread→socket thread socket wait frequency Block IO sector count thread→device thread sector requests to device epoll wait time thread→epoll thread epoll wait time Sleeping/ Multiplexing epoll wait count thread→epoll thread epoll wait frequency Idle IO epoll file wait epoll→virtual file time epoll waits for file futex wait time thread→futex time waiting for a futex Lock Futex futex wait count thread→futex futex wait frequency futex wake count thread→futex futex wake frequency

reduce resource overhead, the collector reads only statistical summaries at one-second intervals, even though the eBPF programs trace events at a much higher frequency. In the following Section, we describe the probes and techniques employed to realize the metrics described in Table I.

automatically discovered). The set of instrumented metrics (Table I) are collected for all discovered processes and stored in an OLAP database [49]. (c) In the analysis stage, users interact with the collected metrics through a user interface (UI) or an API that connects to the database storing the metrics. Based on the users selecting a target process and two time intervals, baseline and comparison (i.e., application under normal performance and degraded conditions), our approach generates two views. First, a process dependency graph of all discovered processes; And second, a thread dynamics graph showing interactions between threads and kernel resources at the granularity detailed in Section IV-A. Using these graphs, users can execute predefined queries to pinpoint state changes and resource contention. This analysis forms the core of the performance degradation analysis process (detailed in Section IV-C) and directly feeds into our selective thread tracking algorithm (Algorithm 1) which systematically traces performance degradation.

Diagnostic Workflow: Figure 3 illustrates the high-level diagnostic workflow of our approach with three stages. (a) The application deployment, is external to our method. Excluding application deployment as a requirement, is a key design objective which further eliminates the need for recompilation, modification or prior instrumentation. Consequently, our approach supports a wide range of applications and enables runtime monitoring on any sufficiently recent Linux system with eBPF support. (b) The monitoring stage initiates when a user specifies a bootstrapping process (e.g., a service’s main process) for the metric collector. The system then automatically deploys eBPF probes to track the specified process and dynamically discovers all directly and indirectly interacting processes through IPC mechanisms such as sockets and pipes. This discovery is transitive, constructing a complete process dependency graph without manual intervention (e.g., in Figure 3, if process A communicates with B, and B with C, all are monitored and

IV. M ETHODOLOGY This section describes our instrumentation methodology, i. e., how to collect the metrics in Table I, and how to leverage the metrics for performance degradation analysis.

4

pipe wait {time,count}) account for the time a thread sleeps in its read call while waiting for data to be written on the other end of the pipe. Sockets, unlike pipes, operate as two-way communication streams with both transmit/receive buffers. For inter-thread communication via sockets, each thread must have a socket mapped to the other, determined by the socket family, type, and protocol. For example, AF_INET, SOCK_STREAM, and IPPROTO_TCP sockets communicate via a TCP connection identified by the protocol 5-tuple: ⟨TCP protocol, source IP, source port, destination IP, destination port⟩. To monitor the time and frequency a thread waits for socket data reception, denoted by socket wait time and socket wait count metrics (see Table I), we probe kernel’s sock_recvmsg function. When invoking this function, the kernel passes socket specific data that identifies its respective domain specific source and destination identifiers. We currently support AF_INET, AF_INET6, and AF_UNIX domains. Similarly, on the sending end, we probe domain-specific functions for AF_INET, AF_INET6, and AF_UNIX, i. e., inet_sendmsg, inet6_sendmsg, and unix*sendmsg to monitor socket wait {time,count} metrics of thread socket data transmission. By collecting these metrics, we can estimate the duration a thread is suspended during interactions with pipe/socket subsystems, highlighting straggling external dependencies that may impede an application’s progress. Futex: Fast userspace mutexes (or futexes) serve as a thread synchronisation mechanism provided by the kernel via the futex system call [50]. Two threads synchronize by agreeing on a memory address (uaddr parameter) referencing an integer value. For thread sleep requests, the kernel compares the value at uaddr with the val parameter; if equal, the thread sleeps. To wake sleeping threads, another thread calls futex with the same uaddr, specifying val threads to wake. Common val values are 1 (i. e., wake one thread) and INT_MAX (i. e., wake all threads) [50]. The futex subsystem enables two key use cases. First, it can mediate resource access, limiting the number of threads that can access a particular resource. This functionality underpins various locking mechanisms such as read-write locks, mutexes, and semaphores. Second, futexes can be used as a work scheduling mechanism. In this scenario, a thread is put to sleep when it has no work to process and is awakened by another thread when work becomes available. An example of this is seen in Rust’s standard library, which uses futexes in its multi-producer single-consumer (mpsc) message passing implementation. Here, the consumer is parked using futex [51] when the message queue is empty and is awakened by a producer when a new message is appended. To monitor thread interactions with the futex subsystem, we instrument the sys_enter_futex (i. e., thread entering the futex system call) and sys_exit_futex (i. e., thread exiting the futex system call) tracepoints. We use the op argument to distinguish between futex wake and sleep oper-

A. Metric Collection Scheduling: A thread’s scheduling metrics refers to its time spent in CPU-related states, which are categorized into three main types. Running state describes when a thread executes on a CPU core, corresponding to our runtime metric in Table I. In the Idle state, the thread is not executing; external activity must place it back on a CPU core’s runqueue. This state can be further divided into interruptible and uninterruptible states, where the time spent in TASK_INTERRUPTIBLE state maps to the sleep time metric, while the time in the TASK_UNINTERRUPTIBLE state corresponds to our block time metric. A subset of this uninterruptible time is captured by our iowait time metric, which reflects the time spent in the TASK_UNINTERRUPTIBLE state when the in_iowait field is set to 1. Finally, in the Runnable state, a thread is ready to run on a core but will not execute while another thread is active. Preemptive schedulers, such as Linux’s completely fair scheduler (CFS), manage core time distribution among threads in a runqueue, ensuring fair allocation. The time a thread spends in runqueue without being scheduled is recorded by our rq time metric. Pipes and Sockets: In multi-threaded architectures it is common to segregate functionality between the different threads belonging to an application. Sending and receiving data to and from other threads is a fundamental requirement of these architectures to coordinate work. Pipes and sockets are two such mechanisms that facilitate this communication. When a thread writes to a pipe, data accumulates in a kernel memory buffer. This data is then forwarded in a FirstIn First-Out (FIFO) fashion to any thread that reads from the same pipe. However, userspace threads use file descriptors (fd) to identify the read and write ends of a pipe. File descriptors take the form of integer values that are used to extract the underlying struct file pointed at by this descriptor. Multiple file descriptors can point to the same backing resource with varying read/write permissions. For example, there may exist a file descriptor with write permission, and another with read permission, both referring to the same pipe. Therefore, to uniquely identify the backing resource pointed at by the file descriptor, we create a Backing Resource Identifier (BRI) with a combination of the inode’s virtual file system identifiers contained in the struct file data structure, namely: i_ino representing the inode identifier; and s_dev denoting superblock device identifier. In the above example with both file descriptors writing to same pipe, the pipe’s i_ino and s_dev will be equivalent, resulting in same BRI. To instrument the time and frequency a thread waits in read/write calls to a FIFO file, given by pipe wait {time,count} metrics (see Table I), we trace the kernel’s vfs_{write,read} functions. The struct file argument in these functions is used to compute the BRI, enabling attribution of a thread’s wait time to specific pipes. For example, if the O_NONBLOCK flag is unset (i. e., changing the file descriptor’s behavior from non-blocking to blocking mode), our metrics (i. e.,

5

ations, collecting: total time and frequency a thread sleeps on a specific uaddr, represented by futex wait time and futex wait count metrics (see Table I). The frequency of successful wake operations on a uaddr, i.e., waking at least one thread, is given by futex wake count metric. This enables observing an application’s lock contention and work scheduling patterns, crucial for understanding how degraded performance propagates across different application threads. Multiplexing IO: In synchronous programming, with the O_NONBLOCK flag unset, a thread executes a system call (e.g., recv) on a single file descriptor (e.g., socket) and must wait for its completion before proceeding to other tasks. High-performance web servers challenged this approach by introducing a scenario where a single thread manages multiple connections concurrently, waiting for incoming data on multiple connections simultaneously [52]. For network IO-intensive servers, this approach offers several advantages, including: enhanced CPU cache utilization; decreased memory footprint; and minimized context switching. The kernel mechanism enabling this new paradigm is multiplexing IO. In essence, a thread specifies which events it wants to consume from a particular file descriptor and adds it to an interest list. It can then wait for all registered fds using a specific system call, which returns when one or more fds have events ready for consumption1 . Linux offers three APIs for this purpose: select, poll, and epoll [53]. Select and poll share a similar interface where a single system call is used to both express a thread’s interests and wait until an event is ready for consumption from file descriptor. For select, we probe entry and exit of do_select, while for poll, we probe entry and exit of do_sys_poll. Upon returning from these functions, we collect BRI of all registered file descriptors and increment each BRI’s wait time (e. g., pipe wait time, socket wait time metrics in Table I). In essence, a single call to select or poll attributes a thread’s wait time to all registered BRIs (rather than just one). In contrast, epoll’s API offers three system calls for interacting with the kernel’s event poll subsystem. First, epoll_create generates an epoll resource and returns a file descriptor (epfd) for userspace code to manage its interest list. Second, a thread calls epoll_ctl with the epfd to add/remove file descriptors from the interest list. Third, a thread invokes epoll_wait with the epfd to wait for its registered fds. Given epoll’s use of separate system calls to register interest in a fd’s events (using epoll_ctl) and to wait for registered fds (using epoll_wait), our eBPF programs keep a copy of all fd BRIs registered with each epoll resource. Updating this copy is done, when fd events are registered and de-registered using ep_remove and ep_insert functions. Furthermore, given that an epfd is used by userspace threads to reference an epoll resource, an epoll’s BRI is derived from the kernel memory address pointing to its

struct eventpoll data structure. This approach differs from the previous method of calculating a file descriptor’s BRI because epoll inodes are part of the anonymous inode filesystem, which reuses the same inode for all resources of the same type. Given that multiple threads can wait for the same epfd, our first epoll related metric measures the time and frequency a thread spends waiting for a particular epoll BRI, represented by epoll wait time and epoll wait count metrics (see Table I). Additionally, we account for the time a registered file descriptor is awaited for during an epoll_wait system call (i. e., epoll file wait metric), attributed to the epoll BRI it was registered with. Disk IO: Given the data persistence requirements of many OLDI applications (e.g., databases and message brokers), their high-level performance metrics can be significantly affected by the maximum throughput achieved when writing data to disk. This is typically because these applications can only confirm message receipt after successful data storage. However, given that disk access is orders of magnitude slower than CPU cache or RAM access, monitoring the time an application’s thread spends waiting for disk request completion is crucial. While this information is partially covered by the iowait time and block time scheduling metrics, sector count metric (see Table I) provides a more comprehensive view of block device activity. It allows for calculating the total in-flight device requests and determining the proportion of these requests belonging to a target application. B. Application Dynamics Comparison B A t1

A

t3

(a)

t7

(b) B

t5

t4

t6

t2 f1

t1

t6

t4

B

A

t5

t2

f2

t3 t7

(c)

Fig. 4: Comparison of application interaction observability provided by different approaches, where purple lines denote TCP connections: (a) Coroot [39]; (b) xCapture [43]; (c) Our approach. Thread interactions are illustrated by directional arrows.

This section clarifies the distinctions between two existing observability approaches and our metric collection methodology. For comparison, we selected Coroot [39], a wellestablished observability tool that collects metrics at the service level (similar to [40]), and xCapture [43], which represents methods that collect thread state metrics for TSA. Each approach is qualitatively evaluated using the same setup consisting of two communicating services: an HTTP server

1 There are other cases that may induce an early return, such as, signal handling and elapsed timers

6

(Service B) and a client (Service A). The server exposes two endpoints, one CPU-bound and another disk-bound, while the client periodically issues requests that exercise both. Figure 4a presents the information collected by Coroot [39], which monitors applications at the cgroup level. It captures hardware resource usage per service (e.g., CPU, disk, and memory) and tracks the network connections between them. While this view effectively highlights coarse-grained performance bottlenecks, it provides limited visibility into the internal behavior of individual threads within each process. Figure 4b illustrates the data collected by xCapture [43] for the same application. This approach enhances observability by gathering the metrics required to analyze how each application thread spends its time across the six TSA states. While this provides valuable insight into where execution time is spent within kernel subsystems, it still lacks the context needed to explain why a thread is delayed or where contention originates. To this end, our approach, illustrated in Figure 4c, attributes each thread’s time to specific kernel resources. By doing so, it captures not only subsystem activity but also the interactions among threads mediated by those resources. This additional context exposes the application’s thread dynamics, i.e, how threads depend on and influence one another through shared locks, pipes, connections, and other synchronization mechanisms. As shown in Figure 4c, threads t1 and t4 act as work schedulers for their co-located threads, delegating tasks via scheduling futexes (indicated by the directional links between threads). Futexes f1 and f2 further reveal that threads t2 and t3, and t6 and t7, are contending for shared resources. Furthermore, threads t5 and t6, as seen in the thread dynamics graph, show these threads are responsible for performing the disk-intensive tasks. Consequently, our approach exposes not only where time is spent, but also how performance degradation propagates across threads and services through these kernel-level interactions. This visibility allows analysts to move beyond coarse-grained metrics and pinpoint the specific thread–resource relationships responsible for degraded performance, making our approach particularly effective for analyzing multi-threaded applications.

t2 via a futex resource. The futex node f1 corresponds to a synchronization primitive used to coordinate access to shared resources. Since both threads wait and wake on the same futex, the edges connecting them to f1 are bi-directional. Similarly, pipe p1 connects a writing thread (t2) to a reading thread (t1), forming a dependency that indicates data transfer through an IPC channel. As such, the graph exposes the propagation paths through which a thread’s degraded performance may influence other threads within the application. Algorithm 1: Selective Thread Tracking 1

Input: target application, target metric time series Output: Set of candidate threads Ttrack and their anomalous metrics Initialize: Ttrack ← ∅;

Tentry ← threads with socket_wait_{time,count} activity for target application; 3 Ttrack ← Ttrack ∪ Tentry ; 4 Tseen ← ∅; 5 repeat 6 Mf lagged ← ∅; 7 foreach thread t ∈ Ttrack \ Tseen do 8 foreach metric m ∈ TSA states for thread t do 9 if distribution of m shifts during KPI degradation then 10 Mf lagged ← Mf lagged ∪ {(t, m)}; 11 end 12 end 13 Tseen ← Tseen ∪ {t}; 14 end 15 Tnew ← ∅; 16 foreach (t, m) ∈ Mf lagged do 17 if m is a futex, pipe, or socket metric then 18 BRI ← Backing Resource Identifier from metric m; 19 Tcounterparts ← all threads that interacted with BRI (e.g., via futex_wake, pipe write, etc.); 20 Tnew ← Tnew ∪ Tcounterparts ; 21 end 22 end 23 Ttrack ← Ttrack ∪ Tnew ; 24 until no new threads are added to Ttrack ; 25 return (Ttrack , Mf lagged ); 2

requests

Selective Thread Tracking: Building on the thread p1 dynamics, our performance t1 t2 degradation analysis employs a selective thread f1 t3 t4 tracking procedure outlined in Algorithm 1. The algorithm takes a target appli- Fig. 5: Illustration of an application’s thread dynamics derived cation and time-series of its from resource-level metric collecperformance metrics (e.g., tion. response time or throughput) as input, and outputs candidate threads with anomalous metrics correlated to performance degradation. The algorithm begins by identifying entry-point threads (Tentry ), defined as threads that form the application’s interface to incoming service requests, by detecting any thread that reads from or writes to IPv4/IPv6 sockets using the socket metrics defined in Section IV-A. These threads are added to the initial tracking list (Ttrack ). Next, we perform correlation analysis which analyzes the time-series data for all metrics in Table I for all threads that have not yet been analyzed in Ttrack . Metrics with significant statistical distribution shifts during performance degradation periods suggested by methods such

C. Performance Degradation Analysis with Thread Dynamics Thread State Analysis (TSA) provides a strong foundation for diagnosing performance degradation. It relies on observing shifts in thread state metrics and identifying the subsystems constraining application performance. However, by collecting thread state metrics at resource-level granularity, our approach goes further: it reveals not only which subsystem is causing the slowdown but also how the degradation propagates across interacting threads. This section defines a systematic procedure for analyzing performance degradation in online data-intensive applications using this extended form of TSA. Example: Figure 5 illustrates an example of an application thread dynamics graph constructed from the collected metrics. Each arrow between two threads, such as t1→t2, represents a userspace schedule relationship where thread t1 schedules

7

as Mann Whitney U, Kolmogorov-Smirnov, or change point detection, are flagged for further analysis. For each flagged IPC metric (futex, pipe, socket*), the algorithm utilizes their granular resource identification. When thread t shows anomalous wait time on a specific resource (identified by its BRI, defined in Section IV-A), all threads interacting with the same BRI are identified as counterpart threads (Tnew ) and added to Ttrack if not already in the list. These steps iterate until no new unique threads are added to Ttrack , allowing us to capture all inter-dependent threads contributing to performance degradation. The algorithm concludes by returning the final set of interdependent threads Ttrack and their flagged metrics (Mf lag ) associated with specific kernel resources. This methodology effectively narrows the diagnostic search space by tracing dependency chains between threads, focusing exclusively on those directly or indirectly interacting with outward-facing threads. However, the algorithm optimistically assumes degradation propagates toward an entry-point thread. If no resource contention is found after exhausting the metric search space in Algorithm 1, a computationally expensive full search across all process metrics can be performed. The selective thread tracking algorithm has an over all time complexity of O(k · n) , where n, k denote system threads and BRI dependencies for each thread, respectively. Given that we set an upper bound on the total amount of BRIs tracked per thread, the algorithm scales linearly with the number of threads involved in performance degradation.

wrapped around a sentiment analysis model [55] in a FastAPI web-app, deployed using a Python-based webserver gateway interface gunicorn. Most of these selected applications span across a wide range of domains and have been widely used in previous literature [10], [56]–[58]. Additionally, the chosen applications provide the variability with regards to combining and utilizing different threading models, inter-process communication (IPC) methods, and user-space virtual machines. Regarding threading models, MySQl uses a thread pool, ML-inference a process pool, and Kafka, Teastore, Cassandra, and Redis use asynchronous multiplexing I/O for handling client requests. Concerning userspace virtual machines, Kafka and Cassandra operate on the Java Virtual Machine (JVM), while the MLinference service runs on the Python Virtual Machine. Lastly, with regards to IPC, Kafka and Teastore leverage pipes to communicate between threads, while futex mechanisms are used by all applications for synchronisation and work scheduling activities. Workload Generation: We used YCSB [59] and TPCC [60] benchmarks to generate read- and update-intensive workloads for MySQL, while for Cassandra, only YCSB was used. For Redis, we used the official [61] and the memtier [62] benchmarks to saturate the server. Workload for the MLinference server and Teastore applications was generated using the Locust [63] framework. We applied the same workload pattern for both cases, varying load between 2 – 60 users. For the ML-inference server specifically, the server is queried about the sentiment of tweets we extract from a twitter dataset [64], whereas Teastore’s requests are generated by the request generator provided in their official repository [54]. Finally, for Kafka we generated a variable workload pattern by creating a custom producer with an increasing production rate of up to 450.000 events per second. Degradation Scenarios: We utilized a combination of two approaches to degrade the selected applications performance. The first approach used benchmarks to increase the application’s load to the point of saturation, impairing its performance, similar to [65]. The second approach induced a combination of CPU, disk and lock contention to interfere with the application’s activity and hinder its performance. CPU contention involved either underprovisioning CPU resources (e.g. Teastore and Cassandra) or competing for same core on-cpu time (e.g. Redis). As for disk contention, inspired by stress-ng’s [66] hdd stress procedure, we start 80 threads that execute synchronous writes to disk (O_SYNC flag set) at their maximum achievable rate, to compete for the underlying disk resource. Lastly, with regards to lock contention, we induced this type of contention on MySQL due to its ACID compliance [67]. We performed this by executing two distinct YCSB workloads reading from and writing to the same dataset. Target Performance Metrics: To evaluate the instrumented metrics’ robustness to different high-level KPIs, we vary the target performance metrics in each experiment. In case of MySQL, we use two target metrics: 95th percentile response time (seconds) and throughput (requests per second), which

V. E XPERIMENTS This section presents experimental setup, selected applications, induced workload and performance degradation scenarios to evaluate the diagnostic capability of our method. Experimental Setup: We implemented our method as a Rust executable that leverages libbpf to interface with bpf. It cyclically gathers kernel statistics every second using our custom eBPF programs, transforms them into metrics listed in Table I, and persists them for further analysis. The metric collector’s complete source code and this Section’s reproducibility artifact2 is available in an online GitHub repository [21]. We executed our experiments on i7-11850H Intel server @ 2.50 GHz running Ubuntu 22.04 LTS, Jammy Jellyfish (x86_64) OS with 6.8.12 Linux kernel, 8 CPU cores, 2×16 GB DDR4 RAM, and 1 TB SSD disk. Selected Applications: We selected six online dataintensive applications for validation. Our application set includes a relational and a NoSQL database, MySQl and Apache Cassandra. We also selected Redis which fits into the category of in-memory Key-value datastores, while Kafka is a message broker middleware. To incorporate a broader spectrum of applications, our evaluation set also includes a microservicebased benchmark application TeaStore [54]. Finally, we also evaluate a machine learning inference (ML-inference) service 2 Artifact is also included in the two-page Artifact Description (AD)/Artifact Evaluation (AE) appendix, at the end of the paper.

8

50

100

1.0 0.5

Relative Time (s)

0

50

Wait Time (s/s)

Share (%)

100

Relative Time (s)

0.75 0.50 0.25 100

Relative Time (s)

0.50 0.25

(g)

50

200 150

1) MySQL: Experiment Scenario & Target Metric Observation: We executed TPCC and YCSB read-intensive workloads over 120 s. Two controlled interventions were introduced during this period. The first intervention, beginning 25 s into execution and lasting 30 s, injected a disk contention workload. The second intervention, starting at 85 s and lasting 30 s, ran a YCSB update-intensive workload that induced both lock contention with the YCSB read workload and disk contention with the TPCC benchmark due to update operations. Figures 6a and 6b present TPCC throughput and YCSB 95th percentile latency across the experiment. During the first intervention, TPCC throughput declines sharply, while YCSB latency remains unaffected, consistent with TPCC’s disk-intensive behavior versus YCSB’s in-memory reads. The second intervention, in contrast, degrades both metrics, confirming interference between the concurrent read and update workloads.

50 0 0

100

Relative Time (s)

50

100

Relative Time (s) (f)

0.75 0.50 0.25 50

t4 t4 t3 t3

100

1.00

0.000

100

Relative Time (s)

(e)

1.00

50

t4 t4 t3 t3

0.75

0.00 0

Wait Time (s/s)

Time in IOWait (s/s)

A. Disk Constrained 50

(c)

1.00

(d)

0.000

1

(b)

t1 t2 t3

kernel identifiers. Each case highlights only the most relevant metrics and interactions necessary to explain the observed performance degradation. All experimental data and analysis artifacts are available in an online public repository [21].

252:1 252:0 259:3 252:2

2 0 0

100

Relative Time (s)

(a) 100 75 50 25 00

50

1e5

3

Wake Count

0

1.5

Wake Count

150

2.0

Sectors

200

Response Time (ms)

Throughput (req/s)

250

4

100

Relative Time (s) (h)

400

t5 t6 t3 t5 t6 t3

200 0 0

50

100

Relative Time (s) (i)

Fig. 6: MySQL target metric observation and performance degradation deconstruction: (a) TPCC workload throughput; (b) YCSB read-intensive workload 95th percentile response time; (c) Sector requests per device (identified by their major:minor|values; (d) Device request share for MySQL threads; (e) Per thread futexes’ (f1, f2) wait time; (f) Per thread futexes’ (f1, f2) wake activity; (g) Thread t2’s iowait time; (h) Thread t2’s futexes’ (f3, f4) wait time; (i) Per thread futexes’ (f3, f4) wake activity.

t3 t6

f1,f2 wake + wait wake f3,f4 wait wake

wait + wake

t2

t4

wait

Fig. 7: MySQL thread dynamics for the degradation path: blue threads denote entry-point threads.

Deconstructing Performance Degradation: Following the procedure in Section IV-C, we first identify the entry-point threads t3, t4, and t6 using their socket wait time,count metrics. Threads t3 and t6 then wake t2 via futexes f3 and f4 (Fig. 6i). The decrease in t2’s futex wait time (Fig. 6h), alongside a rise in its iowait time, signals disk I/O contention in both interventions. Figure 6c shows that during the first intervention MySQL competes with co-located processes for disk access, whereas the second intervention reveals increased futex activity (Fig. 6i) indicating an additional client, handled by t3 is contending for the same disk resource. The second degradation path involves lock contention between YCSB read and update-intensive workloads. Thread t4’s futex wait time rises sharply (Fig. 6e), and mutual wake activity between t3 and t4 (Fig. 6f) confirms synchronization over shared futexes. Since t3 handles updates and t4 handles reads, this lock contention directly contributes to YCSB’s latency degradation. Overall, the thread dynamics (Fig. 7) illustrate how localized lock and I/O contention propagates through futex-mediated scheduling and shared I/O channels, ultimately degrading application-level throughput and latency. 2) Kafka: Experiment Scenario & Target Metric Observation: For the Kafka experiment, we progressively increased

measures MySQL’s performance as seen by the YCSB and TPCC workloads. Cassandra’s performance is measured using median response time, while Kafka uses the total production rate (i.e. throughput) in events per second, as a proxy of its performance. Similarly, we used 95th percentile response time as a target metric to estimate ML-inference server, Redis and Teastore application performance. VI. R ESULTS This section demonstrates how the eBPF-based metrics defined in Table I and collected using methodology in Section IV-A are used to analyze application performance degradation, under the workload patterns and scenarios from Section V, through the procedure outlined in Section IV-C. Our goal is to infer not only the root cause of degradation but also how it propagates across threads as a result of the application’s thread dynamics. By correlating thread state shifts with interactions over shared kernel resources, we can trace how localized contention or blocking cascades through dependent threads, ultimately constraining user-facing performance. Degradation analysis results for each application introduced in this section are grouped by their primary resource constraint. For clarity, threads, futexes, epoll instances, and pipes are denoted as t, f, e, and p, respectively, instead of using raw

9

Relative Time (s)

100

Relative Time (s)

20 10 100

200

Relative Time (s) (d)

100

1.00

200

0.75

150

0.50 0.25 0.00

0

200

Relative Time (s) (e)

200

Relative Time (s) (c)

Write Count

30

0

100 0

50

100

150

Relative Time (s)

0.75 0.50 0.25 0.000

100 50 200

Relative Time (s)

0.05 0.04 0.03 0.02 0.01 0.000

2

100

150

Relative Time (s)

(f)

(d)

Fig. 8: Kafka target metric and performance degradation deconstruction: (a) Ideal/measured throughput; (b) Kafka threads’ runtime; (c) Kafka threads’ block time; (d) Combined connection wait time for epoll e1; (e) Thread t9 epoll e1 wait time; (f) Pipe p1 write activity.

100

150

0.5 0.4 0.3 0.2 0.1 0.00

100

150

Relative Time (s) (e)

150

100

150

1.00

252:1 252:0 259:3

50

100

(c)

1 0 0

50

Relative Time (s)

(b)

1e5

50

50

Relative Time (s)

(a)

t2 t5 t7 t1 t4 t6 t8 t3 total

00

125

Runtime (s/s)

0.00

(b)

Wait Time (s/s)

Total Wait Time (s/s)

(a)

0

200

150

1.00

Share (%)

0.000

200

0.5

175

Runqueue Time (s/s)

0.25

200

Sectors

0.50

Response Time (µs)

0.75

t1 t4 t7 t2 t5 t8 t3 t6 total

1.0

IOWait Time (s/s)

100

Block Time (s/s)

1.00 measured ideal

Runtime (s/s)

Throughput (events/s)

1e5 4 3 2 1 00

0.75 0.50 0.25 0.00 0

50

Relative Time (s) (f)

Fig. 10: Cassandra target metric observation and performance degradation deconstruction: (a) YCSB update-intensive workload median response time; (b) Cassandra threads’ rq time throughout the first intervention; (c) Threads’ runtime throughout the second intervention; (d) Thread iowait time throughout the second intervention; (e) Sector requests per device; (f) Device request share for Cassandra threads.

the production load until the broker could no longer sustain the incoming rate. Figure 8a compares the ideal and actual production rates, showing that the system begins to falter around an ideal rate of 450.000 events per second, where the observed production rate drops by approximately 55–77% below target. Deconstructing Performance Degradation: We t1 wai ite t wr start by identifying the ent9 read p1 write it a trypoint threads responsit8 w ble for external communiFig. 9: Kafka thread dynamics for cation (t9-t11) using the the degradation path: blue threads socket wait time,count and denote entry-point threads. epoll wait time,count metrics. These threads each manage dedicated epoll resources to handle incoming producer requests and send acknowledgments. During the degradation phase, Figure 8e shows that thread t9 spends approximately 100% of its time waiting on its epoll backing resource identifier (BRI). Combined with Figure 8d, this suggests that t9 is awaiting for another type of resource than client connections through epoll e1. Examining the resources registered with e1 reveals a pipe p1 that shows a sharp decrease in write activity (Figure 8f) throughout the same time period as kafka saturated. The threads responsible for writing to this pipe (t1-t8) are part of a Kafka worker pool, whose combined time in uninterruptible wait, saturated at the same moment as Kafka’s throughput degraded. Mapping these interactions to the thread dynamics graph (Fig. 9) reveals a clear propagation path: worker threads (t1-t8) saturate in block time, delaying writes to the internal pipe; this stalls the pipe’s consumer, thread t9, which then delays replies to client requests. 3) Cassandra: Experiment Scenario & Target Metric Observation: For the Cassandra experiment, we execute a YCSB update-intensive workload while measuring the median

...

response time (Figure 10a) as the target metric. Two interventions are introduced. The first, at 33 s, launches a 30 s YCSB read-intensive workload targeting the same dataset to induce lock contention. The second, from 100 s to 125 s, introduces a disk contention workload to interfere with Cassandra’s disk operations by saturating the underlying device. Deconstructing Performance Degradation: For first scenario, Figure 10b shows a significant increase in rq time of the request-facing threads which led to Cassandra’s degraded performance. Contrary to our expected cause of degradation, CPU contention, and not locking, was the cause of degradation. In the second intervention, Figure 10a shows a slight increase in Cassandra’s median response time. Unexpectedly, Figure 10c shows increased runtime despite the applied load remaining constant, which suggests the intervention led to memory contention, that manifested as an increase in runtime due to additional stalled CPU cycles. To further investigate why disk isn’t the cause of performance degradation, Cassandra’s IO path shown in Figures 10d, 10e, and 10f show that data is persisted to disk asynchronously, and, consequently, does not directly interfere with the client requests. Further investigating Cassandra’s documentation [68] reveals that with commitlog_sync set to periodic and commit_log_sync_period to 10000, Cassandra waits 10 000 ms before fsync’ing the commit log, explaining the observed behavior. B. CPU Constrained 1) ML-inference: Experiment Scenario & Target Metric Observation: As described in Section V, we evaluated an ML-inference server deployed with gunicorn configured with

10

50

100

Relative Time (s)

(a)

0.000

50

20 0 0

100

Relative Time (s)

(b)

0

0

50

100

30 20 10 0

Relative Time (s)

0

50

100

Relative Time (s)

(d)

60

80

100 120

Relative Time (s) (a)

t1/rq t2/run t2/rq t1/run

0.25 0.00

60

80

50

(c)

1.0 0.8 0.6 0.4 0.2

100

Relative Time (s)

20

100

Relative Time (s)

40

60

80

Relative Time (s)

100

0.8 0.6 0.4 0.2 0.0

(e)

to measure 95th percentile latency (Figure 12a). As an intervention, the official Redis benchmark [61] runs from 76 s for 30 s, saturating the server with additional workload. Deconstructing Performance Degradation: Given Redis’s largely single-threaded architecture, entrypoint thread (t1) in Figure 12b is responsible for handling requests and communicating with clients. At the start of the intervention, t1’s runtime sharply increases. Shortly, a background Redis thread (t2) begins competing for same CPU core, resulting in contention that raises the observed rq time for both t1 and t2 for approximately 13 s. Once t2 terminates, t1 resumes uninterrupted execution, and CPU usage stabilizes at 66%, allowing the server to efficiently handle the remaining load.

0.75 0.50

50

100

Fig. 13: Teastore target metric observation and performance degradation deconstruction: (a) Teastore load pattern; (b) End-toend 95th percentile response time; (c) Webui’s threads’ rq time; (d) Webui’s threads’ runtime; (e) Time-series histogram of webui’s wait time for the constrained service.

1.00

Share (s/s)

Response Time (µs)

1e4

1.0 0.8 0.6 0.4 0.2 0.00

50

Relative Time (s) (b)

(d)

(e)

Fig. 11: ML-inference target metric observation and performance degradation deconstruction: (a) ML-inference load pattern; (b) MLinference 95th percentile response time; (c) ML-inference threads’ rq time; (d) ML-inference combined connection wait time; (e) Futex wake activity; 10.0 7.5 5.0 2.5 0.0

00

(a)

Runtime (s/s)

Wake Count

Wait Time (s/s)

5

100

Relative Time (s)

(c)

40 10

50

100

1.0 0.8 0.6 0.4 0.2 0.00

Relative Frequency

0 0

0.25

40

200

Runqueue Time (s/s)

100

0.50

Response Time (ms)

50

Relative Time (s)

2000

60

0.75

Wait Time (s/s)

0 0

4000

1.00

Users

20

6000

Runqueue Time (s/s)

40

Response Time (ms)

Users

60

100 120

Relative Time (s) (b)

Fig. 12: Redis target metric observation and performance degradation deconstruction: (a) Memtier benchmark 95th percentile response time; (b) Redis runtime and rq time activity.

four worker processes. A variable load pattern (Figure 11a) was generated using Locust, ranging from 2 to 60 concurrent users to induce short bursts of very high workload. The 95th percentile response time (Figure 11b) serves as our target metric, exhibiting a 100× increase, from 75 ms to 7500 ms, under degraded conditions. Deconstructing Performance Degradation: We begin our analysis with the gunicorn worker thread responsible for handling client requests. Figure 11d shows the time this entrypoint thread spends waiting for incoming connections, closely reflecting the load pattern applied to the ML-inference server. The short, intermittent bursts of activity indicate that this worker only receives requests when load balancing assigns it new clients. Upon receiving data, the entrypoint thread delegates request handling to other threads within the same worker process via futex-based synchronization, as shown in Figure 11e. The resulting rq time activity of these handler threads (Figure 11c) exhibits a sharp increase in CPU contention during load surges. This behavior indicates that CPU saturation within the worker threads is the primary factor behind the ML-inference server’s degraded performance. 2) Redis: Experiment Scenario & Target Metric Observation: Redis experiment uses the memtier benchmark [62]

C. Externally Constrained 1) Teastore: Experiment Scenario & Target Metric Observation: In multi-service systems, performance degradation can originate from external dependencies. Our final experiment deploys the Teastore [54], monitoring the webui component while constraining the persistence service. A variable load pattern is applied (Figure 13a), and the client’s 95th percentile response time is measured as the target metric (Figure 13b). Deconstructing Performance Degradation: We analyze the system’s rq time and runtime in Figures 13c and 13d. The webui component shows low CPU usage throughout, with runqueue activity increasing at peak load. This runqueue latency increase doesn’t fully explain the sharp response time rise. Figure 13e presents a time series histogram of webui service socket wait time derived from connections to the constrained persistence service. Higher y-axis positions indicate longer wait times, while bin brightness shows relative frequency within each timestamp column.

11

baseline compare

0.1 0.00.0

0.1

Block Time (s) (a)

baseline compare

0.3 0.2 0.1 0.00

10

20

Pipe Write Count (b)

Relative Frequency

0.2

Relative Frequency

Relative Frequency

TABLE II: Instrumentation overhead: Latency mean and stdev with(out) instrumentation. 0.4

baseline compare

Latencyw/out Latencywith Latency Mean ± SD (ms) Mean ± SD (ms) Overhead (ms) Redis 2.053 ± 0.155 2.425 ± 0.310 0.372 Cassandra 1.689 ± 0.002 1.758 ± 0.007 0.070 MySQL 47.193 ± 2.344 47.495 ± 3.023 0.302 Kafka 2.170 ± 0.051 2.231 ± 0.033 0.061 App

0.2 0.00.0

0.2

Wait Time (s/s)

0.4

(c)

Overhead Analysis: We evaluate overhead for four standalone applications using the same workloads from Section V, comparing latency with(out) our instrumentation over 10 runs. Table II shows the latency mean and standard deviation for both scenarios. Induced instrumentation overhead depends on the frequency and computation of our eBPF programs and varies with each application’s kernel resource needs. We observe that instrumentation impacts Cassandra and Kafka least (≈ 0.07 ms) and Redis and MySQL most (≈ 0.3 ms), likely because MySQL and Redis process requests immediately, while Kafka and Cassandra defer processing. Moreover, under saturation benchmarks (i.e., presented in Sections VI-A2, VI-B1, VI-B2), where Kafka, Redis, and ML-inference were heavily loaded, our tool’s overhead remained low: i.e., 10% of a single CPU core and 115KB/s disk writes. This efficiency is attributable to eBPF’s in-kernel statistical aggregation, which minimizes userspace processing.

Fig. 14: Alternative compare and baseline distribution representation for: (a) Fig 8c; (b) Fig 8f; (c) Fig 6h. Listing 1: Templated query that extracts baseline and compare distribution data for an application’s thread block time. SELECT ts, blkio_share, ’baseline’ FROM taskstats_view WHERE {{ pid_filter }} AND {{ baseline_filter }} AND blkio_share > 0 UNION ALL SELECT ts, blkio_share, ’compare’ FROM taskstats_view WHERE {{ pid_filter }} AND {{ compare_filter }} AND blkio_share > 0

Figure 13e highlights three key regions. The most notable, 70 s – 92 s interval, shows persistence service wait times shifting from 0 s – 0.1 s to 0.9 s – 1 s, coinciding with peak load (i. e., 60 users). Within the 35 s – 48 s interval, Figure 13e also shows an increased wait time for persistence service connections, also coinciding with an increase in user count (15 users). Lastly, the 0 s – 10 s interval, aligns with the initial high latency, despite a low user count (2 users). These insights indicate the degradation is external to the webui service, demonstrating the selected metrics’ utility in identifying both internal and external sources of contention.

VIII. C ONCLUSION Application performance degradation diagnosis requires overcoming the constraints of thread state analysis to uncover the thread dynamics that underpin performance bottlenecks. In this paper, we presented an application-agnostic performance degradation analysis method that extends TSA by leveraging eBPF to instrument fine-grained thread-resource interactions across six critical kernel subsystems. The 16 implemented metrics captures the precise relationships between threads and resources like futexes, sockets, and disks, further revealing how degradation propagates through an application’s internal architecture. While our current focus is on thread–resource interactions, extending this metric set with complementary system-wide statistics is planned for future work. We demonstrated the efficacy of our approach by validating it through extensive experiments on a diverse set of six OLDI applications, including MySQL, Kafka, and Redis. Under various workload and contention scenarios, the collected metrics successfully pinpointed the root causes of degradation, such as identifying lock-contending threads, disk IO bottlenecks, and CPU saturation. Furthermore, our method showed capability of distinguishing internal bottlenecks from external service dependencies (e.g., TeaStore microservice benchmark). More importantly, this granular observability was achieved by our method with minimal instrumentation overhead, confirming its feasibility for real-world deployment.

VII. D ISCUSSION Mitigating Metric Noise: To enhance diagnostic clarity, we augment time-series views with a distributional representation of metric values between baseline (i.e, normal) and comparison (i.e, degraded) periods. This shift from temporal to statistical domain can enable robust quantification of state changes using statistical tests (e.g., Mann-Whitney U, Kolmogorov-Smirnov) and effect size measures (Wasserstein distance, Cohen’s d). For example, Figure 14a transforms the noisy block time signal from Figure 8c into two distinct distributions, clearly revealing a shift toward higher thread block wait times. Similarly, Figure 14b shows reduced pipe write frequency in Kafka’s degraded phase from Figure 8f, while Figure 14c highlights decreased futex wait times in MySQL from Figure 6h, which is consistent with increased thread activity in other subsystems. These distributional views are generated through templated queries integrated into our diagnostic workflow, allowing systematic, statistically grounded comparison without manual inspection of volatile time-series data. For example, Listing 1 extracts an application’s block-time distribution using parameterized user input.

ACKNOWLEDGMENT This work was supported and funded by European Union (EU) CETPartnership (CET-P) FLEXI project and Dutch

12

Research Council (NWO) (Grant EP.1602.24.003).

[19] D. N. Jha, G. Lenton, J. Asker, D. Blundell, and D. Wallom, “Holistic runtime performance and security-aware monitoring in public cloud environment,” in 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE, 2022, pp. 1052–1059. [20] C. Seo, Y. Chae, J. Lee, E. Seo, and B. Tak, “Nosql database performance diagnosis through system call-level introspection,” in NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium. IEEE, 2022, pp. 1–9. [21] D. Landau and N. Saurabh, “Prism,” Feb. 2026. [Online]. Available: https://github.com/EC-labs/prism [22] S. Nagar, B. Singh, V. Kashyap, C. Seetharaman, N. Sharoff, and P. Banerjee, “Where is your application stuck?” in Linux Symposium. Citeseer, 2007, p. 71. [23] D. Gavin, “Performance monitoring tools for linux,” Linux Journal, vol. 1998, no. 56es, pp. 1–es, 1998. [24] S. Kreutzer, C. Iwainsky, M. Garcia-Gasulla, V. Lopez, and C. Bischof, “Runtime-adaptable selective performance instrumentation,” in 2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2023, pp. 423–432. [25] Z. Wang, H. Hu, L. Kong, X. Kang, Q. Xiang, J. Li, Y. Lu, Z. Song, P. Yang, J. Wu et al., “Diagnosing application-network anomalies for millions of {IPs} in production clouds,” in 2024 USENIX Annual Technical Conference (USENIX ATC 24), 2024, pp. 885–899. [26] V. Prasad, W. Cohen, F. Eigler, M. Hunt, J. Keniston, and J. Chen, “Locating system problems using dynamic instrumentation,” in 2005 Ottawa Linux Symposium. New York, NY: IEEE, 2005, pp. 49–64. [27] M. Desnoyers and M. R. Dagenais, “The lttng tracer: A low impact performance and behavior monitor for gnu/linux,” in OLS (Ottawa Linux Symposium), vol. 2006. Citeseer, 2006, pp. 209–224. [28] B. Cantrill, M. W. Shapiro, A. H. Leventhal et al., “Dynamic instrumentation of production systems.” in USENIX Annual Technical Conference, General Track, 2004, pp. 15–28. [29] eBPF, “ebpf,” https://docs.kernel.org/bpf/, 2025, accessed: 2025-07-20. [30] Iovisor, “Bpf compiler collection (bcc),” https://github.com/iovisor/bcc, 2025, accessed: 2025-07-20. [31] Bpftrace, “bpftrace,” https://github.com/bpftrace/bpftrace, 2025, accessed: 2025-07-20. [32] Libbpf, “libbpf source library,” https://github.com/libbpf/libbpf, 2025, accessed: 2025-07-20. [33] Libbpf-rs, “libbpf-rs: A Rust wrapper around libbpf,” https://github.com/ libbpf/libbpf-rs, 2025, accessed: 2025-07-20. [34] M. R. Wyatt, S. Herbein, K. Shoga, T. Gamblin, and M. Taufer, “Canario: Sounding the alarm on io-related performance degradation,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2020, pp. 73–83. [35] L. Pham, H. Ha, and H. Zhang, “Baro: Robust root cause analysis for microservices via multivariate bayesian online change point detection,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 2214–2237, 2024. [36] M. Li, Z. Li, K. Yin, X. Nie, W. Zhang, K. Sui, and D. Pei, “Causal inference-based root cause analysis for online service systems with intervention recognition,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 3230– 3240. [37] J. Lin, P. Chen, and Z. Zheng, “Microscope: Pinpoint performance issues with causal graphs in micro-service environments,” in Service-Oriented Computing: 16th International Conference, ICSOC 2018, Hangzhou, China, November 12-15, 2018, Proceedings 16. Springer, 2018, pp. 3–20. [38] H. Qiu, S. S. Banerjee, S. Jha, Z. T. Kalbarczyk, and R. K. Iyer, “{FIRM}: An intelligent fine-grained resource management framework for {SLO-Oriented} microservices,” in 14th USENIX symposium on operating systems design and implementation (OSDI 20), 2020, pp. 805– 825. [39] Coroot, “Coroot,” https://docs.coroot.com/, 2025, accessed: 2025-10-09. [40] Pixie, “Pixie,” https://docs.px.dev/, 2025, accessed: 2025-10-09. [41] dynatrace, “dynatrace runtime support,” https://docs.dynatrace.com/ docs/ingest-from/technology-support/application-software, accessed: 2026-01-16. [42] new relic, “application performance monitoring,” https://docs.newrelic. com/docs/apm/new-relic-apm/getting-started/introduction-apm/, accessed: 2026-01-16. [43] xCapture, “xcapture,” https://github.com/tanelpoder/0xtools, 2025, accessed: 2025-10-09.

R EFERENCES [1] D. Meisner, C. M. Sadler, L. A. Barroso, W.-D. Weber, and T. F. Wenisch, “Power management of online data-intensive services,” in Proceedings of the 38th annual international symposium on Computer architecture, 2011, pp. 319–330. [2] A. Sriraman and T. F. Wenisch, “{µTune}:{Auto-Tuned} threading for {OLDI} microservices,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. 177–194. [3] P. Manghwani, J. Loyall, P. Sharma, M. Gillen, and J. Ye, “End-to-end quality of service management for distributed real-time embedded applications,” in 19th IEEE International Parallel and Distributed Processing Symposium. IEEE, 2005, pp. 8–pp. [4] Z. Li, Q. Chen, S. Xue, T. Ma, Y. Yang, Z. Song, and M. Guo, “Amoeba: Qos-awareness and reduced resource usage of microservices with serverless computing,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2020, pp. 399– 408. [5] I. Arapakis, X. Bai, and B. B. Cambazoglu, “Impact of response latency on user behavior in web search,” in Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, 2014, pp. 103–112. [6] A. Bouch, A. Kuchinsky, and N. Bhatti, “Quality is in the eye of the beholder: Meeting users’ requirements for internet quality of service,” in Proceedings of the SIGCHI conference on Human factors in computing systems, 2000, pp. 297–304. [7] D. J. Dean, H. Nguyen, P. Wang, and X. Gu, “{PerfCompass}: Toward runtime performance anomaly fault localization for {Infrastructure-asa-Service} clouds,” in 6th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 14), 2014. [8] B. Gregg, Systems performance: enterprise and the cloud. Pearson Education, 2014. [9] R. L. Sites, Understanding software dynamics. Addison-Wesley Professional, 2021. [10] J. Grohmann, P. K. Nicholson, J. O. Iglesias, S. Kounev, and D. Lugones, “Monitorless: Predicting performance degradation in cloud applications with machine learning,” in Proceedings of the 20th international middleware conference, 2019, pp. 149–162. [11] Y. Cheng, X. Huang, Z. Liu, J. Chen, X. Gao, Z. Fang, and Y. Yang, “Fedge: An interference-aware qos prediction framework for black-box scenario in iaas clouds with domain generalization,” in 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2024, pp. 128–138. [12] R. Yanggratoke, J. Ahmed, J. Ardelius, C. Flinta, A. Johnsson, D. Gillblad, and R. Stadler, “Predicting real-time service-level metrics from device statistics,” in 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM). IEEE, 2015, pp. 414–422. [13] J. Ahmed, T. Josefsson, A. Johnsson, C. Flinta, F. Moradi, R. Pasquini, and R. Stadler, “Automated diagnostic of virtualized service performance degradation,” in NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium. IEEE, 2018, pp. 1–9. [14] I. Cohen, J. S. Chase, M. Goldszmidt, T. Kelly, and J. Symons, “Correlating instrumentation data to system states: A building block for automated diagnosis and control.” in OSDI, vol. 4, 2004, pp. 16–16. [15] S. Zhang, I. Cohen, M. Goldszmidt, J. Symons, and A. Fox, “Ensembles of models for automated diagnosis of system performance problems,” in 2005 International Conference on Dependable Systems and Networks (DSN’05). IEEE, 2005, pp. 644–653. [16] Y. Gan, M. Liang, S. Dev, D. Lo, and C. Delimitrou, “Sage: practical and scalable ml-driven performance debugging in microservices,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2021, pp. 135–151. [17] Y. Gan, Y. Zhang, K. Hu, D. Cheng, Y. He, M. Pancholi, and C. Delimitrou, “Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices,” in Proceedings of the twenty-fourth international conference on architectural support for programming languages and operating systems, 2019, pp. 19–33. [18] M. Rezvani, A. Jahanshahi, and D. Wong, “Characterizing in-kernel observability of latency-sensitive request-level metrics with ebpf,” in 2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2024, pp. 24–35.

13

[44] “Reduce the infrastructure agent: CPU footprint — New Relic Documentation,” https://docs.newrelic.com/docs/infrastructure/ infrastructure-troubleshooting/troubleshoot-infrastructure/reduceinfrastructure-agents-cpu-footprint/, [Accessed 14-01-2026]. [45] “Troubleshooting large memory usage (Node.js): New Relic Documentation.” https://docs.newrelic.com/docs/apm/agents/nodejsagent/troubleshooting/troubleshooting-large-memory-usage-nodejs/, [Accessed 10-01-2026]. [46] “Controlling Measurement Overhead: Dynatrace.” https://www. dynatrace.com/resources/ebooks/javabook/controlling-measurementoverhead/, [Accessed 10-01-2026]. [47] “Process deep monitoring: Dynatrace,” https://docs.dynatrace.com/docs/ observe/infrastructure-observability/process-groups/configuration/pgmonitoring, [Accessed 10-01-2026]. [48] B. Gregg, “The tsa method,” https://www.brendangregg.com/tsamethod. html, 2025, accessed: 2025-10-07. [49] M. Raasveldt, H. Muehleisen et al., “Duckdb,” in Proceedings of the 2019 International Conference on Management of Data. ACM, 2019. [50] U. Drepper, “Futexes are tricky,” Futexes are Tricky, Red Hat Inc, Japan, vol. 4, 2005. [51] R. Lang, “Rust parker implementation,” https://github.com/rustlang/rust/blob/eb33b43bab08223fa6b46abacc1e95e859fe375d/library/ std/src/sys/sync/thread parking/futex.rs, 2025, accessed: 2025-07-20. [52] T. Brecht, G. Janakiraman, B. Lynn, V. Saletore, and Y. Turner, “Evaluating network processing efficiency with processor partitioning and asynchronous i/o,” ACM SIGOPS Operating Systems Review, vol. 40, no. 4, pp. 265–278, 2006. [53] M. Kerrisk, The Linux programming interface: a Linux and UNIX system programming handbook. No Starch Press, 2010. [54] J. Von Kistowski, S. Eismann, N. Schmitt, A. Bauer, J. Grohmann, and S. Kounev, “Teastore: A micro-service reference application for benchmarking, modeling and resource management research,” in 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 2018, pp. 223–236. [55] J. M. Pérez, M. Rajngewerc, J. C. Giudici, D. A. Furman, F. Luque, L. A. Alemany, and M. V. Martı́nez, “pysentimiento: A python toolkit for opinion mining and social nlp tasks,” 2023. [56] Y. Xing, X. Wang, S. Torabi, Z. Zhang, L. Lei, and K. Sun, “A hybrid system call profiling approach for container protection,” IEEE Transactions on Dependable and Secure Computing, 2023. [57] T. Palit, Y. Shen, and M. Ferdman, “Demystifying cloud benchmarking,” in 2016 IEEE international symposium on performance analysis of systems and software (ISPASS). IEEE, 2016, pp. 122–132. [58] K. Wang, S. Wu, K. Suo, Y. Liu, H. Huang, Z. Huang, and H. Jin, “Characterizing and optimizing kernel resource isolation for containers,” Future Generation Computer Systems, vol. 141, pp. 218–229, 2023. [59] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, “Benchmarking cloud serving systems with ycsb,” in Proceedings of the 1st ACM symposium on Cloud computing, 2010, pp. 143–154. [60] S. T. Leutenegger and D. Dias, “A modeling study of the tpc-c benchmark,” ACM Sigmod Record, vol. 22, no. 2, pp. 22–31, 1993. [61] Redis, “Redis benchmark,” https://redis.io/docs/latest/operate/oss-andstack/management/optimization/benchmarks/, 2025, accessed: 2025-0720. [62] RedisLabs, “Memtier benchmark,” https://github.com/RedisLabs/ memtier benchmark, 2025, accessed: 2025-07-20. [63] Locust, “An open source load testing tool.” https://locust.io, accessed: 2025-07-20. [64] Kaggle, “Twitter sentiment analysis dataset,” https://www.kaggle.com/ datasets/jp797498e/twitter-entity-sentiment-analysis, 2025, accessed: 2025-07-20. [65] B. Gregg, “Thinking methodically about performance,” Communications of the ACM, vol. 56, no. 2, pp. 45–51, 2013. [66] Stressng, “stress next generation,” https://github.com/ColinIanKing/ stress-ng/tree/master, accessed: 2025-07-20. [67] MySQL, “Innodb and the acid model,” https://dev.mysql.com/doc/ refman/8.4/en/mysql-acid.html, accessed: 2025-07-20. [68] Cassandra, “Cassandra storage engine documentation,” https://cassandra. apache.org/doc/4.1/cassandra/architecture/storage engine.html, accessed: 2025-07-20.

14

Appendix: Artifact Description/Artifact Evaluation This two-page appendix contains the Artifact Description (AD) and Artifact Evaluation (AE). The complete source code for the metric collector and analysis interface is also released opensource in an online GitHub repository [21]. The experiment datasets, the notebooks used for analysis, and detailed instructions/scripts to reproduce the experiments and results presented in Sections V and VI are publicly available at https://github.com/EC-labs/ipdps2026-prism-artifact/ releases/download/v1.0.0/archive.tar.

Expected Reproduction Time (in Minutes) Without analysis, reproducing the 6 experiments in Section V requires approximately 30 min: 1 min to setup the target application and metric collector (Prism), 4 min to execute the workloads and degradation scenarios, and another 1 min for post-processing the target KPI metrics to the format expected by Prism’s analysis UI. With regard to the time for analysis, this may vary between experiments assuming the Selective Thread Tracking Algorithm 1 is followed. Nevertheless, to facilitate this process, Prism provides an interface with which one can navigate the collected metrics, and templated queries.

Artifact Description (AD)

Artifact Setup (incl. Inputs)

I. OVERVIEW OF C ONTRIBUTIONS AND A RTIFACTS

Hardware: The artifact has no strict hardware requirements, although adequate computational resources are recommended for replication purposes. Our setup, however, consisted of a i7-11850H Intel server @ 2.50 GHz running Ubuntu 22.04 LTS, Jammy Jellyfish (x86_64) OS with 6.8.12 Linux kernel, 8 CPU cores, 2×16 GB DDR4 RAM, and 1 TB SSD disk. Software: The artifact was evaluated on a machine with the Linux 6.8.12 kernel (with BPF/BTF enabled). Additionally, the artifact also requires installing Docker and the Nix package manager. To fully reproduce our configuration, we installed Docker 27.5.1 and Nix 2.28.4. The remaining software dependencies, and their respective versions are pinned by the provided flake.nix and flake.lock files in the root directory of the artifact. Datasets / Inputs: The artifact’s flake.nix and flake.lock files automatically manage the dataset and input dependencies for the experiments. As such, these are downloaded and resolved when an experiment’s run script is executed. Installation and Deployment: Some experiments (e.g., Cassandra and Kafka) require a running Docker daemon. Similar to the previous point, the remaining installation and deployment dependencies are managed and pinned by the flake.* files in the artifact’s root directory.

A. Paper’s Main Contributions C! : A comprehensive set of metrics that capture an application’s thread dynamics; C2 : A metric collector (Prism) that gathers the metrics listed in Table I and stores them in a structured database for analysis; C3 : A user interface with templated queries that enable exploration of an application’s thread dynamics through the metrics database. B. Computational Artifacts A1 : https://github.com/EC-labs/ipdps2026-prism-artifact/ releases/download/v1.0.0/archive.tar Artifact ID A1

Contributions Supported

Related Paper Elements

C 1 , C2 , C 3

Figures 6-14

II. A RTIFACT I DENTIFICATION A. Computational Artifact A1 Relation To Contributions

Artifact Execution

This artifact A1 provides the instructions and data necessary to reproduce the analysis in Sections V and VI pertaining to C1 -C3 .

Each of the six experiments includes the following stages: (1) Setup: Downloading required Docker images and deploying the target application. (2) Execution: Starting and bootstrapping Prism’s metric collector on the application, and managing workloads and degradation scenarios. (3) Postprocess: Raw KPI metrics (e.g., per-request latency) may be post-processed into more informative metrics (e.g., 95th percentile latency). (4) Cleanup: Experiment services and resources are terminated. (5) Analysis: The metric collector’s

Expected Results Assuming similar computational resources to those indicated in Section V, the application should illustrate similar thread dynamics and degradation patterns to those presented in Section VI.

15

data and target KPI metric are uploaded to Prism’s UI for analysis.

The dependencies include standard packages (e.g., docker, bash), Prism’s metric collector (prism-mc), and workload packages (packages.{redis,memtier}). The nix run command ensures these are available at runtime. The Redis execution script at redis/run.sh begins with the setup stage, downloading the Redis image and starting a container.

Artifact Analysis (incl. Outputs) Analysis requires uploading the *.db3 metric database and target KPI metric files to Prism’s UI, which then provides preexisting analyses like a process-level dependency map showing processes interacting with the bootstrapping process. A Debug page allows executing any query against the backing *.db3 database. To facilitate Selective Thread Tracking, we provide templated queries in the UI’s analysis/src/sql directory. Each query can generate line, histogram, or bar plots if the resulting data supports the plot type.

$ docker pull redis:7.4.2 $ docker run # ... # redis:7.4.2

The execution stage bootstraps the Prism metric collector on the Redis processes and manages starting the workload generators to stress the server: $ metric-collector # ... --pids <pid-list> $ memtier_benchmark # ... --test-time 120 $ redis-benchmark # ...

Artifact Evaluation (AE) The artifact evaluation uses the Redis experiment as an example; the process for other experiments is similar. We assume the dependencies listed in the Artifact Description (i.e., Nix, Docker, and a recent Linux version with eBPF enabled) are already installed.

The post-processing stage computes the 95th percentile latency from the per-request latency generated by the memtier benchmark: $ percentile-ps # ... --percentile 0.95

Finally, on script exit, the cleanup function is executed to terminate the resources created throughout the experiment, i.e., the redis server and Prism’s metric collector:

A. Computational Artifact A1 Artifact Setup (incl. Inputs) Start by downloading and unpacking the provided artifact, and changing into the unpacked archive’s directory:

$ sudo kill -SIGINT "$PRISM_PID" $ docker stop redis

$ mkdir archive && \ tar -xvf archive.tar -C archive && \ cd archive

Artifact Analysis (incl. Outputs) To analyse the data, start the analysis interface:

The archive root contains folders for each experiment in Section V, each with its dependency management and run scripts. It also includes flake.nix and flake.lock, which pin the software dependencies.

$ NIX_CONFIG=’experimental-features = nix-command ,→ flakes’ nix run github:ec-labs/prism#analysis

Artifact Execution Run an experiment with nix run, using the flake in the repository root, select an experiment, and pass a directory for storing results. For example, the following command runs Redis’ experiment and stores results in ./results: (a)

$ NIX_CONFIG=’experimental-features = nix-command ,→ flakes’ nix run .#redis-experiment ./results

(b)

(c)

Fig. 1: User interface screenshots: (a) KPI page; (b) Ripple page; (c) Debug page.

Root privileges are required to start and terminate Prism’s eBPF collector, so the user is prompted for the root password after setup. The aforementioned command fully executes the Redis experiment. The following description explains the script’s dependencies and activities; the listed commands are for documentation only and should not be executed. The software dependencies for Redis’ execution script are listed in redis/default.nix. A subset is shown below:

On the interface’s homepage http://localhost: 8501, upload the *.db3 database file from the ./results/<experiment> directory. Then, on the KPI page, upload the target metric file. For Redis experiment, selecting baseline and comparing periods displays a graph with highlighted periods, as in Figure 1a. The baseline and compare periods serve as template variables. The Ripple page shows a process-level dependency graph. Specifying redis as the bootstrapping process yields a graph like Figure 1b. Lastly, the Debug page allows executing custom/templated queries. Running the pid runqueue share query with redis’ pid as pid_filter (e.g., (pid=707523)) and selecting histogram plot yields a plot like Figure 1c.

runtimeDeps = with pkgs; [ bash docker prism-mc packages.redis packages.memtier # ... ];

16