Towards Runtime-Aware Observability for Java Virtual Threads

Towards Runtime-Aware Observability for Java Virtual Threads

arXiv:2611.14999v1 [cs.PF] 11 Nov 2026

WENJIE DING, School of Computing, Newcastle University, United Kingdom OS-level observability tools join application requests to kernel events through the OS thread ID, assuming that a unit of work occupies one OS thread until it completes. Java virtual threads break this assumption: many virtual threads are multiplexed onto a few carrier threads, so a stable thread ID hides a changing unit of work, and the join fails silently. Existing virtual-thread-aware channels republish context or identity on every mount and unmount, and the JVMTI-based one pays this cost for the lifetime of the JVM, whether or not an observer is attached. We bind a 64-byte off-heap trace buffer to each virtual thread, at a fixed address for its lifetime, and publish the address to eBPF consumers through four USDT probes in a patched OpenJDK; a re-implemented JVMTI channel over the same buffer serves as a controlled comparison. The four-probe design attributed all 768 test markers correctly on both the content and the identity plane, while the naive two-probe design silently misattributes trace context. With no consumer attached, publication costs at most about 33 ns per transition; the remaining dormant cost is buffer allocation, about 0.37 𝜇s per virtual thread, which is negligible on transition-heavy workloads but nearly doubles the cost of a creation-only microbenchmark. The JVMTI channel costs about 20% on transition-heavy workloads and about 1.8 𝜇s per virtual thread on the creation-only one. We also show that, of the two update strategies OTEP 4947 permits, only in-place rewriting guarded by the record’s valid flag is sound for readers that cache the record’s address, as lifecycle-event readers do.

1

Introduction

1.1

Motivation

Observability data comes from two layers that see different things. The application knows which request it is serving; the operating system knows what that work costs in CPU time, blocking and I/O. Neither can answer on its own the question operators actually ask: which request caused this CPU consumption, or this blocking? Answering it requires a join: attaching application-level request information to events observed at the OS level. In practice, the join key is the OS thread ID. Profilers aggregate samples by thread ID, eBPF programs use it as their map key, and zero-code tracers such as DeepFlow, Grafana Beyla and vPath correlate spans by it [19]. The key is correct only if each unit of work occupies one OS thread exclusively until it completes, an assumption we call one-execution-per-thread. No specification states it, yet tools rely on it widely, so when it breaks, the join fails silently. OpenTelemetry’s eBPF instrumentation shows what this looks like in practice: with virtual threads enabled, 63% of downstream spans lost their parent request, and 19% of log lines carried another request’s trace ID [16, 17]. Java virtual threads [20] break this assumption by design. They let a server keep the thread-perrequest programming model while serving far more concurrent requests than it has OS threads, by scheduling many virtual threads onto a few carrier threads (𝑀 : 𝑁 scheduling). To the OS, a carrier’s thread ID stays the same while the unit of work it runs keeps changing. This is not a defect but the source of the performance benefit, since switching between virtual threads involves no kernel transition; making virtual threads visible to the OS would defeat their purpose. Existing and emerging standards let a runtime publish its trace context to outside readers, but only at process or OS-thread granularity [1, 9]. Virtual-thread-level attribution has nevertheless been achieved: Elastic uses a JVMTI agent [7], and OpenTelemetry’s eBPF instrumentation has since tracked virtual-thread mounts through bytecode instrumentation [15]. Both pay on the Author’s Contact Information: Wenjie Ding, wenjie.ding77@gmail.com, School of Computing, Newcastle University, Newcastle upon Tyne, United Kingdom.

2

Wenjie Ding

mount/unmount path, so the question is no longer whether attribution is possible, but what it costs. We ask three questions: Q1 Can the mapping from carrier thread to virtual thread be reconstructed at the OS level without the JVMTI capability? Q2 Can this be free while no consumer is attached? Q3 What determines the overhead of each publication channel? To answer them, we modified OpenJDK to publish a per-virtual-thread trace buffer through virtualthread lifecycle events consumed by an eBPF program, and re-implemented the existing JVMTIbased channel over the same buffer protocol as a controlled comparison. 1.2

Contributions • A characterisation of the correlation problem under 𝑀 : 𝑁 scheduling. We apply soundness and completeness to virtual-thread attribution and evaluate it on two separate planes (Section 4.2): the content plane (is the trace context correct?) and the identity plane (is the event attributed to the right virtual thread?). A trace context can be correct while attributed to the wrong virtual thread, and a single-plane check accepts such cases. • A protocol-level finding about OTEP 4947. OTEP 4947 lets writers update the published context in either of two ways: by swapping the record pointer, or by rewriting a fixed record in place, guarded by the record’s valid flag. Both are safe under the reader model the OTEP implicitly assumes, in which readers re-resolve the record’s address on every read. A virtual thread has no OS-resolvable identity, so a reader must either rely on the runtime republishing a carrier-side pointer on every transition, or cache the address it learns from lifecycle events, as ours does. We make this assumption explicit and show that, for caching readers, only in-place rewriting is sound: a pointer swap leaves the cached address pointing at a stale or released record (Section 5.2). Our experiments observe exactly this failure: a reader holding a stale address returns a well-formed context from a released buffer (Section 7.2). • A mechanism and a decomposition of its cost. We bind an off-heap buffer to the lifecycle of each virtual thread in OpenJDK and publish its address through four USDT probes. We show that the naive two-probe (freeze/thaw) reconstruction is not merely incomplete but unsound: it silently misattributes trace context. A second channel over the same buffer, built on JVMTI, isolates the capability tax. While dormant, our publication path costs at most about 33 ns per transition, whereas the JVMTI channel costs about 90–105 ns per transition (around 20% on transition-heavy workloads) and about 4.6× our channel’s per-thread cost on a creation-only workload (Section 7).

The rest of this paper is organised as follows. Section 2 introduces virtual threads, OpenTelemetry trace context, USDT/eBPF and JVMTI, and Section 3 reviews existing approaches to exposing trace context. Section 4 characterises the correlation problem and defines the correctness criteria. Sections 5 and 6 present the design and implementation of our mechanism, and Section 7 evaluates its correctness and cost against the JVMTI channel. Section 8 discusses trade-offs and limitations, and Section 9 concludes. 2 2.1

Background Virtual Threads

Virtual threads were introduced in Java 21 [20] to preserve the thread-per-request style of server applications while decoupling application concurrency from the number of OS threads. The JVM

Towards Runtime-Aware Observability for Java Virtual Threads

3

schedules many virtual threads (𝑀) onto a small pool of carrier threads, which are ordinary platform threads (𝑁 ). Attaching a virtual thread to a carrier is called mounting; detaching it, unmounting. When a virtual thread parks, for example on a blocking operation, the runtime copies its stack frames to the heap (freeze) and unmounts it; when it is resumed, the frames are copied back onto a carrier (thaw). Freeze and thaw are implemented once, in the HotSpot C++ runtime, and all three execution engines (the template interpreter and the C1 and C2 compilers) converge on them. Two transitions bypass them: the first mount of a virtual thread performs no thaw, since nothing has been frozen yet, and the final unmount, when the task completes, performs no freeze. 2.2

OpenTelemetry and Trace Context

A trace is the collection of all work done to serve a single request; each unit of work within a trace is a span [21]. Request-level tracing has a long line of systems work, from X-Trace and Dapper to Pivot Tracing and Canopy [8, 12, 13, 21]; this work addresses one link in that chain, the join between OS-level events and the request executing on a thread. OpenTelemetry, a vendor-neutral observability framework that evolves through OpenTelemetry Enhancement Proposals (OTEPs), follows the W3C Trace Context specification [24], whose core fields are the trace ID and the span ID. The OpenTelemetry SDK keeps a current context, including the current span, for each thread, and all changes to it go through the ContextStorage interface. OTEP 4947 [9], merged in July 2026, specifies how an SDK publishes the active trace context to external readers. The SDK maintains a record for each thread, holding the trace ID, the span ID and a valid flag, and publishes a pointer to it in an ELF (Executable and Linkable Format) TLS (Thread-Local Storage) variable, so that an outside reader, such as the OpenTelemetry eBPF profiler [14], can locate a thread’s record through a known symbol. Writers may update the context either by swapping the TLS pointer to a different record, or by rewriting a fixed record in place while its valid flag is cleared. The protocol requires readers to read only while the target thread is stopped or interrupted, and implicitly assumes that they resolve the address afresh on every read. 2.3

USDT, Tracepoints and eBPF

A User Statically-Defined Tracing (USDT) probe is an instrumentation point compiled into a program, described by an ELF note that records its name, location and arguments [3]. While no tracer is attached, the probe site is a NOP; attaching a tracer replaces it with a trap into the kernel. A kernel tracepoint is the kernel-side counterpart: it is defined independently of the observed application and fires in the context of the calling task, so it sees the task’s thread ID but no application state [10, §2.9]. eBPF programs can be attached to both, USDT probes through uprobes, and keep state across events in key–value maps. An eBPF handler runs synchronously in the context of the task that triggered it: until the handler returns, that task executes no user-space code [10, Ch. 2]. 2.4

JVMTI and Virtual Threads

The JVM Tool Interface (JVMTI) [18] lets a native agent, running inside the JVM process, request capabilities and receive callbacks for registered events; Elastic’s implementation is built on it (Section 3.2). For virtual threads, JVMTI support has a cost that does not depend on what the agent does. In HotSpot (JDK 21), as soon as any JVMTI environment is created, whether by an agent loaded at start-up or by one attached later, every mount and unmount is routed through a notifying transition path, and in production this is never switched off again; the HotSpot source itself notes

4

Wenjie Ding

that this has a performance impact.1 The cost is therefore paid on every transition, whether or not an observer is attached. We call it the capability tax. 3 3.1

Related Work Observability from Inside the Runtime

Some runtimes already attribute work to their user-level threads, but with the consumer inside the runtime. JDK Flight Recorder (JFR) [11] attributes its own events, such as I/O, allocation, and virtual-thread events like jdk.VirtualThreadStart and jdk.VirtualThreadPinned [20], directly to virtual threads, without needing the carrier. Its producer and consumer both live inside the JVM, and its consumer path is asynchronous. Our consumer, by contrast, sits outside the process, must attribute each event at the moment it occurs, and needs the carrier thread ID as the bridge to OS-level events (Section 4.1). Go’s goroutines [22] are the closest analogue to virtual threads. Go maintains goroutine profiler labels that profiling infrastructure can associate with execution [23], and an external reader can start from the OS thread and walk the runtime’s own structures to the running goroutine’s labels, which is how eBPF profilers such as Parca attribute trace context to goroutines [2]. Java offers no such path: starting from a carrier thread, there is no externally resolvable location for the mounted virtual thread’s context. The closest candidate, the VirtualThread object itself, can be moved by the garbage collector, so an external reader cannot cache its address. Our mechanism provides exactly this missing piece: an off-heap address that is stable for the virtual thread’s lifetime. 3.2

The Elastic JVMTI Implementation

The Elastic Distribution of OpenTelemetry Java [7] attributes trace context to virtual threads through a JVMTI agent that registers the VirtualThreadMount/Unmount extension events. Every thread, platform or virtual, has an off-heap context buffer; on each mount, the agent writes the virtual thread’s buffer address into a TLS slot of the carrier thread, and on each unmount it clears the slot. This is a pointer update rather than a copy of the record, and it preserves the resolve-at-read model: carriers are OS threads, so an OS-side reader resolves their TLS exactly as it would for platform threads. The price is that visibility must be republished on every mount and unmount, and that the agent pays the capability tax (Section 2.4) whether or not an observer is attached. Both costs fall on the transition path that virtual threads are designed to make cheap: the more often a workload transitions, the more it benefits from virtual threads, and the more it pays. 3.3

OpenTelemetry eBPF Instrumentation

Concurrently with this work, OpenTelemetry eBPF Instrumentation (OBI) met the same problem from the reader’s side [16]. Its Java agent keyed in-process trace context by kernel thread ID; with virtual threads enabled, the carrier changed between a handler’s entry and its outgoing call in 91.5% of requests, and 63.3% of downstream spans lost their parent. The fix [15] uses bytecode instrumentation (ByteBuddy) at the exit of VirtualThread.mount() and unmount(): on every transition, the agent reports the mounted virtual thread’s ID to eBPF through an ioctl, and eBPF keeps a carrier-to-virtual-thread map. Trace-context publication through the OTEP map is nevertheless suppressed while a virtual thread is mounted: keeping it would have left 48% of log 1 JvmtiExport::get_jvmti_interface in src/hotspot/share/prims/jvmtiExport.cpp (JDK 21): “There is a perfor-

mance impact when VTMS transitions are enabled.” The only function that disables them is a WhiteBox testing hook, documented as unsafe while virtual threads are running.

Towards Runtime-Aware Observability for Java Virtual Threads

5

Table 1. Three levels of trace-context exposure. Level

Bound to

Address discovery

Attributable unit

Process (OTEP 4719) process named mapping, /proc service Thread (OTEP 4947) platform thread symbol, resolve-at-read OS thread Virtual thread (this work) vthread lifetime lifecycle event, cached unit of work

lines with a wrong trace ID, and the project chose no data over wrong data [17]. Restoring it remains an open issue, and the cost of refreshing context on every transition has not been measured. OBI is therefore a third publication channel, alongside JVMTI and our USDT probes. Like the JVMTI channel, it pays on every mount and unmount, here with a system call, and it reports correctness but no overhead measurements. Its experience also confirms, on a production tracer, the failure mode our correctness criteria are designed to catch: wrong attribution that looks like valid data (Section 4.2). In short, virtual-thread-level attribution is already achievable, but both existing channels pay on every mount and unmount, and the JVMTI channel pays its tax for the lifetime of the JVM, whether or not an observer is attached. This work asks whether that cost can instead be made conditional on an observer being present. 3.4

Trace Context Exposure: A Three-Level Framework

We compare approaches to exposing trace context along two axes (Table 1). The first is the identity the context is bound to: a process, an OS thread, or a logical unit of work. The second is how a reader obtains the context’s address: by resolving it on every read (resolve-at-read, typically through a symbol lookup), or by receiving it once from the runtime and caching it (publish-once-and-cache). The second axis is rarely discussed, because for platform threads the two models make no practical difference; for virtual threads, they do (Section 5.2). OTEP 4719 [1] binds context to a process, publishing it in a named memory mapping that readers locate through /proc/<pid>/maps; it can tell which service an execution belongs to, but not which request. OTEP 4947 binds context to an OS thread: readers locate each thread’s record in ELF TLS through a known symbol. It does not require runtimes for which OS-thread-local context is the wrong execution-context model to implement this mechanism; such runtime-specific discovery mechanisms are left out of scope, but the OTEP recommends that they reuse its record format. A virtual thread has no OS-resolvable identity, so resolve-at-read can reach it only through its carrier, and the runtime must then republish on every mount and unmount: a buffer pointer in Elastic’s case, the virtual-thread ID in OBI’s. Our mechanism is the runtime-specific alternative the OTEP leaves room for: it reuses the record format and replaces TLS lookup with lifecycle events and cached addresses. The three levels are not a ranking: each serves a different reader model, and resolve-at-read is the right choice for the readers OTEP 4719 and 4947 target. 4 4.1

Problem Characterisation The Correlation Problem under M:N Scheduling

At the OS level, an event carries exactly one execution identity: the thread ID (tid). Attributing it to a unit of work takes two steps, event (tid, 𝑡) → vthread → trace context, first determining which virtual thread was running on the carrier at time 𝑡, then reading that virtual thread’s trace context at that moment. Under one-execution-per-thread, the first step is

6

Wenjie Ding

trivial: the tid itself is the answer. Under 𝑀 : 𝑁 scheduling, the answer changes over time, and virtual-thread scheduling produces no event the OS can see, such as a system call, a context switch or a scheduler tracepoint. A consumer can therefore hold an expired mapping without any way of knowing it is stale. The two steps also depend on different components: the first on the runtime, which knows what is mounted, and the second on the observability framework, which maintains the trace context. Each can fail independently, which motivates evaluating them separately. 4.2

Correctness Definitions

C1: Attribution. Every OS-level event that user code causes on a carrier thread must be attributed to the virtual thread that was running at that moment, with the trace context that was current in Java at that moment (the ground truth). C2: Separate planes. Trace-context correctness (the content plane: trace ID and span ID) and virtual-thread attribution (the identity plane) must be evaluated separately. They are maintained by different components: the observability framework writes the trace context, while the observed runtime provides the virtual-thread identity. Checking the content alone cannot distinguish a mechanism that works from one that fails but happens to return the right context. C3: Soundness and completeness. A mechanism is complete if every such event receives an attribution, and sound if every attribution it makes is correct. An incomplete mechanism fails loudly: the consumer sees that an event is unattributed. An unsound mechanism fails silently: a misattributed record carries a trace ID and span ID that are valid in both format and content, and nothing in the record reveals that it belongs to another virtual thread. For observability, unsoundness is therefore more dangerous than incompleteness. Such failures are not hypothetical: a stale mapping can still resolve to a well-formed record (Section 7.2). 5

Design

OTEP 4947’s symbol-based TLS lookup requires a thread identity the kernel can see, which virtual threads deliberately lack (Section 2.1). This constraint narrows the design space to two representative designs: JVMTI event callbacks, and USDT lifecycle probes consumed by an eBPF reader. 5.1

The Root Design Question: Where the Buffer Address Lives

The first decision is whom the per-thread trace buffer belongs to: the carrier thread or the virtual thread. • Carrier (Elastic-style). An OS-side reader resolves the carrier’s TLS as usual, but the buffer address must be republished whenever a virtual thread mounts, so the total publication work grows linearly with the number of mount/unmount transitions, on top of the JVMTI capability tax. • Virtual thread. The off-heap buffer lives exactly as long as its virtual thread. The consumer then only needs to know which virtual thread is running on which carrier, which can be derived from existing virtual-thread lifecycle events. We choose the second option. Its cost is that OS-side consumers must become event-driven and adopt the publish-once-and-cache model. 5.2

Trace Buffer Design

OTEP 4947 implicitly assumes that each sampling operation locates the record through the thread’s TLS pointer and reads it immediately; it does not consider readers that cache the record’s address. The protocol’s correctness rests on two preconditions: the read happens only while the target

Towards Runtime-Aware Observability for Java Virtual Threads

7

thread is stopped or interrupted, which the OTEP requires explicitly, and the address is resolved afresh on every read, which it assumes implicitly. For virtual threads, keeping resolve-at-read means republishing a carrier-visible TLS pointer on every transition, which adds a cost to every mount and unmount (Section 7.4). Our design instead publishes the buffer address through USDT probes, and readers cache it. Producer and consumer remain unaware of each other: the virtual thread does not know whether a reader exists, and the reader learns about the buffer only through lifecycle events. The reader model can now be stated precisely: an eBPF consumer receives the buffer address once per virtual-thread lifecycle event and keeps it for an arbitrary duration, from one thaw to the next freeze, while the buffer’s contents may be rewritten any number of times without re-resolution. Every design decision that follows is a consequence of this model. OTEP 4947 allows two strategies for updating the record and leaves the choice to the writer: pointer swap and in-place mutation. A pointer swap replaces the pointer rather than the record it points to, so the publication is atomic, and a reader that resolves the address on every read never sees a stale record. A caching reader, however, keeps the old address, and every subsequent read silently returns the stale context; if the freed record is reallocated to another virtual thread, the reader even returns that thread’s context. In-place mutation keeps the record’s address stable for the virtual thread’s entire lifetime, so a cached address remains valid for as long as the buffer exists. This is exactly what a caching reader needs, but it moves the consistency problem into the buffer: the reader may observe a record in the middle of a rewrite. The general rule is that any signal placed outside the reader’s access path is invisible to a caching reader. The validity signal must therefore sit where the reader already looks, inside the record itself: the valid flag. Of the two strategies the OTEP permits, only in-place mutation is sound for caching readers. Can a reader observe a torn record when a read overlaps a write? Three properties rule this out or make it detectable. First, the writer follows the OTEP 4947 in-place rewrite protocol, valid=0 → release fence → payload → release fence → valid=1: the first fence orders the cleared flag before any payload write, and the second orders the complete payload before the flag is set, so the design keeps the correctness the OTEP guarantees for its own reader model. Second, both of our readers, the USDT handler and the system-call tracepoint handler, run synchronously in the context of the writer’s own carrier task, not in another process or thread. While a handler runs, the task is in the kernel and cannot execute user-space writer code, so reads and writes are ordered by the task’s own program order. Third, this same-task argument requires that the task not enter the kernel between valid=0 and valid=1, which JVM machinery such as safepoints could violate. If it does, the reader sees the flag at 0 and reports the event as invalid: an honest, detectable failure in the sense of C3 (Section 4.2). The flag thus signals both a rewrite in progress and, while no context is attached, the absence of context. The buffer layout (Table 2) follows OTEP 4947 with one exception: our prototype was built against an earlier draft in which byte 25 was reserved, whereas the merged OTEP defines it as trace-flags, which our prototype does not populate. The fixed 64 B size follows the OTEP’s suggestion that a frugal writer keep the whole record within a 64-byte cache line. The trace ID and span ID are stored byte by byte in W3C (network) byte order. 5.3

USDT Probe Placement: Constraints, Dead Ends, and Four-Probe Decomposition

The semantically natural place for the probes would be the mount and unmount boundaries themselves. We place them deeper in the runtime instead, so the implementation location differs from the semantic one. The reason is that a USDT probe needs native code on the execution path, and the question becomes where such code already exists, and what it would cost to add it where it does not (Table 3).

8

Wenjie Ding

Table 2. Trace buffer layout (64 B per virtual thread), following OTEP 4947 [9]. Bytes

Field

Content

0–15 16–23 24 25 26–27 28–63

traceId spanId valid trace-flags attrs-data-size attrs-data

W3C trace ID, network byte order W3C span ID, network byte order 1: record complete; 0: being rewritten or no context W3C trace flags (reserved in the draft we implemented; not populated) length of attrs-data encoded attributes (not used in this work)

Table 3. Logical transitions and their USDT probes. Logical Transition

Native Code on Path

USDT Probe

remount (2nd to N-th time) thaw—C++ runtime, shared by all three engines vthread__thaw non-final unmount (park) freeze—same as above vthread__freeze first mount enterSpecial stub enters Java directly, no thaw vthread__start final unmount (termination) continuation returns normally, no freeze vthread__end

Remounts and non-final unmounts are cheap to cover: freeze and thaw are independent of the execution engine, so probes placed there fire regardless of JIT compilation state. The first mount and the final unmount, in contrast, run in JIT-compiled Java code with no native call site on the path; this is not a defect but precisely why virtual threads are cheap. Introducing a native site there has three options, each with a cost: (1) A JNI call on every mount, which adds a native transition to every mount, so the total work grows with the number of mounts. (2) A new intrinsic, which would have to be implemented in the template interpreter, C1 and C2: too invasive for a tracing feature. (3) Piggybacking on the existing JVMTI hooks. notifyJvmtiMount/Unmount are intrinsic candidates guarded by a flag, and when the flag is off, C2 compiles them away. A probe placed in their native implementations would fire under the interpreter but become unreachable once the caller is JIT-compiled; turning the flag on brings back the capability tax. The way out is that the first mount and the final unmount each happen only once per virtual thread. A native call at these two points therefore costs 𝑂 (1) per virtual-thread lifetime, and we add the start and end probes there. 5.4

Consumer Model and Event-Level Equivalence

The consumer is event-driven: it keeps a map from carrier thread ID to the virtual thread currently mounted on it. The map’s size is bounded by the size of the carrier pool. Event-level equivalence. All the consumer needs to know is which virtual thread runs on which carrier, that is, when virtual threads mount and unmount. The stock JVM does not expose these transitions, but they can be reconstructed precisely from the four probes: • Mount = {start}∪{thaw(kind=top)}: start covers the first mount, and kind=top selects exactly one event per logical remount. • Unmount = {freeze(result=ok)} ∪ {end}: result=ok corresponds to a successful park, and end to termination.

Towards Runtime-Aware Observability for Java Virtual Threads

9

• freeze(result=pinned_*) ̸→ unmount: the virtual thread is pinned and stays on its carrier, so the mapping is kept. • thaw(kind=return_barrier) ̸→ any transition: it is an internal frame-reconstruction step within a single logical remount and must be ignored. Insertion and deletion rules. The consumer inserts a mapping only on start and thaw(kind=top); return-barrier thaws neither insert nor delete. We observed return-barrier thaws even after a virtual thread had terminated, so inserting on every thaw would reintroduce a stale mapping and, with it, silent misattribution. On end, the consumer deletes the mapping unconditionally, since the semantics of end are exact. Two counters in the consumer check these rules at run time: they count freeze and end events that find no mapping to delete. In all six four-probe correctness runs (Section 7.2), both counters stayed at zero, and the 128 insertions in each run were matched by exactly 128 deletions. Window boundaries. The reconstructed mount window deviates slightly from the semantic one, in opposite but safe directions. For a remount, the window is slightly wider: thaw fires before the Java-side mount completes. At the two lifecycle endpoints, it is slightly narrower: start fires after the first mount has completed, and end fires before run() returns, while the final unmount happens afterwards. In both directions, the mismatch covers only JVM-internal execution, and user code runs strictly inside the reconstructed window. At the edges, an event is either attributed to the correct virtual thread or explicitly left unmapped, never attributed to another virtual thread. 5.5

Soundness and Completeness Analysis

Our first design used only the two freeze/thaw probes, and it achieves neither property of C3 (Section 4.2). Completeness is lost at the first mount, which fires no thaw, and soundness is lost at termination, which fires no freeze and therefore leaves a stale mapping behind. Section 7.2 confirms both failures experimentally. 6

Implementation

The implementation has three parts: the JDK allocates a buffer for each virtual thread and publishes its address (Sections 6.1–6.3), an eBPF consumer attributes OS-level events through it (Section 6.4), and an OpenTelemetry SDK plugin writes trace context into it (Section 6.5). Figure 1 shows the buffer’s lifecycle and the events that publish it. 6.1

Buffer allocation

Each virtual thread gets a 64 B off-heap buffer, whose address is stored in a new field, VirtualThread. traceBufferAddress. We implemented two ways of allocating it: by the JVM itself, or by a JVMTI agent from outside the JVM. Section 7.4 compares them. 6.1.1 JVM allocation. We modified OpenJDK2 so that it allocates the buffer itself. The allocation points follow one rule: the buffer’s lifetime must strictly contain every interval during which a consumer holds its address, or the consumer may read freed memory. The buffer is therefore allocated in VirtualThread.start() and freed in afterDone(). 6.1.2 JVMTI agent. The JVMTI channel leaves the JVM’s allocation path untouched: an agent allocates the buffer in its VirtualThreadStart callback, writes the address into the same traceBuffer Address field through JNI, and frees the buffer in VirtualThreadEnd; the JVM dispatches both 2 The modification adds 157 lines and removes 2, in 12 files across the Java and C++ sides of OpenJDK, on top of jdk21u

commit 57b3005fc5, branch vthread-lifecycle-probes [5].

10

Wenjie Ding time

parked

mounted on carrier 𝐴

allocate buffer start()

Java side

run(task) begins

mounted on carrier 𝐵 resumes

task running

native call; reads addr

start

freeze

(vid, addr)

(vid, result)

invisible to OS

task returns

thaw

end

(vid, kind)

(vid)

free buffer afterDone()

C++ side

trace buffer

kernel · eBPF

off-heap, 64 B, address fixed: +0 traceId(16) · +16 spanId(8) · +24 valid(1) · OTEP 4947 insert tid𝐴 ↦→ ⟨ vid, addr ⟩ delete (ok)

buffer lifecycle (patched JDK)

insert (kind = top) tid𝐵 ↦→ ⟨ vid, addr ⟩ delete (by tid)

USDT probes (-XX:+VThreadTraceProbes)

eBPF map state

Fig. 1. Lifecycle of a virtual thread’s trace buffer: allocation and release on the Java side, the four USDT probes in HotSpot, and the eBPF map state they drive as the thread parks on carrier 𝐴 and remounts on carrier 𝐵.

callbacks from its existing notifyJvmtiStart/End hooks. The agent is our own re-implementation of the Elastic-style design [7], not Elastic’s code: it requests only the can_support_virtual_ threads capability and, in its default mode, registers only the VirtualThreadStart/End events, besides VMInit. An optional publish=jvmti mode additionally registers the VirtualThread Mount/Unmount extension events and, as Elastic does, writes the mounted virtual thread’s buffer address into a carrier-thread TLS slot on every transition [4]. The agent shares the buffer protocol with the JVM-allocation path and differs only in the notification and allocation channel, so the comparison isolates the capability tax defined in Section 2.4. 6.2

USDT Probes

To keep the modification small, we added the four probes to HotSpot’s existing hotspot.d DTrace provider, so they behave like HotSpot’s other probes; Appendix B lists their definitions. The freeze and thaw probes fire in HotSpot’s shared C++ continuation code (continuationFreezeThaw.cpp). The start and end probes fire from Java: on entering and leaving run(task), VirtualThread calls a native method that fires the probe, once per virtual thread. These native methods are deliberately not intrinsic candidates, so the JIT cannot remove the calls (Section 5.3). Their position also preserves the nested-lifetime rule for both allocation channels: start fires after the JVMTI start callback, so an agent-allocated address is already in place, and end fires before the JVMTI end callback, so the consumer drops its mapping before an agent frees the buffer. 6.3

Gating

Two switches, both fixed at JVM start-up, control the mechanism: the HotSpot product flag -XX:+VThreadTraceProbes enables the four USDT probes, and the system property -Dvthread. trace.jvmAlloc=true enables buffer allocation. Configurations S0–S2 in Table 4 are combinations of the two. We first gated the probes with SDT semaphores, the usual USDT mechanism, but they cannot be used in Linux HotSpot. The build never defines the semaphore variables, so the JVM fails to link, and HotSpot’s header include order records a semaphore address of zero in every probe note, so no tracer could enable them. HotSpot gates its own probes with product flags as well, and we follow this convention.

Towards Runtime-Aware Observability for Java Virtual Threads

11

Like the JVMTI capability, the flag cannot be changed while the JVM is running; the difference is what it costs to leave on. With the flag on and no tracer attached, the probe sites are NOPs, and the cost (S2−S1) is below the measurement’s resolution: at most about 33 ns per event (Section 7.3). Enabling the JVMTI capability alone, by contrast, costs about 20% on transition-heavy workloads (Section 7.4). The flag can therefore stay on by default, and the cost of publication then depends only on whether a tracer attaches. This is the precise sense in which the cost of our mechanism is conditional on an observer being present. 6.4

Consumers

To attribute an OS-level event, the consumer needs the event’s carrier thread ID. As a representative OS-level event, we use the tracepoint:syscalls:sys_enter_write tracepoint: it fires on the OS side, independently of the JVM, and identifies only the carrier thread (tid), not the virtual thread that issued the call. The consumer maintains a map from carrier thread ID to the mounted virtual thread’s ID and buffer address, following the rules of Section 5.4. Its correctness rests on two invariants: a carrier maps to at most one virtual thread at any time, which the insertion and deletion rules guarantee, and each mapping’s lifetime lies within its buffer’s lifetime, which the allocation points of Section 6.1 guarantee. When sys_enter_write fires, the consumer looks up the carrier’s entry, reads the valid flag at the cached address, and either discards the record or reads the trace ID and span ID at their fixed offsets. It never writes to the buffer. 6.5

OpenTelemetry SDK Integration

The buffer is written by the OpenTelemetry SDK through a ContextStorage decorator [4]. All reads and writes of the SDK’s current context go through the ContextStorage SPI, so a decorator registered as an SDK plugin sees every change without any modification to OpenTelemetry itself; its only requirement is our patched JDK, whose Thread.getTraceBufferAddress() returns the buffer’s address. The decorator follows one rule: after any change to the current context, such as attaching a span or closing a scope, the buffer reflects the new current context. Closing a scope restores the previous context, which the decorator republishes through the same path, so nested scopes need no special case: closing an inner scope republishes the outer span. If the restored context has no valid span, the decorator leaves the valid flag at 0. Each update follows the in-place rewrite protocol of Section 5.2. The publication path needs no changes to OpenTelemetry’s source code, no bytecode weaving and no -javaagent: it depends only on a library-level SPI. The mechanism thus splits into three decoupled parts: the JDK (or a JVMTI agent) allocates the buffer and manages its lifetime, the USDT probes expose its address, and the SDK plugin writes trace context into it. The three communicate only through the traceBufferAddress field, which is why the two allocation channels of Section 6.1 are interchangeable: the writer cannot tell which one allocated the buffer. 7

Evaluation

We evaluate the mechanism against the three questions of Section 1.1: whether it attributes OS-level events correctly (Q1, Section 7.2), what it costs while no consumer is attached (Q2, Section 7.3), and what determines the cost of each publication channel (Q3, Sections 7.3 and 7.4). 7.1

Setup

Configurations. We decompose the mechanism’s cost into four configurations (Table 4). We report differences between configurations rather than absolute times: each difference isolates one component, and the baseline cost common to both configurations cancels.

12

Wenjie Ding

Table 4. Configurations used to decompose the mechanism’s cost. Each adds one component to the previous one, so each difference between adjacent configurations isolates a single component. S0, the patched JDK with both switches off, is the reference: the Java-side gates are constants read once at VirtualThread class initialisation, the USDT sites are NOPs, and each HotSpot freeze/thaw site still tests the probe flag. S0 was not measured against an unpatched JDK. State Buffer USDT Consumer Isolated by Component S0 S1 S2 S3

off on on on

off off on on

– – absent attached

– S1−S0 S2−S1 S3−S2

reference (+8 B per vthread) buffer allocation publication residual observation

Table 5. Experimental environments.

Host CPU

Memory OS / kernel bpftrace

Performance (Sections 7.3–7.4)

Correctness (Section 7.2)

AWS EC2 c7i.metal-24xl, bare metal x86-64, 96 logical CPUs (SMT 2), 1 NUMA node; performance governor, Turbo Boost disabled, no pinning 188 GiB Ubuntu 24.04 LTS, Linux 6.17.0-1017-aws v0.20.2

OrbStack VM, Apple-silicon host aarch64, 11 vCPUs

15 GiB Fedora 43, Linux 7.2.12 v0.24.2

Patched JDK OpenJDK 21.0.12-internal, tag vthread-probes-v2 [5]; SHA-256-verified release asset (x86-64), built from the same tag (aarch64) Consumer ebpf-vthread-correlate @ 18a6b80b [4] 1.37, defaults: 5 forks, 5 × 10 s warmup, 5 × 10 s – JMH measurement, 1 thread

Workloads. Three JMH microbenchmarks exercise different parts of the virtual-thread lifecycle: • Transition: one virtual thread calls Thread.yield() 10,000 times. Each yield is one freeze and one thaw, so the workload consists almost entirely of mount/unmount transitions. • Park/Unpark: two virtual threads hand control to each other 10,000 times, each blocking in LockSupport.park() until the other calls unpark(); this hand-off is closer to real waits than yield. • Churn: each operation starts 1,000 virtual threads with a near-empty body and joins them. The threads never park, so only start and end fire, and the workload isolates the per-thread lifecycle cost. Measurement. We use JMH 1.37 with its default settings: average time per operation, five forks, and five 10 s warmup and five 10 s measurement iterations, so that JIT compilation settles during warmup and forks average out differences in code and heap layout between JVM instances. Performance results were collected on a bare-metal machine, and correctness runs on a virtual machine (Table 5). Uncertainty. Each JMH score is reported with JMH’s default 99.9% confidence interval halfwidth. For the √︃ difference between two configurations, we report the quadrature sum of the two half-widths, 𝑒 12 + 𝑒 22 , and mark a contrast n.s. when the resulting interval contains zero. An n.s.

Towards Runtime-Aware Observability for Java Virtual Threads

13

Table 6. Correctness validation results, summed over all runs of each configuration (128 markers per run). Identity-plane Unmapped markers have no virtual thread in the consumer’s map. Content plane Protocol

Identity plane

Scope mode Runs Correct Unmapped Invalid Misattr. Correct Unmapped Misattr.

Four-probe clean leaky-final Two-probe clean leaky-final

5 1 5 1

640 128 565 114

0 0 18 4

0 0 57 0

0 0 0 10

640 128 560 112

0 0 18 4

0 0 62 12

contrast therefore means the effect is below the resolution of the measurement, not that it is zero; where it matters, we report the interval’s upper bound. 7.2

Correctness

We validate correctness (Q1) with an independent dual-channel harness, built on the definitions of Section 4.2. The ground truth is carried by a channel that never passes through the trace buffer of Section 5.2: the harness determines each marker’s expected trace context and virtual thread, while the eBPF consumer independently records the trace context and virtual thread it attributes to the same event. Each run uses 128 markers, from 16 virtual threads emitting 8 markers each. A marker is a single write system call issued by the virtual thread, whose length encodes the sender; the order of a sender’s markers gives the sequence number [4]. The two scope modes in Table 6 differ in whether the harness closes the final scope: leaky-final deliberately violates scope discipline and leaves a valid record in the buffer when the virtual thread terminates. Each marker is classified independently on the content plane and the identity plane. The runs were performed on a local aarch64 virtual machine (Table 5). The four-probe protocol attributed every marker correctly on both planes in all six runs (768/768), satisfying the criteria of Section 4.2. The result is the same under leaky-final, which shows that its correctness is ensured by the protocol rather than by the SDK’s scope discipline. The two-probe protocol failed as predicted in Section 5.5, and it failed deterministically: every one of its six runs has exactly 16 failing markers, and they are always the first marker of each of the 16 virtual threads, emitted before the thread’s first park. A virtual thread’s first mount fires no thaw, so the two-probe consumer has no entry for it, and a terminated virtual thread never fires freeze, so its entry is never deleted. The first marker therefore finds either an empty slot (Unmapped, 2–7 per run depending on scheduling) or a stale entry left by an earlier virtual thread on the same carrier. Only the split between these two outcomes varies between runs; the 16 failures do not. What a stale entry produces depends on the buffer it points to, and this is where the two planes diverge. Under clean scopes, the terminated thread’s buffer was left with valid = 0, so the content plane reports an honest Invalid, while the identity plane still attributes the marker to the wrong virtual thread. Under leaky-final, the stale buffer was freed while still holding valid = 1: all 10 content misattributions passed the valid-flag check and returned a well-formed trace context of another virtual thread, read from released memory. This is the silent failure of C3, and it shows that the scope mode of the producer decides whether the same structural defect surfaces as an honest failure or as silent misattribution. Finally, in five of the six runs, 1–2 markers per run (7 in total) have a correct trace context but a wrong virtual thread. This is consistent with the allocator reusing the freed buffer address for the current virtual thread, so the stale entry happens to point

14

Wenjie Ding

at the right record. A content-only check would count these markers as correct, which is why C2 requires both planes. Churn’s freeze/thaw event count is zero, so the two-probe protocol’s coverage on this workload is 0%: a structural blind spot, not a probabilistic miss. Both protocols use the same consumer and the same buffer protocol and differ only in which lifecycle events publish the buffer address. Every two-probe failure has the same root cause: the consumer’s mapping does not follow the buffer’s actual lifetime, either because no entry exists yet at the first mount or because a stale entry outlives a terminated virtual thread. The four-probe protocol succeeds because start/end close exactly these gaps, reconstructing the mount/unmount semantics at both ends of the lifecycle. 7.3

Cost Decomposition

Table 7 reports the differences between the configurations. The first three rows give the dormant cost, with no consumer attached. Allocation (S1−S0). Allocation is measurable only on Churn, at +377 ± 30 𝜇𝑠 per operation. This is expected: the buffer is allocated once per virtual thread, and Churn creates 1,000 virtual threads per operation, against one for Transition and two for Park/Unpark. Dividing by 1,000 gives 0.377 𝜇𝑠 per virtual thread. Projected onto Transition, the same per-thread cost is several hundred times smaller than the measurement’s resolution (±225 𝜇𝑠). The allocation cost is therefore consistent across the three workloads: it is paid once per virtual thread and becomes visible only when creating virtual threads dominates the workload. On Churn, whose baseline cost is about 0.41 𝜇𝑠 per virtual thread, it nearly doubles the cost of creating one. Publication residual (S2−S1). The USDT residual is non-significant on all three workloads: with the probes enabled but no consumer attached, the extra work is evaluating the probe arguments and executing a NOP, with no system call and no allocation. The point estimates on Transition and Park/Unpark are positive (+2.91% and +2.35%), and the intervals bound the effect at about +6%. Normalised by the number of probe events per operation (see below), this is 15 ns (upper bound 33 ns) per event on Transition and 10 ns (upper bound 26 ns) on Park/Unpark. Total dormant cost (S2−S0). The total dormant cost is the sum of these two components: +2.47% (n.s.) on Transition and +0.57% (n.s.) on Park/Unpark, where neither component is resolved, and +89.38% on Churn, that is, 0.371 𝜇𝑠 per virtual thread, consistent with S1−S0 within its uncertainty and therefore dominated by allocation. The dormant cost of our mechanism is therefore not zero: the publication path is free to within the measurement’s resolution, but the buffer allocation is paid for every virtual thread, whether or not a consumer is present. Observation cost (S3−S2). With a consumer attached, the cost is 119.62% on Transition and 152.24% on Park/Unpark. The two workloads fire different numbers of probe events per operation, 20,021 and 39,364; divided by these counts, the cost is 648 ± 17 ns and 689 ± 23 ns per event. The two per-event costs are close, which is consistent with observation cost scaling with the number of probe events rather than the number of virtual threads; with two workloads, this is consistency, not proof of a scaling law. Churn does not fit the same model: it fires only 2,000 probe events per operation (start and end for each of 1,000 virtual threads), spread across carrier threads on different CPUs. Its observation cost is −34 ± 16 𝜇𝑠 (−4.28%), a statistically significant decrease. Attaching a consumer cannot plausibly make the workload faster, and we have no explanation for this result; we report it as an unexplained measurement artefact of a workload dominated by thread creation and joining, not as a real speed-up.

Towards Runtime-Aware Observability for Java Virtual Threads

15

Table 7. Cost decomposition: differences between the configurations of Table 4. Cells show the difference ± propagated error in 𝜇s/op and the relative change; the percentage denominators are S0, S1, S0, S2 and S0, respectively. n.s. marks an interval containing zero. Contrast

Transition Park/unpark

Churn

−45 ± 225 −308 ± 517 +377 ± 30 -0.43% (n.s.) -1.74% (n.s.) +90.86% +307 ± 351 +409 ± 600 −6 ± 19 S2−S1: USDT residual, real buffer +2.91% (n.s.) +2.35% (n.s.) -0.78% (n.s.) +262 ± 325 +101 ± 527 +371 ± 31 S2−S0: total dormant deployed +2.47% (n.s.) +0.57% (n.s.) +89.38% +12977 ± 347 +27111 ± 903 −34 ± 16 S3−S2: observation cost +119.62% +152.24% -4.28% +13238 ± 219 +27212 ± 850 +337 ± 29 S3−S0: total cost, consumer attached +125.05% +153.68% +81.28%

S1−S0: allocation alone

7.4

Two-Channel Comparison

This experiment delivers the same buffer address through different channels while keeping everything else fixed. All four variants run on the patched JDK with the same buffer protocol and with the USDT probes off; they differ only in which channel allocates the buffer and whether the JVMTI agent is loaded: Baseline no agent and no JVM allocation; JVM alloc the JVM allocates the buffer (our channel); Agent the JVMTI agent allocates the buffer; JVMTI publish the agent also publishes the buffer address to carrier TLS on every mount and unmount, as Elastic does. Figure 2 (detailed data in Table 8) shows that JVM allocation is non-significant on Transition and Park/Unpark and costs +93.4% on Churn, that is, 0.38 𝜇𝑠 per virtual thread. This is the same component as S1−S0 in Table 7, measured in a separate experiment, and the two agree. The JVMTI channel costs about 20% on Transition (+19.6% Agent, +20.9% JVMTI publish) and Park/Unpark (+20.0%, +20.6%). Two observations locate this cost. First, the Agent variant registers only VirtualThreadStart/End, which fire once per virtual thread, yet it already pays almost the full 20% on workloads with one or two virtual threads per operation; adding per-transition TLS publication (JVMTI publish) adds only about one percentage point, within the measurement’s uncertainty. The cost is therefore not the publication work itself, but the capability tax: loading the agent switches every mount and unmount onto the notifying transition path (Section 2.4). Second, normalised by the same probe-event counts as above, the Agent cost is about 105 ns per event on Transition and 89 ns on Park/Unpark: similar across the two workloads, as expected of a per-transition tax. For comparison, the dormant USDT residual is bounded at 33 ns per event. On Churn, the JVMTI channel costs +433.7% (Agent) and +446.6% (JVMTI publish), that is, about 1.78 and 1.83 𝜇𝑠 per virtual thread, roughly 4.6–4.8× the 0.38 𝜇𝑠 of JVM allocation. Here the JVMTI path pays, for every virtual thread, two callbacks with JNI field accesses in addition to the allocation itself. Both channels deliver the same 8-byte address, so the difference lies in the delivery channel, not in what is delivered. The JVMTI channel’s cost is set by the capability tax and grows with the number of mount/unmount transitions, and it is paid whether or not an observer is attached. With

16

Wenjie Ding

runtime relative to baseline (×)

Agent

JVMTI publish

JVM alloc 5.34×

6

+433.7%

5.47× +446.6%

5 4 3 2 1

1.93× +93.4%

1.20×

1.21×

+19.6% baseline

+20.9%

1.02× +1.6% n.s.

1.20×

1.21×

+20.0%

+20.6%

Transition

1.00× +0.4% n.s.

Park/unpark

Churn

Fig. 2. Overhead of the two publication channels relative to the baseline across the three workloads. n.s. indicates that the corresponding uncertainty interval includes zero.

no consumer attached, the JVM-allocation channel pays only at the two endpoints of each virtual thread’s lifetime, so its cost grows with the number of virtual threads created. 8 8.1

Discussion The Design Space, Not a Horse Race

The two channels are different points in a design space rather than competitors. The JVMTI channel, which Elastic has shown to work, serves any reader: it writes the buffer address into ELF TLS, where every reader with access to the process’s memory can find it, which is the generality OTEP 4947 aims for. Its price is the capability tax, paid on every transition for as long as the JVM runs, whether or not an observer is attached. Our USDT channel gives up that generality: a reader must be attached to the probes before the lifecycle events it needs fire, and must be able to read process memory, which in practice limits it to eBPF. In return, the cost of publication becomes conditional on a reader (Section 6.3), and the only unconditional cost is the per-thread buffer allocation, which is significant only on creation-dominated workloads. On transition-heavy workloads, giving up reader-agnostic generality thus lowers the dormant price from around 20% to below the measurement’s resolution (Figure 2). The design departs from the OTEP’s TLS mechanism, but it is the kind of runtime-specific discovery mechanism the OTEP leaves room for, and it reuses the OTEP’s record format. Because observation cost follows probe events, and probe events follow mount and unmount transitions, an operator can estimate the cost of observing a workload from its transition rate: on our hardware, roughly 650–700 ns per probe event (Section 7.3). Tasks that rarely unmount pay almost nothing, while tasks that block often pay in proportion to how often they do. 8.2

Limitations

Our evaluation has six limitations. (1) Workloads. We use purpose-built microbenchmarks and harnesses, not application-level workloads such as Spring or Quarkus services. (2) Platforms and scale. Our correctness evidence comes only from aarch64 and our performance evidence only from bare-metal x86-64. The correctness runs are also small (six per

Towards Runtime-Aware Observability for Java Virtual Threads

17

protocol, one for each leaky-final configuration), although the two-probe failure pattern was identical in every run. (3) Synthetic context. The correctness harness writes synthetic trace contexts rather than real OpenTelemetry spans, so it validates the mapping protocol rather than the full OpenTelemetry stack. Conversely, because the verification path bypasses the SDK, the result does not depend on the SDK’s scope discipline. (4) OBI. We did not measure OBI’s bytecode-instrumentation channel (Section 3.3), which appeared concurrently with this work. Its per-transition system call places it with the JVMTI channel in our cost model, but we have not tested this. (5) Transition boundaries. The four probes reconstruct the semantics of mount and unmount but not their precise boundaries (Section 5.4); tools that need exact transition latencies would need true mount/unmount hooks. (6) Consumer cost. Our benchmark consumer performs the full hot-path work of maintaining the map and counting events, but does not read the buffer. A production consumer adds a map lookup and a buffer read for every traced event, and the total depends on which events it traces. 9

Conclusion

Under 𝑀 : 𝑁 scheduling, the one-execution-per-thread assumption behind OS-level attribution no longer holds. We restored the join between OS-level events and units of work by binding a trace buffer to the lifetime of each virtual thread and publishing its address through four lifecycle probes, and we compared this channel with a JVMTI channel over the same buffer protocol. Our answers to the questions of Section 1.1 are: Q1 Yes. The four-probe design satisfied the correctness criteria on both the content and the identity plane for all 768 test markers, while the two-probe design is unsound (Section 7.2). Q2 Partly. The publication path is free to within the measurement’s resolution while no consumer is attached: S2−S1 is non-significant on all three workloads, at most about 33 ns per transition. The buffer allocation is not free: it costs 0.371 𝜇𝑠 per virtual thread, which is invisible on Transition and Park/Unpark but nearly doubles the cost of a creation-dominated workload (Section 7.3). Q3 Each channel’s cost is set by what it scales with. The JVMTI channel pays the capability tax on every mount and unmount: about 20% on Transition and Park/Unpark, and about 1.8 𝜇𝑠 per virtual thread on Churn. The JVM-allocation channel pays only per virtual thread: non-significant on Transition and Park/Unpark, and 0.38 𝜇𝑠 per virtual thread (+93.4%) on Churn (Section 7.4). Elastic and, concurrently, OpenTelemetry’s eBPF instrumentation have achieved virtual-threadlevel attribution by publishing on every transition. Our contributions are a different publication mechanism, whose dormant cost is a fixed allocation per virtual thread rather than a cost per transition, and which avoids the capability tax, and, to our knowledge, the first quantification of this cost structure in Java. Future work includes evaluating the mechanism on production web workloads, on both aarch64 and x86-64, and measuring OBI’s bytecode-instrumentation channel within the same cost model. Artifact and Data Availability The patched OpenJDK is available at [5], and the eBPF consumer, JVMTI agent, OpenTelemetry integration, benchmarks and correctness harness at [4]. The raw data behind every table and figure

18

Wenjie Ding

in Section 7, namely the twelve correctness runs and the formal benchmark batch, is archived together with instructions for regenerating the tables and the figure [6]. References [1] Ivo Anjo. 2026. OTEP 4719: Process Context: Sharing Resource Attributes with External Readers. OpenTelemetry Enhancement Proposal, merged 7 April 2026. https://github.com/open-telemetry/opentelemetry-specification/blob/ 34f27fdc4ad113f26c781b45b8a91542f21413d3/oteps/profiles/4719-process-ctx.md Accessed: 2026-11-07. [2] Frederic Branczyk. 2024. Correlating Tracing with Profiling using eBPF. https://www.polarsignals.com/blog/posts/ 2024/03/05/correlating-tracing-with-profiling-using-ebpf Accessed: 2026-08-09. [3] Bryan M. Cantrill, Michael W. Shapiro, and Adam H. Leventhal. 2004. Dynamic Instrumentation of Production Systems. In 2004 USENIX Annual Technical Conference (USENIX ATC 04). USENIX Association, Boston, MA, 15–28. https://www. usenix.org/conference/2004-usenix-annual-technical-conference/dynamic-instrumentation-production-systems [4] Wenjie Ding. 2026. ebpf-vthread-correlate: eBPF Consumer, JVMTI Agent, OpenTelemetry Integration and Correctness Harness. GitHub repository. https://github.com/Neltharion1127/ebpf-vthread-correlate/tree/arxiv-v1 Tag arxiv-v1 on branch main; the experiments ran commit 18a6b80b. Accessed: 2026-11-11. [5] Wenjie Ding. 2026. Patched OpenJDK with Virtual-Thread Lifecycle USDT Probes. GitHub repository, Neltharion1127/jdk21u. https://github.com/Neltharion1127/jdk21u/tree/vthread-probes-v2 Branch vthread-lifecycle-probes, release tag vthread-probes-v2. Accessed: 2026-11-11. [6] Wenjie Ding. 2026. Raw Data for “Towards Runtime-Aware Observability for Java Virtual Threads”. GitHub repository, directory paper-data, tag arxiv-v1. https://github.com/Neltharion1127/ebpf-vthread-correlate/tree/arxiv-v1/paperdata Accessed: 2026-11-11. [7] Elastic. 2026. Elastic Distribution of OpenTelemetry Java. GitHub repository. https://github.com/elastic/elastic-oteljava/tree/v1.12.0 Version 1.12.0. Accessed: 2026-08-01. [8] Rodrigo Fonseca, George Porter, Randy H. Katz, and Scott Shenker. 2007. X-Trace: A Pervasive Network Tracing Framework. In 4th USENIX Symposium on Networked Systems Design & Implementation (NSDI 07). USENIX Association, Cambridge, MA, 271–284. https://www.usenix.org/conference/nsdi-07/x-trace-pervasive-network-tracing-framework [9] Scott Gerring and Ivo Anjo. 2026. OTEP 4947: Thread Context: Sharing Thread-Level Information with External Readers. OpenTelemetry Enhancement Proposal, merged 17 July 2026. https://github.com/open-telemetry/opentelemetryspecification/blob/631bbb4b749f7f02069fd4e22e1ea23f18bdd2cf/oteps/profiles/4947-thread-ctx.md Accessed: 2026-1107. [10] Brendan Gregg. 2019. BPF Performance Tools: Linux System and Application Observability. Addison-Wesley, Boston, MA. [11] Markus Grönlund and Erik Gahlin. 2017. JEP 328: Flight Recorder. https://openjdk.org/jeps/328 Accessed: 2026-08-09. [12] Jonathan Kaldor, Jonathan Mace, Michał Bejda, Edison Gao, Wiktor Kuropatwa, Joe O’Neill, Kian Win Ong, Bill Schaller, Pingjia Shan, Brendan Viscomi, Vinod Venkataraman, Kaushik Veeraraghavan, and Yee Jiun Song. 2017. Canopy: An End-to-End Performance Tracing and Analysis System. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP ’17). ACM, Shanghai, China, 34–50. doi:10.1145/3132747.3132749 [13] Jonathan Mace, Ryan Roelke, and Rodrigo Fonseca. 2015. Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP ’15). ACM, Monterey, CA, 378–393. doi:10.1145/2815400.2815415 [14] OpenTelemetry Authors. 2026. OpenTelemetry eBPF Profiler. GitHub repository. https://github.com/open-telemetry/ opentelemetry-ebpf-profiler Accessed: 2026-11-08. [15] OpenTelemetry eBPF Instrumentation. 2026. Correlate Java Traces across Virtual Thread Mounts. GitHub pull request #2261, merged 10 June 2026. https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/2261 Accessed: 2026-11-08. [16] OpenTelemetry eBPF Instrumentation. 2026. Java In-Process Trace Context Keyed on Kernel Thread Id Breaks under JDK 21 Virtual Threads. GitHub issue #2242. https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/ issues/2242 Opened 3 June 2026. Accessed: 2026-11-08. [17] OpenTelemetry eBPF Instrumentation. 2026. Virtual-Thread-Aware Log Enrichment for Java (traces_ctx_v1). GitHub issue #2284. https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/issues/2284 Opened 9 June 2026, open at the time of writing. Accessed: 2026-11-08. [18] Oracle Corporation. 2023. JVM Tool Interface. https://docs.oracle.com/en/java/javase/21/docs/specs/jvmti.html Accessed: 2026-07-31. [19] Linh-An Phan, MingXue Wang, Guangyu Wu, Wang Dawei, Chen Liqun, and Li Jin. 2025. CrossTrace: Efficient CrossThread and Cross-Service Span Correlation in Distributed Tracing for Microservices. arXiv preprint arXiv:2508.11342. https://arxiv.org/abs/2508.11342

Towards Runtime-Aware Observability for Java Virtual Threads

19

[20] Ron Pressler and Alan Bateman. 2023. JEP 444: Virtual Threads. https://openjdk.org/jeps/444 Accessed: 2026-07-29. [21] Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. 2010. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Technical Report dapper-2010-1. Google, Inc. https://research.google.com/archive/papers/dapper-2010-1.pdf [22] The Go Authors. 2026. The Go Programming Language Specification. https://go.dev/ref/spec Accessed: 2026-08-09. [23] The Go Authors. 2026. Package runtime/pprof. https://pkg.go.dev/runtime/pprof Profiler labels: pprof.Do, pprof.WithLabels. Accessed: 2026-11-07. [24] W3C. 2021. Trace Context. W3C Recommendation. https://www.w3.org/TR/trace-context/ Accessed: 2026-07-29.

A

Complete Benchmark Matrix

Table 8. Complete benchmark matrix. Time and allocation are raw JMH scores ± error; Δ𝑡 reports the difference from baseline and its relative change. n.s. denotes an interval containing zero. alloc. (B/op) reports managed-heap allocation only and excludes the 64 B off-heap trace buffer. Workload

Variant

Time (𝜇s/op)

Δ𝑡 (𝜇s/op)

Relative

alloc. (B/op)

Baseline 10668.0 ± 234.4 – – 240763.0 ± 112.5 Agent 12763.5 ± 252.0 2095.6 ± 344.2 +19.6% 240844.9 ± 138.2 JVMTI publish 12897.8 ± 204.9 2229.8 ± 311.3 +20.9% 240848.9 ± 136.9 JVM alloc 10838.9 ± 167.7 170.9 ± 288.2 +1.6% (n.s.) 240763.7 ± 114.3 Park/unpark Baseline 17606.2 ± 379.6 – – 460835.9 ± 2151.7 Agent 21125.5 ± 478.3 3519.3 ± 610.7 +20.0% 465810.4 ± 1392.0 JVMTI publish 21228.4 ± 454.7 3622.2 ± 592.4 +20.6% 464970.0 ± 1063.6 JVM alloc 17684.5 ± 390.4 78.2 ± 544.6 +0.4% (n.s.) 456728.5 ± 4045.9 Churn Baseline 409.7 ± 27.9 – – 320508.0 ± 351.0 Agent 2186.7 ± 3.9 1777.0 ± 28.2 +433.7% 318446.6 ± 25.5 JVMTI publish 2239.6 ± 7.3 1829.8 ± 28.9 +446.6% 318463.4 ± 26.3 JVM alloc 792.2 ± 13.3 382.5 ± 30.9 +93.4% 316057.4 ± 0.7 Transition

B

USDT Probe Definitions

probe vthread__freeze(uintptr_t, int, uintptr_t); //args: virtual thread id, freeze result, trace buffer address probe vthread__thaw(uintptr_t, int, uintptr_t); //virtual thread id, thaw kind, trace buffer address probe vthread__start(uintptr_t, uintptr_t); //virtual thread id, trace buffer address probe vthread__end(uintptr_t); //virtual thread id

Listing 1. DTrace probes

Related documents

• Towards Runtime-Aware Observability for Java Virtual Threads#1649029
• Optimalisasi Manajemen Thread dalam Sistem Operasi untuk Meningkatkan Kinerja Multithreading#1066974
• Beyond Locks and Thread IDs: Static Data Race Detection Off The Beaten Path (Extended Version)#627765
• OmniLink: Specification-Based Trace Validation of Concurrent Systems with Substep Linearizability#1624566
• CVE-2026-91305 — CRITICAL 9.1#1404522