Nanvix: A Multikernel OS Design for High-Density Serverless Deployments Carlos Segarra† Pedro Henrique Penna‡ Enrique Saurez★ Íñigo Goiri★ Peter Pietzuch† Shan Lu‡△ Rodrigo Fonseca★ † Imperial College London ‡ Microsoft Research △ University of Chicago ★ Azure Research - Systems
Faasm SEUSS
Serverless providers strive for high resource utilization by optimizing deployment density: how many applications can be deployed per host server. However, achieving high deployment density without compromising application performance or isolation remains an open challenge. High density can be achieved by sharing components across applications, yet applications from different tenants must be strongly isolated from each other due to the risk of side-channel attacks. Sharing components across applications from the same tenant, if done naively, can introduce contention on host resources thus negatively affecting application performance. We describe Nanvix, a new multikernel OS that disaggregates ephemeral execution state, unique per application invocation, from long-lived persistent state, shared among invocations from the same tenant. Applications in Nanvix execute inside a lightweight user VM running a micro-kernel that implements threads and memory, and forwards all I/O requests to a system VM. The system VM runs a macro-kernel with a rich set of device drivers and is shared among all invocations from the same tenant. Nanvix’ split design achieves strong hypervisor isolation across tenants without sacrificing application performance, and reduces same-tenant contention by multiplexing all I/O requests to the system VM. Thanks to a system-wide co-design, Nanvix achieves order-of-magnitude lower application start up times with moderate I/O overheads. When replaying a production trace, Nanvix needs 20-100× fewer host servers compared to stateof-the-art systems, improving deployment density.
1
Inter-tenant isolation Not isolated Isolated Compatible
Dandelion Virtines Containers
Deployment density
arXiv:2604.11669v1 [cs.OS] 13 Apr 2026
Abstract
Nanvix bil
rt:
-of
ate
St
ion
lat
ti pa
e-a -th
Not Compatible
m Co
ity
so +I
Unikraft RunD
Azure Functions
Firecracker Virtual machines
Cold start latency
Figure 1. Serverless design space (Serverless providers wish to optimize deployment density as a proxy for resource efficiency, but struggle to do so while maintaining low cold start latencies, strong inter-tenant isolation, and application compatibility.)
service level objective (SLO) targets [33]. Unfortunately, it is hard to achieve deployment density while maintaining low cold start latencies and strong isolation: high density can be achieved by sharing components across applications, but excessive sharing can introduce resource contention and side-channels. An orthogonal but equally desirable property is compatibility with existing programming frameworks: being able to run existing software on standard runtimes, instead of requiring a completely new programming model and targeted rewrites [28, 48, 54]. Fig. 1 illustrates the trade-offs made by different systems. Many prior works relax inter-tenant isolation by sharing the same process substrate via software-fault isolation [12, 48], or the same operating system (OS) kernel, via containerization [9, 17]. In this paper, we target multi-tenant serverless deployments that cannot compromise on inter-tenant isolation and must isolate applications using virtual machines (VMs). Systems such as Azure Functions, explicitly sacrifice deployment density for lower cold start latency by pre-warming VM instances in a pool [2]. Other systems achieve deployment density, cold start latency, and inter-tenant isolation,
Introduction
The past decade of serverless research has driven significant reductions in application cold start latency through lightweight isolation (e.g. micro-VMs) [1, 54], fast snapshot restoration [16], elastic scaling under bursty load [10, 28], and increasingly sophisticated orchestration layers [43]. As a result, production serverless systems can launch new applications in less than a hundred milliseconds, even in periods of high contention [10, 33]. Cold start latency, however, is not the only metric that needs to be optimized. From the perspective of the serverless provider, equally important are inter-tenant isolation and deployment density – how many applications can be co-located on a single server while still meeting per-request latency 1
or file systems. This minimal-feature design ensures each user VM has a small memory footprint and veryfast startup, addressing the cold start latency without compromising deployment density or security. Macro-kernel. It includes the full set of device drivers and kernel modules needed to serve I/O requests that the micro-kernel cannot handle. The system VM, similar to other cloud VMs running a fully-featured OS, has a considerably higher cold start latency and memory footprint than a user VM. Fortunately, unlike ephemeral user VMs, a system VM is long-lived, serves requests from many user VMs from the same tenant, and can be proactively auto-scaled. In addition, given that it does not execute tenant-specific code, all system VM instances can be restored from a generic snapshot.
System VM
VM
VM
App
App
User VM
Fullyfeatured guest OS
Fullyfeatured guest OS
Microkernel
App
Connector Macrokernel
Hypervisor
Hypervisor
(a) Traditional VMs
(b) Nanvix
Figure 2. Nanvix is a multikernel OS for serverless (Existing multi-tenant serverless platforms offer compatibility by including all OS functionality in the guest, fundamentally limiting performance and density. Nanvix, on the other hand, advocates for a disaggregated design.)
We adopt a clean-slate (co-)design for the micro-kernel, its VMM, and the userspace components of the system VM, while employing a trimmed Linux as the macro-kernel. We implement all components in Rust and release them as opensource. Nanvix executes unmodified applications written in C, C++ and Rust after compilation with our toolchain. It also executes Python, JavaScript, and WebAssembly (WASM) applications by compiling the language runtime. Our evaluation shows that user VMs start up in a handful of milliseconds, an order-of-magnitude faster than restoring a micro-VM from a snapshot [50]. A system VM and a user VM are collectively deployed in less than 30 milliseconds. Thanks to the explicit sharing of the system VM, we can deploy 30–50% more sandboxes per GiB than other snapshotbased baselines. When replaying a production trace, a system using Nanvix would require 20–100× fewer servers to serve the trace compared to one using Firecracker [1] or optimized snapshot-based systems.
but sacrifice application compatibility by imposing substantially different programming and execution models [28, 54]. Unfortunately, the systems that do retain compatibility (under virtualization) [1, 29, 33] end up paying a large, often order-of-magnitude, density and latency penalty compared to those which do not. This is because they use a guest OS that was not designed for ephemeral execution and work around it with advanced VM snapshot techniques [3, 10] or using lightweight isolation inside a parent VM [33]. Ultimately, these system-level solutions shift the pressure to other parts of the system (see §2). On this landscape, our key observation is that the high cost of compatibility is the root cause of cold start latency and low deployment density. This cost cannot go down in a secure and fundamental way as long as we expect the guest OS inside each VM to implement all the features required for compatibility such as devices, network and file systems (see Fig. 2a). Consequently, to achieve a minimum virtualization environment for serverless applications, we argue for disaggregating the ephemeral execution state of every application from the long-lived persistent state of the OS.
2
Towards Dense Serverless Deployments
We next argue why deployment density is an important metric for serverless providers (§2.1), how can it be achieved securely (§2.2), and why is it challenging to do so (§2.3).
Our proposal. To fulfill this idea, we present Nanvix, a new multikernel OS for high-density serverless deployments. Nanvix splits the system services of the guest OS across two virtualized environments: (1) a lightweight user VM with a micro-kernel that supports application’s ephemeral execution with thread and memory management; (2) a feature-rich system VM with a macro-kernel that provides system functionality (i.e. networking and file system) for all user VMs belonging to the same tenant (see Fig. 2b). Nanvix is a unique multikernel OS where two kernels provide complementary properties that together meet serverless performance, compatibility, security, and density goals:
2.1
Why deployment density?
Early workload characterizations of production serverless deployments showed that cold start time dominated request latencies [21, 46, 53]. Since then, serverless research has focused on reducing cold start latencies by optimizing every component on the request critical path, from the controlplane [43] to the execution sandbox [1]. However, cold start latency, as a proxy for request latency, is a tenant-facing metric and does not directly measure serverless provider’s efficiency. Some production serverless systems report function cold starts below 100 ms [10], while others offer getting rid of cold starts altogether for a price premium [2, 4]. Deployment density, on the other hand, is a provider-facing metric. Achieving high deployment density means using
Micro-kernel. It implements processes, threads, scheduling, memory management, IPC, and a POSIX compatibility layer, but eschews device virtualization, network, 2
100 Containers
Frequency [%]
80
Region 1 R2
60
R3 R4
R5 Base VM
40 20 0
Stranded
Grow Latency
Base VM
(b) Dynamic base VM
Figure 4. Containers in VM deployment (This approach employs a base VM that can either have a static allocation of resources, in which case some resources may be stranded, or may grow and shrink dynamically, adding latency to the request’s critical path.)
101 102 103 104 105 106 107 # concurrent same-tenant reqs. on arrival
Figure 3. CDF of the # of in-flight requests from the same tenant when a request arrives (We plot the CDF across five different regions based on the Huawei trace from 2025 [21]. In red we highlight the frequency of there being zero in-flight requests from the same tenant.)
vCPU Mem (MiB)
1 –
4 –
8 –
– – – 1 4 8 128 512 1024 128 512 1024
p50 (ms) p99 (ms)
73.5 94 141 22 22.5 25 77 86 141 104 226 490 24 24.9 36.1 89.5 108 380
Table 1. Hotplug latency (We measure the time to hotplug vCPU and/or memory with ACPI and virtio-mem, respectively, and report p50 and p99.)
fewer physical servers to serve the same load, yielding higher utilization, lower cost, and higher revenue from providers’ perspective. Deployment density is determined by the dominant resource. If the workload consumes a lot of memory, such as using large container images or deploying large pre-warmed VM pools [2], memory capacity will dominate deployment density. On the other hand, if the workload follows a bursty arrival pattern, which is common in serverless [2, 46], application start throughput, which is often dominated by contention on host OS resources [10, 33], can limit the deployment density. Note that these two metrics are related. Techniques that shorten the cold start latency can reduce the amount of time each request needs to get served, and hence may improve the deployment density through increased time sharing among requests on each server. However, if the cold start latency is improved by consuming extra system resources, the deployment density may drop as discussed in §1. 2.2
Memory
(a) Static base VM
Prob. no concurrent requests 100
Ctrs
Memory
there is an 80% probability that when a request arrives there is another in-flight request from the same tenant, creating resource sharing opportunities. 2.3 Why is sharing hard? Existing approaches to exploit intra-tenant sharing are constrained by using a full guest OS to offer application compatibility, and need to work around this limitation to offer high deployment density. Containers in VMs. A common approach is to deploy sametenant applications inside containers, instead of VMs, inside a parent, per-tenant, base VM. This deployment strategy amortizes the cost of the guest OS, image layers, and system libraries across application invocations of the same tenant [33]. However, it introduces a fundamental sizing tension. Fig. 4 shows two approaches to deploying this containersin-VMs scheme. When the base VM has a fixed allocation of vCPUs and memory (see Fig. 4a), the scheduling of requests to base VMs boils down to a bin-packing problem. This approach is likely to strand resources if requests cannot be perfectly bin-packed or base VMs become fragmented over time. To avoid such resource stranding, we can allow the base VM to grow or shrink dynamically based on request load (see Fig. 4b). However, if the growing or shrinking happens proactively, as part of an auto-scaling policy, it will often leave underutilized resources [26], impacting density. If it happens reactively, the time to grow or shrink the base VM contributes to the cold start latency. In Tab. 1, we show that hot-plugging a vCPU or memory balloon adds over 80
How to improve deployment density?
A common approach to improving deployment density is to share resources across applications. However, in a multitenant serverless environment, there is little room for intertenant sharing due to many security constraints. The hypervisor cannot allocate applications from different tenants in sibling cores that share micro-architectural state nor employ memory sharing techniques like Kernel Samepage Merging (KSM) due to the risk of side-channel attacks [8, 24, 35, 56]. Snapshot images may be shared, but under specific circumstances that we expand on in §2.3. Intra-tenant sharing is therefore the focus of this paper. In Fig. 3 we show that, even without inter-tenant sharing, there is a big opportunity to share resources across applications of the same tenant. Based on the Huawei trace from 2025 [21], 3
ms of overhead, and hot-unplugging can be substantially worse [30, 33].
System VM (1 per tenant) ... FS stack Net stack WASI stack
Macro-kernel Drivers
User VM syscalls
JS app Microkernel
WASM app User VM
Devices pass-through Hypervisor
Host OS
Devices
Figure 5. Overview of Nanvix (Nanvix adopts a multikernel design where a micro-kernel executes application code and a macro-kernel multiplexes I/O requests from user VMs belonging to the same tenant.)
Low cold start latency. With high probability (see Fig. 3), when a new invocation request arrives there is already a running system VM instance for that tenant, so only the user VM start up is on the critical path. The user VM emulates a very simple machine model and thus avoids expensive initialization operations like scanning ACPI tables or enumerating PCI devices. The micro-kernel boots in one or two milliseconds, and the end-to-end latency for a cold HTTP echo takes a handful of milliseconds, including the control-plane (see Tab. 3).
Summary. Deployment density is an efficiency metric that directly correlates with provider’s resource utilization. It is related to, but different from, cold start latency. Existing multi-tenant systems struggle to achieve high density and low cold start latency due to the reliance on heavyweight guest OSs to guarantee application compatibility.
3
Routing layer
VM snapshots. An alternative to avoid the resource stranding and performance overheads of containers in VMs is to deploy each application in a different VM restored from a snapshot. What memory contents are included in the snapshot and how they are restored determines whether the same snapshot can be used to restore VMs from different tenants. For a snapshot to be safely re-used across tenants it must include neither tenant code nor system state that may have been influenced by the execution of tenant code like the GC [59] or JIT [25] compiler state in a managed language runtime such as Python, Java, or .NET. In addition, the guest OS must reinitialize the entropy pool and other sources of randomness after a restore [15, 37]. In practice, AWS Lambda SnapStart [3] and Ant FaaS [10] use per-function and per-tenant snapshots, respectively. To address the increased disk and memory pressure of having more and diverse snapshots, each system employs different advanced snapshot management mechanisms. SnapStart chunks snapshots in blocks and serves them on-demand from a distributed storage layer with two levels of caching across availability zones. Ant FaaS uses a hierarchical tree storing incremental snapshots restored with copy-on-write (CoW). There is no quantitative analysis on the impact of these techniques on the host server’s memory and disk pressure in production, but they undeniably add considerable management and system complexity.
User VM Python app
High deployment density. Each application executes in a different user VM and all I/O is multiplexed by a single system VM (per tenant). This disaggregation addresses the resource stranding issues of containers-in-VMs and reduces resource contention because each user VM gets its own share of vCPU and memory, and only system VMs contend for expensive network resources like namespaces or TAP devices. In addition, all system VMs are restored from the same generic snapshot, further reducing Nanvix memory footprint and improving deployment density.
Nanvix Overview
Our goal in Nanvix is to execute serverless applications in a minimum virtual execution environment while retaining application compatibility. Our approach, as highlighted in Fig. 2b, is to disaggregate the system services offered by the guest OS in those essential for the execution of applications and those necessary to interact with external devices, and serve both from different virtualized environments that can communicate with each other. Nanvix adopts a multikernel [7] design where a microkernel executes serverless applications and forwards all I/O system calls to a macro-kernel serving I/O requests for all applications from the same tenant (see Fig. 5). Both kernels and their supporting systems software are co-designed for the specific purpose of executing short-lived ephemeral applications, and this co-design enables Nanvix to achieve all the design goals from Fig. 1.
Inter-tenant isolation. The only shared substrate among user VMs hosting application from different tenants is the underlying hypervisor. On the other hand, user VMs that host applications from the same tenant share the system VM. Worker threads in the system VM executing system calls from different user VMs from the same tenant can be further isolated from each other. In addition, Nanvix leverages the communication layer between the system VM and the user VM as a low-friction vantage point for system call filtering and interposition (see Fig. 7). 4
All-in-all, Nanvix intra-tenant isolation guarantees are comparable to those of containers that share the container runtime (e.g. containerd) and guest OS, used in production serverless systems like Google’s CloudRun [11], Alibaba Serverless Containers [33], and other Kubernetes or Knative based systems [17]. Systems like Azure Functions and AWS Lambda also re-use the same container instance (and thus underlying OS state) through the function’s keep-alive configuration [5].
let n = unistd::read(fd, buf); 1
U
2
Microkernel
VMEXIT
K
redeem 5 use
VMEXIT in
out
4
IO
2
C
Read credits
3 add 1 ReadResponse 5 ReadRequest
Figure 6. User VM architecture (The user VM consists of a micro-kernel and a VMM co-designed for fast start-up times and to minimize VM exits during synchronization and I/O.) credit counter and, when credits are available 4 , issues a VM exit 5 and forwards the response to the user thread 6 . We expand on the credit-based control-flow as well as our design for bulk data transfers in §4.3. 4.2
Micro-kernel
Tab. 2 summarizes the different kernel calls implemented in the user VM’s micro-kernel. The POSIX compatibility layer in userspace translates POSIX calls, e.g. unistd::read, into the corresponding kernel call. The micro-kernel implements threads, processes, and memory management, including thread-local storage. The micro-kernel also implements thread scheduling and synchronization, capability-based access control, and memory-mapped I/O (MMIO) and PMIO. All of these kernel calls are executed locally, within the kernel, and do not rely on the system VM. The micro-kernel also implements a set of inter-kernel (i.e. micro-kernel to macro-kernel) communication kernel calls. send/recv use regular PMIO to send one-off messages, whereas push/pull are used for rendezvous-style bulk data transfer over MMIO, and we expand on them in the next section. Lastly, as illustrated in Fig. 6, the kernel has a dedicated thread that polls for incoming messages from the system VM and delivers them to the corresponding user threads.
User VM
The user VM (§4.1) is Nanvix’ virtual execution environment. It comprises a micro-kernel (§4.2) and a VMM (§4.3) co-designed for low start-up times and near-zero-copy I/O. 4.1
4
VMM
Nanvix’ disaggregated design presented in Fig. 5 favors modularity. The filesystem or network stack can be inside the macro-kernel, in userspace [18], or entirely offloaded to hardware [6], and stacks can be replaced transparently to application code. In addition, other type of API calls, not only I/O system calls, can also be forwarded from the user VM to the system VM. For example, Nanvix support for WASM applications forwards most WASI [55] calls to the system VM. In a cloud environment where POSIX or system calls are less frequent [28, 49], user VMs could also forward other cloud-native APIs like S3 to the system VM, or directly to the storage node.
4
6
3 await
Application compatibility. The cost of specialization and co-design is compatibility. The user VM can only execute guests running the Nanvix micro-kernel. Instead of offering kernel-level compatibility, we aim to offer source code level compatibility by equipping the micro-kernel with a POSIX compatibility layer that enables the execution of unmodified applications after re-compilation. This type of compatibility is a common assumption in other serverless systems [18, 29].
User space Kernel space
U
Overview
Fig. 6 provides a high-level overview of the user VM architecture, contrasting it with state-of-the-art VMMs capable of hosting Linux [1, 20], and illustrates the process of executing a read POSIX call. Application code in Nanvix is executed by one or many User threads in the micro-kernel ( 1 , Fig. 6). After transitioning to kernel space and issuing the (remote) blocking call via port-mapped I/O (PMIO) 2 , the user thread waits for the response 3 . The VMM has a dedicated I/O thread 4 that forwards requests to the system VM 5 (not depicted). The same I/O thread receives the response and forwards it to the micro-kernel 1 . As part of the user VM co-design, the VMM-to-microkernel data path uses a credit-based flow control to minimize the number of VM exits issued by the guest micro-kernel while polling for the response 2 . A dedicated Credits thread adds a credit to a control page in the guest micro-kernel via shared memory 3 . A dedicated Kernel thread monitors the
4.3
VMM
When designing the VMM, we eschew ABI compatibility with arbitrary OSs and kernels in favor of specializing for the micro-kernel. In the process, we make a series of design choices that depart from traditional VMMs such as Firecracker or CloudHypervisor. Shared control-page contract. The user VM’s VMM does not implement standardized virtualization interfaces such as VirtIO [14] or ACPI [13] because it only interacts with the 5
Category Process & Thread management Scheduling & Synchronization Memory management Capability & Process control Thread-local storage Time & Debug Device I/O management Inter-kernel communication
Kernel calls Local execution getpid, gettid, exit, exit_thread, join_thread, create_thread yield, sleep, mutex_lock, mutex_unlock, cond_wait, cond_signal, resume mmap, munmap, mctrl, mcopy capctl, terminate set_thread_data_area, get_thread_data_area gettime, debug mmio_alloc, mmio_free, mmio_info, pmio_alloc, pmio_free, pmio_read, pmio_write Remote execution send, recv, push, pull
Table 2. Classification of kernel calls by functionality and execution locality (Most kernel calls are resolved locally in the micro-kernel, while communication primitives are forwarded to the macro-kernel in the system VM for remote execution.)
system VM and is designed for ephemeral execution. Instead, the guest micro-kernel and the VMM use shared memory pages at fixed guest physical addresses (GPAs) to synchronize control variables: interrupt requests (IRQs), credit counters, paravirtualized clock, among others.
Client apps
U
U
U
U
User VM demux Gateway out: Endpoint in:
0
Endpoint
2 User VM handle 4
3
Syscall filter
Worker threads
Credit-based flow control. As illustrated in Fig. 6, the VMM implements a credit-based control flow to minimize the VM exits issued by the guest when polling for messages. The credit counter is an atomic variable in the guest microkernel’s control page available to the VMM via shared memory. After incrementing the credit counter, the VMM may also inject an interrupt if the guest is in a halted state.
Macro-kernel thread-isol fs
net
Figure 7. System VM architecture (The system VM executes inter-kernel calls from user VMs belonging to the same tenant and interfaces with the different client applications.)
Near zero-copy bulk data transfer. To optimize the transfer of large payloads across the VM boundary, the VMM and the micro-kernel implement a rendezvous-style bulk data transfer protocol over MMIO. As part of the protocol, triggered by the push or pull kernel calls, a control message is sent over PMIO with metadata on the GPAs to read/write data from/to with a scatter/gather syntax. The VMM copies the data once from/to the GPAs to/from the transport buffers to the system VM, and we expand on ways to eliminate this copy in §5.3.
contain nor execute any tenant-provided application code, and its image is generic and shared across all tenants. This means that system VMs, in spite of running a full-weight kernel, can be easily pre-warmed and pooled, thus eliminating the overheads of initializing the different I/O subsystems or device drivers. Fig. 7 presents an overview of the system VM architecture. The system VM maintains a 1:1 mapping between User threads in the micro-kernel (see Fig. 6) and worker threads in the macro-kernel: each system call issued by a specific user thread will always be executed by the same worker thread in the macro-kernel. This design simplifies reasoning about kernel call ordering in multi-threaded applications.
System VM
The system VM executes inter-kernel calls from user VMs belonging to the same tenant. Next, we present an overview of the system VM (§5.1), how it isolates requests from different user VMs (§5.2), and its transport layer (§5.3).
5.2 5.1
User VM
read(STDIN_FILENO) 1
PMIO-defined ABI. The VMM relies on PMIO for basic standard I/O from/to the micro-kernel as well as to send commands from the VMM to the micro-kernel, or remote kernel call requests from the micro-kernel to the VMM. This ABI is kept minimal and simple.
5
User VM
Overview
Isolation
Fig. 7 illustrates the process of executing a read POSIX call from the perspective of the system VM and is complementary to Fig. 6. Client applications (e.g. an interactive user or an event driven workflow) can interact with the user VM via
The system VM multiplexes inter-kernel calls from different user VMs and translates them to system calls executed inside its macro-kernel. Most importantly, the system VM does not 6
a gateway TCP port exposed in the system VM (Fig. 7, 0 ). When the system VM receives the read request from the user VM, it forwards it to the corresponding worker thread based on the user VM and user thread identifier 1 . Worker threads belonging to the same user VM can be grouped together for performance isolation using some form of thread isolation (e.g. cgroups-v2 in Linux) and CPU and NUMA affinity. In terms of security isolation, different threads execute in the same virtual address space but each thread executes a very simple logic: it acquires a handle to the user VM state 2 , including the connection to the endpoint, applies the system call filter 3 and calls the corresponding macro-kernel system call. This simple datapath in userspace and our use of Rust limit the impact of safety bugs. After filtering, inter-kernel system calls will be served by the macro-kernel. One notable exception are reads and writes to stdin and stdout. We decide to connect the user VM’s standard I/O devices to its gateway endpoint such that client applications can directly interact with the user VM 4 . 5.3
Tenant B
User VM
User VM
System VM config.json overlay
mmap
Snapshot
System VM snapshot Control plane daemon
Netns User VM User VM System VM
Standalone user VMs Tenants C & D User User VM VM
Net NS Pool Snapshot mgmt
Figure 8. Nanvix deployment (Groups of user VMs and a system VM belonging to the same tenant are deployed inside a network namespace, and user VMs that do not require any I/O can be deployed in standalone mode.) User VM. The micro-kernel and the VMM are implemented in 17 and 11 kLOC, respectively. The VMM has support for the KVM hypervisor and preliminary support for the Windows Hypervisor Platform (WHP). Nanvix can also integrate with existing embedded VMMs as long as they implement the custom PMIO ABI. In fact, Nanvix can integrate with the Hyperlight VMM [38], a production-grade alternative to Virtines [54], but this integration does not yet have featureparity with the native VMM.
Transport
Nanvix is a message-oriented multikernel. In this regard, Nanvix takes inspiration from micro-kernel systems [34, 39] and can consequently suffer from similar issues in terms of latencies introduced by the transport layer. For reliable control-plane message delivery, the connections between client applications and user VMs to the system VM are established over regular TCP sockets. The latency of a single TCP message, particularly within the same physical server or in the same server rack, can be well within 40 us [58] and is unlikely to dominate start up latency or execution time for serverless functions that do little I/O. For large payloads chunked in multiple messages, the bulk data transfer optimization introduced in §4.3 prevents from sending many small TCP messages, improving throughput. The same rendezvous-style communication pattern could be extended from the user VM’s VMM to the macro-kernel by setting up a shared memory area between the macro-kernel and the VMM, and exposing it to the former via a mechanism such as ivshmem [42]. Note that the rendezvous communication blocks the sender until the receiver has acknowledged the reception of the message. This introduces safety and starvation issues in micro-kernels where the communication parties are two, potentially distrusting, application processes. Nanvix sidesteps these issues because in all communications one of the parties is the user VM’s VMM or the macro-kernel, which are part of the system’s trusted computing base.
6
Tenant A Network namespace
System VM. The system VM, i.e. the components in Fig. 7 besides the macro-kernel, is implemented in 8 kLOC. We use the x86-64 Linux kernel as macro-kernel with a lean configuration profile. We enable KVM and paravirtualized execution, VirtIO, ACPI CPU and memory hotplug, and the core filesystem, namespace, cgroup and memory management features. We disable general-purpose hardware drivers (e.g. USB), but consider adding specialized I/O drivers if necessary. We use Cloud Hypervisor [20] as the VMM for the macro-kernel. The system VM is always spawned from a snapshot loaded in main memory with on-demand paging. We had to patch CloudHypervisor to load the snapshot lazily, as we found that the upstream version did not have support for this feature crucial to achieve high density and low start-up times. Control plane. We implement a control-plane daemon that manages the life cycle of the different system VM and user VM instances running on the same physical server in 6 kLOC. This component determines the connections between user VMs and system VMs, as well as the isolation among components from different tenants. We illustrate a complete deployment of Nanvix in Fig. 8. Groups of user VMs and a system VM belonging to the same tenant are deployed inside a network namespace. All system VM instances are restored from the same base (memory) snapshot, but we apply an overlay on the snapshot configuration file to introduce per-instance modifications like the
Implementation
Nanvix is implemented from scratch in Rust and made available as open-source at: https://github.com/nanvix/nanvix. 7
console file path or TAP device configuration. For user VMs that do not require a system VM, Nanvix supports deploying them in standalone mode. User VMs in Nanvix are ephemeral and run to completion. Once a user VM has finished executing the application code it is deleted. User VM cold start times are low enough (see Tab. 3) that Nanvix can afford to not use keep-alive policies and spawn sandboxes on demand, avoiding the complexities that arise with (safe) sandbox reuse. Application compatibility. Nanvix can natively run C, C++, and Rust applications after compilation with our toolchain. We ship a GCC-based compilation toolchain with a port of newlib and a POSIX shim layer. For Rust applications, we add a custom Nanvix target. Nanvix’ also has ports of Wasmtime, CPython and Deno runtimes to execute WASM bytecode, Python, or JS programs, respectively. Future work. Nanvix is an early-stage prototype and, as such, has limitations that we plan on addressing in the future. The current user VM prototype only implements the i686 32-bit architecture and only supports single-core execution1 . The shared memory, zero-copy, datapath between the user VM and the system VM is also not yet merged. In terms of application compatibility, currently Rust applications targeting Nanvix have increasing support for std. There is also on-going effort to support the NodeJS runtime. None of the above are inherent limitations of Nanvix’ multikernel design, and are a matter of engineering effort. Acknowledging the above limitations, Nanvix codebase is actively maintained and is undergoing formal verification with Verus [31]. The system is developed and tested in company X and will soon be deployed in production.
7
Evaluation
In the evaluation we aim to answer the following questions: i What is the performance of Nanvix and how does it compare to state-of-the-art serverless systems? (§7.2) ii What are the advantages of deploying Nanvix in a serverless fleet when replaying a production trace? (§7.3) 7.1
Experimental set-up
Baselines. We compare Nanvix (NVX) against the opensource solutions used by the leading serverless providers: Firecracker v1.12.0 (FC), CloudHypervisor v45.0 (CLH), and gVisor v20250505 (gV). We also compare against a production unikernel, Unikraft [29] v0.12.3 (UK), as it is increasingly being adopted in the context of serverless [9, 41], and against Linux processes (Proc) and Hyperlight (HL) [38] as lower-bounds on the overhead to instantiate any application and any VM, respectively. Hardware. We run our experiments on a bare-metal Intel Xeon server with two sockets and ten cores per socket. We 1Work on x86_64 64-bit has already started.
8
Baseline
p50 (ms)
p99 (ms)
Slowdown
1097.50 1069.77 347.79 1185.52 41.29 22.41 18.82
37.76x 37.49x 11.50x 5.77x 1.00x 0.27x 0.12x
First time Firecracker Cloud Hypervisor Unikraft gVisor Nanvix Hyperlight Process
1055.05 1047.55 321.28 161.10 27.94 7.68 3.32
Known function CloudHypervisor-S Firecracker-S Nanvix-uV
41.44 69.45 7.26
118.23 83.77 11.37
5.71x 9.56x 1.00x
Table 3. Cold start latency (The First time section reports slowdown relative to Nanvix, spawning both user VM and system VM, while the Known function section reports slowdown relative to Nanvix spawning only a user VM.)
disable hyperthreading and frequency scaling. Our server has 32 GiB of memory, two NUMA regions, and runs Ubuntu 24.04 with the 6.8.0-90-generic kernel. 7.2
Microbenchmarks
To understand the benefits and overheads of Nanvix’ multikernel OS architecture, we deployed a HTTP echo server written in Rust that sends and receives a 32 B payload, and measure different key performance metrics. Cold start latency. We first measure cold start latency, i.e. the time elapsed between sending the HTTP echo request and receiving the response when there are no previouslyrunning instances. The cold start latency includes the time to provision the execution environment, its network devices (excluding the network namespace), and the application itself. We include two additional baselines that benefit from having executed the function once: Firecracker (FC-S) and CloudHypervisor (CLH-S) starting the function from a generic OS snapshot. For Nanvix, the cold start latency measures the time to spawn a system VM and a user VM, whereas NanvixuV measures the time to spawn only a user VM. We report the p50 and p99 cold start latencies over 1000 iterations. Tab. 3 summarizes our results. Nanvix has two orders of magnitude lower cold start latencies than systems that boot Linux from scratch (i.e. Firecracker and CloudHypervisor), and 6-11× lower cold start latencies compared to specialized solutions such as gVisor and Unikraft. In both cases, a noticeable part of the difference comes from setting up the network, including TAP devices. For Unikraft, we use upstream QEMU v8.2.2 as the VMM, whereas the original paper used a highly modified one [29] and only reported boot time, not time to serve the first HTTP request.
Phase
p50 (us)
p95 (us)
p99 (us)
channel_setup partition_create vmem_create vcpu_create kernel_load initrd_load vcpu_reset thread_spawn guest_exec exit_handling
12 470 49 158 1636 60 19 38 952 47
18 613 77 251 3155 112 36 61 1415 64
61 1131 92 275 3339 131 39 66 1751 80
Total
3553
5069
5460
1.3 MiB kernel image into the guest’s address space. The application binary itself, presented as an initrd, only occupies 45 KiB and loads in negligible time. The second contributor to latency is the guest execution time, which includes all the time the vCPU is executing in the guest. These latencies are comparable to other embedded VMMs [38, 54]. Memory footprint. To understand the relationship between cold start latency and memory consumption, we next deploy instances in a closed-loop until we consume 1 GiB of system memory. We present the cold start latency for each instance and their memory contribution, calculated by subtracting the before and after readings of MemAvailable in /proc/meminfo, in a scatter plot. We present the p50 from 10 iterations across both and the minimum number of instances deployed in 1 GiB across all iterations. For Nanvix, we consider two ends of the spectrum in terms of deployment: only one system VM (NVX) as a best case scenario, or one system VM per user VM (NVX-1to1) as a worst case scenario. Fig. 9a summarizes our results. The cold start latencies, on the X axis, are comparable to those reported in Tab. 3. In terms of the memory footprint, the same application executing in a user VM has an order of magnitude lower memory footprint than executing in a (micro) Linux VM, and a 3050% lower memory footprint than the same application being restored from a snapshot with on-demand paging. This is due to Nanvix native VMM being much smaller and simpler than CloudHypervisor or Firecracker, and because Nanvix does not virtualize devices via VirtIO. NVX-1to1 is on par with CLH-S because the system VM instance dominates the memory footprint. These reductions in memory footprint are reflected in higher instance counts per 1 GiB of memory, which we present in Fig. 9b. The ratios are not exactly the same because the instance counts are influenced by peak memory consumption of outliers, whereas the scatter plot shows medians. Compared to the reference baselines, isolating sandboxes in minimal VMs (i.e. HL) reduces deployment count by 3.5× compared to Proc, and adding a minimal OS to the VM (i.e. NVX) reduces it by a similar factor. Most importantly, Nanvix in a worst-case adversarial scenario where each user VM instance is connected to a different system VM (i.e. NVX-1to1) is on par with snapshot-based baselines. This situation corresponds to a deployment where each tenant only runs a single application once, and is extremely rare (see Fig. 3). On the other hand, this experiment is a best-case scenario for snapshot based baselines because we run the same application for all requests, and thus all requests can reuse the exact same snapshot.
Table 4. User VM boot time breakdown (We measure the latency of each phase involved in booting a user VM and executing the HTTP echo application in standalone mode, i.e. without a system VM and interacting with the user VM via standard I/O.)
Compared to snapshot-based baselines (i.e. Firecracker-S and CloudHypervisor-S), Nanvix shows similar cold start latency. This is because, under the hood, Nanvix spawns the system VM from a snapshot using CloudHypervisor, so the latter’s cold start latency dominates end-to-end latency and determines its p99. The reason why Nanvix is 10 ms faster than CloudHypervisor-S is that the system VM uses an inmemory filesystem whereas CLH-S uses a block device. The reason why CloudHypervisor-based systems are marginally faster than Firecracker-S is that the latter must create the network TAP device in a separate (sudo) bash script, and just the call to sudo adds upwards of 10 ms. Most importantly, if we exclude the time to create the system VM because e.g. there is another in-flight function from the same tenant, Nanvix-uV has 6-10× lower cold start latencies than snapshot-based systems, and is comparable to the reference baselines. This makes it possible to spawn a new user VM for each new request. User VM boot time breakdown. Tab. 3 presents the finegrain cold start latencies in Nanvix. When the system VM is already running, a user VM starts in 7 ms at p50, and 11 ms at p99. This time includes sending an HTTP request to the control-plane, spawning a new user VM, opening a port in the system VM’s gateway, reading the payload from the user, sending it all the way to the user thread in the user VM, and back. To isolate the time to create a user VM, Tab. 4 presents a breakdown of the time spent creating (and executing) the user VM for the same application in standalone mode, where we communicate with the user VM via standard I/O (see Fig. 8). Removing the control plane and system VM interactions brings cold start latencies to 3.5 ms at p50 and 5.5 ms at p99. This latency is dominated by the time to load the
Warm start latency. We have shown that Nanvix’ multikernel design reduces cold start latency and improves memory density, but it also introduces an additional hop in the data path (see Fig. 7). Tab. 5 presents the round-trip latency of sending an HTTP echo request with a 32 B payload, and 9
FC CLH UK gV NVX-1to1 FC-S CLH-S NVX HL Proc
Memory footprint
100 MiB 10 MiB 1 MiB 100 KiB 1ms
10ms
100ms Cold start latency
1s
9
100
10s
(a) Cold start against memory footprint
20 18
43
92 104 104 134
101 102 # instances in 1 GiB
552 1646 103
(b) Instance count
Figure 9. Deployment density (We spawn sandboxes in a closed-loop until we consume 1 GiB of system memory, and measure each sandbox’s contribution by reading MemAvailable from /proc/meminfo. Each dot represents the median cold start latency and memory footprint across runs. The sandbox count is the minimum number of sandboxes, across all 10 runs, we can fit in 1 GiB.) p50 (us)
p99 (us)
Slowdown
Firecracker Cloud Hypervisor Unikraft gVisor Nanvix Hyperlight Process
240 195 214 387 325 207 121
345 262 271 495 422 274 174
0.74x 0.60x 0.66x 1.19x 1.00x 0.64x 0.37x
500 us 400 us Latency
Baseline
300 us 200 us 100 us 32 B
512 B 2 KiB 4 KiB Payload size Figure 10. Throughput characteristic (We present the p50, marker, and p99, faded. See Tab. 5 for the legend.)
Table 5. HTTP round-trip latency (Slowdown is reported relative to Nanvix.)
7.3
we report p50 and p99 across a million iterations and the slowdown compared to Nanvix. The results in Tab. 5 show that Nanvix introduces, at worst, a 50% overhead on round-trip latency compared to the other virtualized baselines. Our profiling indicates that this overhead is not due to the additional hop, but rather to the additional signaling in the user VM. This could be addressed by exposing a shared memory area from the system VM to the user VM, and is something we plan on exploring as future work. In Fig. 10 we repeat the same experiment increasing the size of the payload. Thanks to the bulk data transfer optimization described in §4.3, increasing the size of the payload does not affect request latency in Nanvix even though the payload size exceeds the PMIO message size. However, this is not the case when the payload size exceeds a guest page size of 4 KiB. The current bulk data transfer protocol implementation only supports transferring data within a page, but we are working on an improved version that uses scatter/gather syntax across guest pages.
128 B
End-to-end macrobenchmark
In this section we show how Nanvix’ improvements in cold start latency and memory density influence fleet-wide utilization metrics when replaying a production serverless trace. We replay the first minute of the first region of the Huawei trace from 2025 [21] because it includes the unique tenant identifier for each function invocation. We replay a downsampled version of the trace, and keep reducing the downsampling ratio, i.e. increasing requests-per-second (RPS), until we either exhaust memory capacity or measure a spike in per-request latency. Based on the maximum sustainable throughput on a single node we derive how many servers we would need to run this trace assuming perfect load balancing. We drive the trace execution from an open-loop client with a bounded number of in-flight requests, but we size this upper-bound such that the client is never saturated. For Nanvix, the control plane handles the invocation requests and spawns system VM and user VM instances accordingly. For the other baselines, we implement a multi-threaded HTTP 10
Baseline
10 s
Latency
1s 100 ms 10 ms 1
10 100 Request rate [RPS]
1000
Peak RPS Mem (MiB) # Servers
Firecracker Cloud Hypervisor Unikraft gVisor Firecracker-S CloudHypervisor-S Hyperlight Process
43.1 43.1 43.1 8.62 43.1 43.1 862 4310
2211 5479 1617 151 382 125 230 553
Nanvix
862
1087
100 20x 100 20x 100 20x 500 100x 100 20x 100 20x 5 1x 1 0.20x 5
1x
Table 6. Trace replay summary (Peak RPS is the maximum sustainable request rate, and peak memory is taken from the corresponding run. The number of servers is projected based on the single-node peak RPS and the trace size.)
Figure 11. Trace replay (We replay the Huawei trace and measure the per-request p50, marker, and p99, shade, latency. See the table at the right-hand side for the legend.)
server in Rust with a worker pool that we are also careful not to saturate. Each request executes the same HTTP echo application and each request experiences a cold start. This is a slightly pathological case, i.e. one where the serverless platform is configured with a keep-alive of 0, but it helps us test the limits of each solution. Fig. 11 presents the p50 and p99 of the per-request latency as we decrease the downsampling factor of the trace replay, thus increasing the overall RPS. The trace has sudden bursts of requests, very common in serverless traces [46], which introduce higher per-request latency. Still, Nanvix has consistently one to two orders of magnitude lower per-request latency, and achieves an order-of-magnitude improvement in maximum sustainable RPS. Tab. 6 summarizes the peak RPS achieved by each system, as well as the peak memory consumed on the corresponding replay. Based on the memory consumption we can assert that none of the systems exhaust the system memory, so in this case the determining factor for peak RPS is contention on host resources. Given that Nanvix has a simple per-request virtual environment and multiplexes all I/O requests from the same tenant, it can achieve much higher peak RPS. Note that network namespace creation and allocation can very quickly dominate request latency, much earlier than the current system’s elbow points [10, 33]. To prevent namespace allocation from influencing the results, both Nanvix’ control plane and the other baseline’s server allocate namespaces from a pre-initialized pool. Most importantly, based on the peak single-node RPS, a serverless provider using Nanvix would need 20-100× less physical servers to serve the same workload. This is a substantial improvement in achieved density and justifies Nanvix’ design. This projection of total number of servers assumes perfect load-balancing and overlooks the additional challenges from routing functions to a server with a system VM from the same tenant. We also keep system VMs alive after execution, but do not pre-warm them in a pool
(which could be achievable in practice). Given the substantial improvements achieved by Nanvix we do not believe these simplifications affect our claims.
8
Related Work
Execution environments for serverless applications. Serverless providers navigate a trade-off between security, performance, density, and compatibility when choosing an execution environment for applications. Systems like Faasm [48] or CloudFlare Workers [12] achieve low cold starts and high density sacrificing security and compatibility by executing applications in WebAssembly sandboxes. Container-based solutions offer compatibility and performance similar to solutions based on system call filtering like gVisor [57] or Junction [18]. In both cases, applications still share a common OS kernel, posing a security threat. The gold standard for security in the cloud is hypervisor isolation as provided by VMs. VMs, however, are slow to start up and have a high memory footprint. Even micro-VMs like Firecracker [1] or Cloud Hypervisor [20] introduce substantial overheads in terms of cold start latency and deployment density. Systems like Virtines [54] and Hyperlight [38] showed that these overheads are not due to hypervisor isolation, but rather expensive initialization of OS subsystems. Specialized OS solutions, like unikernels [9, 29, 41], try and minimize the guest OS by linking it with the application code in a single binary, but still introduce latency and memory overheads which are an open problem addressed by related work [19]. In addition, most unikernels have compatibility issues due to their single-address space design, that prevents them from supporting applications that use the fork system call [27]. If a bug or vulnerability is found in the unikernel’s system libraries, all affected applications must be recompiled, complicating the management and deployment of such systems at scale. 11
Intra-tenant resource sharing. Ultimately, execution environments adopting VMs and full-weight OSs for compatibility will compromise either cold start latency and/or deployment density. As described in §2, these limitations can be addressed by sharing components across applications belonging to the same tenant. Some systems like RunD-V [33] share the VM and OS explicitly by spawning applications inside containers in a per-tenant VM, whereas others like Ant FaaS [10] or AWS Lambda Snap Start [3] exploit VM snapshots to: (i) reduce cold start latency and per-instance memory footprint [50], and (ii) reduce memory duplication across instances with generic OS images. As argued in §2 all these systems introduce hard-to-quantify trade-offs in resource contention, system complexity, and disk pressure.
be deployed in separate servers for improved load balancing. Such a set-up would introduce additional latency to inter-kernel I/O requests, so it should be weighed carefully. Hardware offload. The current Nanvix prototype uses inmacro-kernel options for the different I/O stacks in the system VM. However, with the increased adoption of hardware offload [22], kernel bypass [18, 58], and storage disaggregation [6], we envision a Nanvix deployment where different I/O stacks are offloaded to different hardware devices. Whether user VM requests are still routed through a centralized component or not is an open question that we plan on exploring as future work. Lessons learned. When we started working on Nanvix we assumed deployment density was determined by perapplication memory footprint. However, after trace analysis and experimentation we realized that cold start latencies also played an important role and, most importantly, resource contention on the host OS resources. Choosing to provide application compatibility at the POSIX layer introduces initial porting efforts [32] but unlocks new opportunities for optimization. In comparison to other systems that also offer POSIX compatibility like unikernels [29] or other libOSs [36] that are constrained by their execution in a single-address space, which limits the type of applications they can support, we do not envision any such limitations for Nanvix.
OS disaggregation. Nanvix follows a different approach in that it disaggregates the components in the guest OS in two virtualized environments. Nanvix multikernel design is inspired by other distributed OSs like Barrelfish [7] or Helios [40], and disaggregated OSs like LegOS [47] and FractOS [52]. The user VM runs a micro-kernel with a message oriented architecture inspired from that of L3 [34] and its multiple successors [23]. Dandelion [28] argues for an explicit I/O and compute separation which enables the system to achieve elasticity at the cost of application compatibility. Alternative isolation mechanisms. Some systems explore different forms of hardware-based isolation and sharing. ORC [44] uses hardware capabilities to share memory at object granularity whereas CubicleOS [45] and ERIM [51] use compartments and memory protection keys. These systems depart from the traditional VM abstraction and generic hardware features, and are orthogonal to Nanvix.
9
10
Conclusions
Nanvix revisits the long-standing VM abstraction and shows that a principled split between ephemeral execution state and persistent system state enables a fundamentally more efficient serverless substrate. By introducing user VMs with a minimal, purpose-built micro-kernel and a shared system VM that encapsulates heavyweight OS functionality, Nanvix avoids the resource stranding, resizing overheads, and snapshot complexity that limit existing container- and VMbased approaches. This separation allows providers to safely amortize OS and runtime costs across same-tenant applications while preserving strong isolation boundaries and maintaining compatibility with unmodified applications. Our evaluation demonstrates that this split design translates into tangible gains: user VMs start in a few milliseconds, system VMs can be pooled and reused across tenants, and the combined deployment path remains well below the latency of hot-plugging or snapshot-heavy baselines. More importantly, Nanvix improves deployment density by making intra-tenant sharing explicit and efficient, enabling significantly more sandboxes per GiB and reducing the number of servers required to serve production-scale traces.
Discussion
We believe Nanvix disaggregated multikernel design offers a deployment and configuration flexibility that can be leveraged in many different use-cases where efficient ephemeral execution matters. Single-tenant deployments. We envision deployment scenarios where it is not necessary to execute the system VM inside a virtualized environment, and it instead can execute in the host as a system process. In this case the system process would be similar to an emulated device. Ephemeral execution of untrusted code. User VMs, particularly in standalone mode, introduce minimal cold start latencies with strong isolation (see Tab. 4). User VMs could therefore be used to sandbox execution of LLM generated code that only needs access to standard I/O. Disaggregated deployment. At the moment, we only consider Nanvix deployments where the system VM and user VM are co-located on the same physical server. However, both VMs already communicate via TCP, so they could also
References [1] Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. Firecracker: Lightweight Virtualization for Serverless Applications. In 17th 12
USENIX Symposium on Networked Systems Design and Implementation, NSDI, 2020. [2] Ahmed Alquraan, Abdelrahman Baba, Rafael Mendes, Sameh Elnikety, Paul Batum, Yan Chen, Seth Safi, Hamid Henry Fine, and Samer AlKiswany. Drops: Managing serverless resource pools in microsoft azure functions. In Proceedings of the Twentyfirst European Conference on Computer Systems, EuroSys ’26, 2026. [3] AWS. Improving startup performance with Lambda SnapStart. https: //docs.aws.amazon.com/lambda/latest/dg/snapstart.html, 2026. [4] AWS. Lambda - Provisioned Concurrency. https://docs.aws.amazon. com/lambda/latest/dg/provisioned-concurrency.html, 2026. [5] AWS. Lambda - Runtime Environment. https://docs.aws.amazon.com/ lambda/latest/dg/lambda-runtime-environment.html, 2026. [6] Wei Bai, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, Rebecca Chow, Jeff Cohen, Mahmoud Elhaddad, Vivek Ette, Igal Figlin, Daniel Firestone, Mathew George, Ilya German, Lakhmeet Ghai, Eric Green, Albert Greenberg, Manish Gupta, Randy Haagens, Matthew Hendel, Ridwan Howlader, Neetha John, Julia Johnstone, Tom Jolly, Greg Kramer, David Kruse, Ankit Kumar, Erica Lan, Ivan Lee, Avi Levy, Marina Lipshteyn, Xin Liu, Chen Liu, Guohan Lu, Yuemin Lu, Xiakun Lu, Vadim Makhervaks, Ulad Malashanka, David A. Maltz, Ilias Marinos, Rohan Mehta, Sharda Murthi, Anup Namdhari, Aaron Ogus, Jitendra Padhye, Madhav Pandya, Douglas Phillips, Adrian Power, Suraj Puri, Shachar Raindel, Jordan Rhee, Anthony Russo, Maneesh Sah, Ali Sheriff, Chris Sparacino, Ashutosh Srivastava, Weixiang Sun, Nick Swanson, Fuhou Tian, Lukasz Tomczyk, Vamsi Vadlamuri, Alec Wolman, Ying Xie, Joyce Yom, Lihua Yuan, Yanzhao Zhang, and Brian Zill. Empowering Azure Storage with RDMA. In 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI, 2023. [7] Andrew Baumann, Paul Barham, Pierre-Evariste Dagand, Tim Harris, Rebecca Isaacs, Simon Peter, Timothy Roscoe, Adrian Schüpbach, and Akhilesh Singhania. The multikernel: a new os architecture for scalable multicore systems. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, 2009. [8] Erik Bosman, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. Dedup est machina: Memory deduplication as an advanced exploitation vector. In 2016 IEEE Symposium on Security and Privacy, SP ’16, 2016. [9] James Cadden, Thomas Unger, Yara Awad, Han Dong, Orran Krieger, and Jonathan Appavoo. SEUSS: skip redundant paths to make serverless fast. In EuroSys, 2020. [10] Xiaohu Chai, Tianyu Zhou, Keyang Hu, Jianfeng Tan, Tiwei Bie, Anqi Shen, Dawei Shen, Qi Xing, Shun Song, Tongkai Yang, Le Gao, Feng Yu, Zhengyu He, Dong Du, Yubin Xia, Kang Chen, and Yu Chen. Fork in the road: reflections and optimizations for cold start latency in production serverless systems. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, OSDI ’25, 2025. [11] Google Cloud. Run. https://cloud.google.com/run, 2026. [12] CloudFlare. The cloud that works for developers, not the other way around. https://workers.cloudflare.com/, 2026. [13] OS Dev. ACPI. https://wiki.osdev.org/ACPI, 2026. [14] OS Dev. VirtIO. https://wiki.osdev.org/Virtio, 2026. [15] AWS Docs. Handling uniqueness with Lambda SnapStart. https://docs. aws.amazon.com/lambda/latest/dg/snapstart-uniqueness.html, 2026. [16] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen. Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’20, 2020. [17] Fission. Open source Kubernetes-native Serverless Framework. https: //fission.io, 2026.
[18] Joshua Fried, Gohar Irfan Chaudhry, Enrique Saurez, Esha Choukse, Íñigo Goiri, Sameh Elnikety, Rodrigo Fonseca, and Adam Belay. Making kernel bypass practical for the cloud with Junction. In NSDI, 2024. [19] Gaulthier Gain, Benoît Knott, Cyril Soldani, and Laurent Mathy. Memory matters: Load-time deduplication for unikernels. In Proceedings of the 2025 ACM Symposium on Cloud Computing, SoCC ’25, 2026. [20] Cloud Hypervisor. Run Cloud Virtual Machines Securely and Efficiently. https://www.cloudhypervisor.org/, 2026. [21] Artjom Joosen, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Luke Darlow, Jianfeng Wang, Qiwen Deng, and Adam Barker. Serverless cold starts and where to find them. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, 2025. [22] Jongyul Kim, Insu Jang, Waleed Reda, Jaeseong Im, Marco Canini, Dejan Kostić, Youngjin Kwon, Simon Peter, and Emmett Witchel. Linefs: Efficient smartnic offload of a distributed file system with pipeline parallelism. In 28th Symposium on Operating Systems Principles, SOSP ’21, 2021. [23] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. sel4: formal verification of an os kernel. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, 2009. [24] Paul Kocher, Jann Horn, Anders Fogh, , Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom. Spectre attacks: Exploiting speculative execution. In 40th IEEE Symposium on Security and Privacy, S&P’19, 2019. [25] Sumer Kohli, Shreyas Kharbanda, Rodrigo Bruno, Joao Carreira, and Pedro Fonseca. Pronghorn: Effective checkpoint orchestration for serverless hot-starts. In Proceedings of the Nineteenth European Conference on Computer Systems, EuroSys ’24, 2024. [26] Leonid Kondrashov, Lazar Cvetković, Hancheng Wang, Boxi Zhou, and Dmitrii Ustiugov. Melding the serverless control plane with the conventional cluster manager for speed and resource efficiency. https: //arxiv.org/abs/2505.24551, 2026. [27] John Alistair Kressel, Hugo Lefeuvre, and Pierre Olivier. ufork: Supporting posix fork within a single-address-space os. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP ’25, 2025. [28] Tom Kuchler, Pinghe Li, Yazhuo Zhang, Lazar Cvetković, Boris Goranov, Tobias Stocker, Leon Thomm, Simone Kalbermatter, Tim Notter, Andrea Lattuada, and Ana Klimovic. Unlocking true elasticity for the cloud-native era with dandelion. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP ’25, 2025. [29] Simon Kuenzer, Vlad-Andrei Bădoiu, Hugo Lefeuvre, Sharan Santhanam, Alexander Jung, Gaulthier Gain, Cyril Soldani, Costin Lupu, Ştefan Teodorescu, Costi Răducanu, et al. Unikraft: Fast, specialized unikernels the easy way. In EuroSys, 2021. [30] Orestis Lagkas Nikolos, Clhoe Alverti, Stratos Psomadakis, Georgios Goumas, and Nectarios Koziris. Squeezy: Rapid vm memory reclamation for serverless functions. In Proceedings of the Twentyfirst European Conference on Computer Systems, EuroSys ’26, 2026. [31] Andrea Lattuada, Travis Hance, Jay Bosamiya, Matthias Brun, Chanhee Cho, Hayley LeBlanc, Pranav Srinivasan, Reto Achermann, Tej Chajed, Chris Hawblitzel, Jon Howell, Jacob R. Lorch, Oded Padon, and Bryan Parno. Verus: A practical foundation for systems verification. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP ’24, 2024. [32] Hugo Lefeuvre, Gaulthier Gain, Vlad-Andrei Bădoiu, Daniel Dinca, Vlad-Radu Schiller, Costin Raiciu, Felipe Huici, and Pierre Olivier. Loupe: Driving the development of os compatibility layers. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 13
Design and Implementation, OSDI’18, 2018. [48] Simon Shillaker and Peter Pietzuch. Faasm: Lightweight isolation for efficient stateful serverless computing. In USENIX ATC, 2020. [49] Ariel Szekely, Adam Belay, Robert Morris, and M. Frans Kaashoek. Unifying serverless and microservice workloads with sigmaos. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP ’24, 2024. [50] Dmitrii Ustiugov, Plamen Petrov, Marios Katebzadeh, Michal Sherr, and Boris Grot. Benchmarking, analysis, and optimization of serverless function snapshots. In Proceedings of the Twenty-Sixth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’21, 2021. [51] Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, and Deepak Garg. ERIM: Secure, efficient inprocess isolation with protection keys (MPK). In 28th USENIX Security Symposium, USENIX Security 19, 2019. [52] Lluís Vilanova, Lina Maudlej, Shai Bergman, Till Miemietz, Matthias Hille, Nils Asmussen, Michael Roitzsch, Hermann Härtig, and Mark Silberstein. Slashing the disaggregation tax in heterogeneous data centers with fractos. In Proceedings of the Seventeenth European Conference on Computer Systems, EuroSys ’22, 2022. [53] Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. Peeking behind the curtains of serverless platforms. In 2018 USENIX Annual Technical Conference, ATC’18, 2018. [54] Nicholas C Wanninger, Joshua J Bowden, Kirtankumar Shetty, Ayush Garg, and Kyle C Hale. Isolating functions at the Hardware Limit with Virtines. In EuroSys, 2022. [55] WASI. WebAssembly System Interface. https://wasi.dev/, 2026. [56] Jidong Xiao, Zhang Xu, Hai Huang, and Haining Wang. Security implications of memory deduplication in a virtualized environment. In 2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN ’13, 2013. [57] Ethan G Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C ArpaciDusseau, and Remzi H Arpaci-Dusseau. The true cost of containing: A gVisor case study. In HotCloud, 2019. [58] Irene Zhang, Amanda Raybuck, Pratyush Patel, Kirk Olynyk, Jacob Nelson, Omar S. Navarro Leija, Ashlie Martinez, Jing Liu, Anna Kornfeld Simpson, Sujay Jayakar, Pedro Henrique Penna, Max Demoulin, Piali Choudhury, and Anirudh Badam. The demikernel datapath os architecture for microsecond-scale datacenter systems. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, 2021. [59] Ziming Zhao, Mingyu Wu, Haibo Chen, and Binyu Zang. Characterization and reclamation of frozen garbage in managed faas workloads. In Proceedings of the Nineteenth European Conference on Computer Systems, EuroSys ’24, 2024.
’24, 2024. [33] Zijun Li, Chenyang Wu, Chuhao Xu, Quan Chen, Shuo Quan, Bin Zha, Qiang Wang, Weidong Han, Jie Wu, and Minyi Guo. Lightweight and holistic-scalable serverless secure container runtime for highdensity deployment and high-concurrency startup. IEEE Transactions on Computers, 2025. [34] Jochen Liedtke. Improving ipc by kernel design. In Proceedings of the Fourteenth ACM Symposium on Operating Systems Principles, SOSP ’93, 1993. [35] Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Anders Fogh, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, and Mike Hamburg. Meltdown: Reading kernel memory from user space. In 27th USENIX Security Symposium, USENIX Security 18, 2018. [36] Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. Unikernels: library operating systems for the cloud. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’13, 2013. [37] Firecracker Maintainers. Entropy for Clones. https: //github.com/firecracker-microvm/firecracker/blob/main/docs/ snapshotting/random-for-clones.md, 2026. [38] Microsoft. Introducing Hyperlight: Virtual machine-based security for functions at scale. https://opensource.microsoft.com/blog/2024/11/ 07/introducing-hyperlight/, 2024. [39] Till Miemietz, Viktor Reusch, Matthias Hille, Lars Wrenger, Jana Eisoldt, Jan Klötzke, Max Kurze, Adam Lackorzynski, Michael Roitzsch, and Hermann Härtig. Metteagle: costs and benefits of implementing containers on microkernels. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, OSDI ’25, 2025. [40] Edmund B. Nightingale, Orion Hodson, Ross McIlroy, Chris Hawblitzel, and Galen Hunt. Helios: heterogeneous multiprocessing with satellite kernels. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, 2009. [41] Federico Parola, Sisu Qi, Akhilesh B. Narappa, and K. K. Ramakrishnan. Sure: Secure unikernels make serverless computing rapid and efficient. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24, 2024. [42] QEMU. Inter-VM Shared Memory device. https://www.qemu.org/ docs/master/system/devices/ivshmem.html, 2026. [43] Lazar Sahraei, Dmitrii Ustiugov, Mahdi Mohammadi Amiri, and Antoni Wolnikowski. Dirigent: Lightweight serverless orchestration. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP, 2024. [44] Vasily A. Sartakov, Lluís Vilanova, Munir Geden, David Eyers, Takahiro Shinagawa, and Peter Pietzuch. ORC: Increasing cloud memory density via object reuse with capabilities. In 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’23, 2023. [45] Vasily A. Sartakov, Lluís Vilanova, and Peter Pietzuch. Cubicleos: a library os with software componentisation for practical isolation. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’21, 2021. [46] Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In Proceedings of the 2020 USENIX Annual Technical Conference, ATC ’20, 2020. [47] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. Legoos: a disseminated, distributed os for hardware resource disaggregation. In Proceedings of the 13th USENIX Conference on Operating Systems 14