Pomegranate: A Lightweight Compartmentalization Architecture using Virtualization Extensions Shriram Raja∗
Zhiyuan Ruan∗
Richard West
[email protected] Boston University Boston, Massachusetts, USA
[email protected] Boston University Boston, Massachusetts, USA
[email protected] Boston University Boston, Massachusetts, USA
arXiv:2605.07008v1 [cs.CR] 7 May 2026
Abstract
formal verification [18]. Others have considered system designs that adhere to the principle of least privilege, ensuring a software component only has the capabilities to do what is necessary but no more [20, 30, 45]. Micro-kernels [18, 22] isolate software components into domains of the least privilege necessary for their correct operation. However, the question remains how to achieve the same level of component isolation in the monolithic systems currently in widespread use. Is it even possible to securely isolate regions within an existing system without significant re-engineering and impact on overall performance? Compartmentalization is one approach that is gaining widespread consideration for increased security and safety of existing monolithic systems [25, 33]. Compartmentalization provides protection domains for separate software components, limiting access to only the code and data that is necessary for a specific unit of functionality. Any compartment that makes non-local procedure calls does so through secure inter-compartment communication mechanisms. Prior work has considered memory protection keys [8, 9, 12, 35, 40] as a way to implement kernel and user-level compartmentalization. However, without an extra ring of protection to enforce access rights to compartments, privileged instructions within the kernel may bypass protection key security. For this reason, techniques such as binary rewriting have been used to scan the kernel for security-critical instructions and replace them with either safe instructions or those which trap into a trusted in-kernel monitor [12]. Rather than having to scan and replace security-critical instructions, we propose a compartmentalization technique that relies on machine virtualization. While virtualization has long been used to consolidate multiple guest OSes onto the same physical machine [2, 31, 36], it also provides a method to employ an extra ring of protection to an existing system. Microsoft now uses the Hyper-V hypervisor to provide virtualization-based security (VBS) for Windows 11 [26]. Others have used virtualization as a way to enforce guest compartmentalization [13, 29], but these works limit compartments to kernel modules (e.g. for device drivers) and leave the core kernel as one large trusted codebase. While some researchers argue that virtualization is too expensive to implement fine-grained kernel compartmentalization [12], we show how it can be used efficiently on a modern Linux system. In this paper, we present
The monolithic nature of widely used commodity operating systems means that vulnerabilities in one software component potentially compromise the entire kernel. Formally verifying these systems, or redesigning them altogether as microkernels, according to the principle of least privilege, requires significant effort. Researchers have therefore considered compartmentalization techniques that minimize or totally avoid changes to existing systems. However, current approaches use techniques such as Memory Protection Keys (MPKs), necessitating extensive code analysis to ensure security, or use virtualization by instrumenting the kernel with calls to the glue code that switches compartments. In this work, we present Pomegranate, a framework that uses hardware-assisted virtualization to securely compartmentalize an existing system with minimal to no modifications to its source code. Allowed interactions between compartments are defined using an access-control policy and strictly enforced using Extended Page Tables. Using special sentry functions, Pomegranate is able to check all crosscompartment transitions without trapping into the hypervisor. We demonstrate the efficacy of Pomegranate on a compartmentalized Linux network stack using the igc NIC driver. Experiments show the overheads of our approach are negligible at MTU-sized packets when compartment boundaries are carefully established to avoid excessive intercompartment communication. CCS Concepts: • Security and privacy → Operating systems security; Virtualization and security. Keywords: Kernel Compartmentalization, Virtualization, Security
1
Introduction
Many operating systems (OSes) in use today are monolithic in structure. A software fault or security breach in any kernel component may compromise the entire system. Researchers have studied ways to improve the safety and security of systems, using techniques such as software fault isolation [6, 11, 24, 41], memory-safe languages [3, 4], hardware memory tagging [1, 5, 25], capabilities [27, 42], and ∗ Both authors contributed equally.
1
a framework called Pomegranate, which uses hardwareenforced virtualization to isolate kernel components into fine-grained compartments. We explain how Pomegranate does this using Intel virtualization extensions (VT-x) and redirection protection (VT-rp), along with other hardwareenforcement features. We show how Pomegranate implements secure and efficient inter-compartment gate calls, with the assistance of a small hypervisor that hosts a policy enforcement manager. This manager is initialized at boot-time with the list of compartments and their access rights to data objects and kernel functions. Using special sentry functions set up by the manager, Pomegranate is able to apply compartmentalization to an existing system with minimal to no modifications of the kernel. The approach incurs low overheads, as a result of inter-compartment communication and invalid accesses being handled in the kernel rather than the hypervisor. Contributions: We describe the implementation of Pomegranate, which is built using the Quest-V partitioning hypervisor [21, 43]. We explain how Pomegranate’s kernel compartmentalization is achieved for both a source-code annotated version of Linux and also an unmodified system binary. The overheads of Pomegranate are demonstrated for varying numbers of compartments applied to the network stack. Results show less than 0.01% throughput overhead at MTU-sized packets for up to 32 compartments, compared to an unmodified monolithic system, when boundaries are carefully chosen to avoid excessive inter-compartment communication.
2
machines using guest physical address (GPA) to host physical address (HPA) memory mappings. EPTs define both the permissions and physical memory range accessible to each guest. Each VMCS contains an Extended Page Table Pointer (EPTP) field that holds the host physical address of the EPT root table. EPTs are not mapped into, or modifiable by, any guest address space. Invalid memory accesses trigger traps into the hypervisor. While EPTs were originally designed to host multiple virtual machines on shared hardware, they are also able to isolate components within a single VM [13, 26, 29]. This approach moves the root of trust from the kernel to the hypervisor. The hypervisor creates multiple EPTs for the same VM, with each EPT corresponding to an isolated compartment. When a compartment executes, the hypervisor switches to the corresponding EPT, which restricts the accessible instructions and data. Virtual Machine Functions (vmfunc): Switching EPTs normally requires trapping into the hypervisor, which is expensive because the hypervisor must save and restore the guest’s architectural state. The Intel vmfunc instruction avoids this cost by allowing the guest to switch EPTs directly. The hypervisor pre-loads an EPTP list with up to 512 EPT root pointers. The guest issues a vmfunc with the desired index to switch the active EPT while remaining in guest mode. The instruction completes in less than two hundred cycles. The vmfunc instruction does not perform any control flow redirection, which implies that after the EPT switch, execution immediately continues from the next instruction. Thus, for an attacker’s insertion of a vmfunc to succeed, the next instruction has to be mapped into the correct destination EPT that the attacker intends to switch to. It also does not perform any control-flow validation. Hence, additional logic is needed to check that a transition is allowed. Prior work has used an interface definition language to generate this glue code [29]. Virtualization Exceptions: EPT violations are typically handled by traps into the hypervisor, but a subset of them are deliverable to the guest via Virtualization Exceptions (#VE). Such exceptions allow the guest to handle certain EPT violations directly, avoiding the overhead of trapping into the hypervisor. Support for Virtualization Exceptions is activated within the VM-execution control settings of a VMCS. Each EPT paging-structure entry that references a page additionally uses bit 63 to control where EPT violations are delivered. When bit 63 is set to 1, #VE delivery is suppressed for the corresponding page and violations trap to the hypervisor. When cleared, EPT violations are delivered to the guest as #VEs. The CPU stores the exit reason, exit qualification (indicating whether the access was a read, write, or instruction fetch), the faulting guest linear and physical addresses, and the current EPTP index in a pre-allocated virtualization-exception information area [9]. We define a custom #VE handler that examines this information area,
Background
Figure 1. Extended Page Table Mechanism Virtualization and EPTs: Intel VT-x [9] provides hardware support for virtualization, including Virtual Machine Control Structures (VMCSs) and dedicated VM entry/exit instructions. One key component is Extended Page Tables (EPTs). Extended Page Tables, shown in Figure 1 are data structures managed by the hypervisor, which isolate virtual 2
• functions that are executable within the compartment, • functions that can be called in another compartment, • data objects that are readable and/or writeable, and • the execution context, including threads and their stacks, effective user ids (euids) that define capabilities, and the domain (e.g., kernel- versus user-space).
checks whether the access is allowed, and switches to the appropriate EPT using vmfunc. This avoids trapping into the hypervisor and does not require generating custom glue code for each compartment transition. Hypervisor Managed Linear Address Translation (HLAT): EPTs control guest-physical to host-physical translation but cannot control guest-virtual to guest-physical translation. An attacker who compromises the guest kernel is able to exploit this gap through a remapping attack [37], as illustrated in Figure 2. Consider a system that stores security policy structures, such as access control lists or compartment permissions, in a page marked read-only in the EPT. Even if the system is compromised, an attacker cannot modify the original data. However, the attacker can modify the guest page table to remap the Guest Virtual Address (GVA) of the sensitive data to a different Guest Physical Address (GPA) that is marked writable in the EPT. The attacker may then populate this new physical page with arbitrary policy data, potentially gaining unauthorized access or escalating privileges. Intel Hypervisor-managed Linear Address Translation (HLAT), part of VT-rp (redirection protection), addresses this by allowing the hypervisor to fix a subset of GVA-toGPA translations. For pages registered in the HLAT paging structures, the CPU resolves the translation using the hypervisor-controlled HLAT entries rather than the guest page tables. Each HLAT entry either completes the translation or triggers a restart from the guest CR3 paging structures, so the hypervisor controls only the subset of mappings that require protection. Guest page table modifications have no effect on HLAT-resolved translations, which prevents the remapping attack described above. The HLAT structures themselves are marked read-only in the EPT, so the guest is unable to modify them. Quest-V Partitioning Hypervisor: We use Quest-V [21, 43], to host a compartmentalized version of Linux in this work. Quest-V is a lightweight separation kernel that statically partitions machine resources among one or more guest sandboxes. Each sandbox manages its allocated resources, so Quest-V does not cause VM-exits during normal guest operation. The small hypervisor codebase also reduces the size of the root of trust.
3
Figure 3 shows an example of a compartment definition. While all functions in a compartment can call any other function within the same compartment, the list of callable functions in other compartments is specified in the can_call list. Thus, the can_call list effectively determines the allowed compartment switches. A policy file containing the definitions of all compartments is given as input to the Pomegranate framework. Ideally, this is provided as part of the boot procedure, before the guest system is loaded and operational. The policy file is then able to be used by the hypervisor to establish guest compartments before the guest itself is fully booted. Without loss of generality, Pomegranate currently establishes compartments in the guest after it has booted, considering boot-time exploits out of scope. 3.1
Threat Model
We assume an adversary that is able to exploit vulnerabilities anywhere in the kernel, including both core kernel code and peripheral subsystems, with the goal of escalating privileges and compromising the entire system. We adopt a mutual distrust model in which compartments do not place trust in one another. Each compartment is treated as potentially compromised and is restricted to accessing only those resources explicitly permitted by the compartment policy. We assume that the hypervisor, sentry code, and developer-specified compartment transition policies constitute our trusted computing base (TCB). Our threat model makes the following assumptions. First, we assume a secure boot process that guarantees the kernel image analyzed at design time is identical to the one loaded at runtime. Second, we focus exclusively on software-based attacks and do not consider physical exploits such as coldboot attacks or hardware tampering. The scope of this work is limited to static compartment configurations; we do not currently support dynamically allocated compartments or runtime policy changes.
System Model
4
Our approach follows the mutual distrust model [19], where no single kernel component is trusted. Instead, the Trusted Computing Base (TCB) is limited to the hypervisor, with the aim of avoiding privilege escalation. To that end, the kernel to be compartmentalized should be statically and dynamically analyzed to determine the compartment boundaries [33]. A compartment is defined by the following information:
Architecture
Figure 4 shows the Pomegranate compartmentalization framework. Linux runs as a guest on the Quest-V hypervisor, which partitions hardware resources at boot-time. As a result, Linux has direct access to its allocated cores, physical memory, and I/O devices. A lightweight monitor handles VM-exits. Extended Page Tables (EPTs) control guestphysical to host-physical address translation. Before booting Linux, Quest-V first creates an initial EPT (𝐸𝑃𝑇0 ), where 3
Figure 2. Remapping Attack cmpt_id: 2 can_execute: func1, func2 can_read: obj1, obj2, obj3 can_write: obj1 can_call: func3 execution_context: euid = root
Linux boots, to map the entire guest. When the gate manager creates other compartments, it modifies 𝐸𝑃𝑇0 to unmap data and code that should not be accessible from 𝐶 0 . Consequently, 𝐶 0 ’s policy need not enumerate all accessible objects—it specifies only objects shared with other compartments and those that are involved in the transition to another compartment. For example, if functions in 𝐶 0 can call func1 in 𝐶 2 (Figure 3) and share read access to the obj2, its definition would be as shown in Figure 5. EPTs enforce permissions at 4 KB page granularity. If objects from different compartments reside on the same page, they receive identical permissions, potentially causing information leakage. We currently use compiler directives to enforce page alignment and discuss sub-page isolation approaches in Section 7.
Figure 3. Example Compartment Definition the entire guest physical address space except the monitor itself, is fully accessible (readable, writable, and executable). This EPT defines the default compartment 𝐶 0 . A new EPT is then created for each additional compartment, with restricted mappings. 4.1
Compartment Initialization
The policy file, which is given as input to the Pomegranate framework, defines which functions and data objects belong to each compartment, referencing them by name. EPT construction, however, requires knowledge of the guest physical addresses for each symbol name. A tamper-proof Compartment Initialization Kernel Module (CIKM), protected by its own EPT mappings, bridges this gap. It parses the policy file, resolves names to guest virtual addresses, and walks the kernel page tables to obtain the corresponding GPAs. Symbol names to guest virtual addresses are either obtained using kallsyms_lookup_name, requiring CONFIG_KALLSYMS, or by guest code analysis. As knowledge of GPAs is only available after guest page tables are established, we leave secure bootup support for future work. The CIKM passes the parsed policy to the hypervisor via a hypercall. The gate manager, within the hypervisor, uses this policy to construct a unique EPT for each compartment. It also populates the EPTP list to enable vmfuncbased switching. Access rights for each compartment, having a specific execution context, are recorded in an access matrix. Once initialization completes, a thread executing in compartment 𝐶𝑖 is restricted to functions and data objects accessible from within 𝐶𝑖 . The default compartment 𝐶 0 is handled differently. It uses the initial EPT (𝐸𝑃𝑇0 ) that is created by Quest-V before
4.2
GVA-to-GPA Translation in Pomegranate
GVA-to-GPA translation uses the kernel page table, which is shared across all user processes.1 For VM-exit-free execution, these paging structures should be accessible to all compartments. However, the user-space portion of the page table differs per process. Without HLAT, each process’s paging structures would need to be mapped into every compartment’s EPT, requiring EPT updates on the creation of each new process. HLAT avoids this overhead while also protecting against remapping attacks. Figure 6 illustrates Pomegranate’s use of HLAT. With HLAT enabled, all GVA-to-GPA translations begin at the HLAT Pointer (HLATP) rather than Guest CR3. All memory accesses 1kstart translation at HLATP 2k, which references paging structures created by the gate manager. The CIKM provides GVA-GPA pairs for compartmentalized objects, which the gate manager uses to populate HLAT entries. Where appropriate, HLAT entries specify that translation should restart from Guest CR3, allowing Pomegranate to include only compartmentalized objects (𝐶 1 to 𝐶𝑛−1 ) in the HLAT structure, to minimize its size. Accesses to these 1With KPTI enabled, Linux maintains separate kernel page tables to mit-
igate Meltdown, incurring TLB flush overhead [13]. EPT-based isolation provides equivalent protection, making KPTI unnecessary. 4
Figure 4. Compartmentalization Architecture cmpt_id: 0 can_execute: foo, bar, … can_read: obj2 can_write: can_call: func1 (cmpt_id=2) execution_context: euid = any
call triggers an EPT violation. The gate manager inspects the faulting address to determine if this is a valid crosscompartment call. However, trapping into the hypervisor costs several hundred to a few thousand cycles (Section 6.1). This overhead is acceptable for invalid accesses but prohibitive for valid calls, which should incur minimal overhead. Virtualization Exceptions (#VE) address this by delivering a subset of EPT violations directly to the guest rather than the hypervisor. For valid accesses, vmfunc switches the EPT without a hypervisor trap. Beyond switching EPTs, complete compartment isolation requires changing the thread stack. The #VE handler performs both operations, acting as the gate manager’s delegate within the guest. We call this handler the sentry, which is securely mapped into every compartment.
Figure 5. Example Default Compartment Definition
5
Secure Compartment Switching
5.1
Sentry
Figure 6. Hypervisor-Managed Address Translation in Pomegranate 1 CALL
objects obtain their GPA directly from HLAT 3’k. Accesses to 𝐶 0 and user-space memory restart from Guest CR3, using guest paging structures to produce the GPA 3k, 4k. Since the HLAT paging structures have the same number of levels as Guest CR3 and each level contains only the GPA of the next lower-level structure, the cost of translating a GVA to GPA through HLAT is the same as through Guest CR3. For objects not mapped into HLAT, the translation restarts from Guest CR3, adding at most the overhead of walking one additional paging structure if the translation is not already cached. EPTs enforce read, write, and execute permissions, but do not specify valid cross-compartment accesses. Consider foo in 𝐶 0 (Figure 5) calling a function func1 which belongs to compartment 𝐶 2 . Since func1 is not mapped in 𝐸𝑃𝑇0 , the
Caller Compartment
(#VE) 4 IRET
2 IRET Sentry
(#VE)
Callee Compartment
3 RET
Figure 7. Sentry Call-Return Path The sentry handles compartment transitions. On each transition, it verifies that the access is permitted by the policy, switches the stack and EPT, and switches control back to the caller when the callee returns. The flow of control is shown in Figure 7. Consider a call made to a function that belongs to a remote compartment. The hardware translates the callee function’s GVA to GPA via HLAT, then checks the EPT for the HPA. Since the callee 5
function is not mapped in current compartment, an EPT violation occurs, invoking the #VE handler (sentry) ( 1kin Figure 7). The sentry saves the caller state, switches to the callee EPT, pushes the caller return address onto the callee stack, and sets up an interrupt stack frame such that when it issues an iret, control transfers to the callee function ( 2kin Figure 7). When the callee function executes the ret instruction, another EPT violation occurs as the return address on its stack points to an address in the caller compartment, leading control back to the sentry ( 3kin Figure 7). The sentry restores the caller stack state, and modifies the exception frame that was pushed in step 1kto again use the iret instruction to return control to the caller ( 4kin Figure 7). Now we explain the working of the sentry in detail, using the pseudocode given in Procedure 1. When an unmapped function is called, the return address is first pushed onto the caller stack, and then as the function is unmapped a #VE occurs. The hardware then disables interrupts and pushes onto the caller stack the following state in the order given: the stack segment (SS), stack pointer (RSP, pointing to the caller return address), flags register (RFLAGS), the code segment (CS), and the instruction pointer (RIP, pointing to the callee function), before transferring control to the handler. Consider the call path first (Lines 13-24). The sentry inspects the #VE stack frame to determine the callee function address and the caller return address (Line 13). Then, to verify a transition, the sentry checks whether the callee address appears in the caller compartment’s can_call list (Line 15). Each compartment has a read-only data region (access matrix row) containing its compartment ID and can_call list. This region has the same GVA and GPA across all compartments but maps to a different HPA as shown in Figure 8. To switch EPTs, the sentry must know the callee compartment’s index. Hence, the can_call list is populated by both the callable function addresses and their corresponding EPT indices, allowing the sentry to identify the correct callee compartment. In some cases, the same function may map to different compartments based on execution context. In Pomegranate, we represent the execution context by the effective user ID. The current eUID is changed in one of three ways: execve calls, seteuid calls, and context switches. execve and seteuid are only mapped into 𝐶 0 , so a thread in other compartments cannot use them to change its eUID. Context switches happen either due to timer interrupts or explicit invocations of the scheduler, e.g., as a result of a blocking system call. The scheduler is only mapped into 𝐶 0 , once again preventing changes to the eUID in other compartments. Additionally, we disable kernel preemption, so timer interrupts that call the scheduler are likewise unable to change the eUID. For all non-𝐶 0 compartments, we maintain a 1:1 mapping of execution context to compartment ID. Hence, the sentry
only needs to check the execution context, to determine access rights, when switching from 𝐶 0 to another compartment. For this, we use the utilities in the kernel. In all other cases, the compartment ID implicitly includes the execution context, and hence the user ID need not be read. The sentry read-only region encodes these context-to-compartment mappings alongside the can_call entries. If the callee address is not found in the can_call list, the access is deemed invalid and the gate manager is invoked using a vmcall to handle the policy violation (Line 16). If a match is found, the sentry prepares to switch to the callee compartment. Before switching EPTs, the sentry pushes any callee-saved registers it will use onto the caller’s stack (Line 17). According to the System V Application Binary Interface (ABI) for 64-bit systems, the first six integer arguments reside in registers, with overflow on the stack. The sentry does not modify these registers, ensuring that the callee receives correct arguments. We currently support functions with at most six arguments. Supporting more would require copying stack-based arguments via a shared page. Pointer arguments must reference memory shared between caller and callee as specified in the policy. The sentry then saves the caller stack pointer in the sentry read-write region (Line 18). For this purpose, we allocate a per-compartment stack in the sentry read-write region shared across all compartments. This data structure is preloaded at initialization with base pointers for each compartment’s stack, allowing the sentry to use identical code regardless of whether a compartment has prior saved state. While this data structure is writable in all compartments, corrupting it cannot redirect control flow: it stores only stack pointers, while the actual stack pages containing return addresses remain modifiable only within their respective compartment. It is possible to achieve greater protection, by splitting each compartment’s read-write region into its own page, similar to the read-only region setup (Figure 8) as each compartment only needs access to the stack of pointers associated with its own stack. Once the sentry has completed all operations on the caller stack, it uses the vmfunc instruction and the compartment ID obtained from the can_call list, to switch to the callee EPT (Line 19). After the EPT switch, the sentry loads the callee compartment’s stack pointer from the sentry r/w region (Line 20). The sentry then sets up a stack frame to ensure that (1) when it issues an iret, control transfers to the callee function, (2) when the callee issues a ret a #VE occurs, and (3) the sentry has enough information in the return path to redirect control to the correct caller address. Callee Stack Setup: The sentry pushes the caller compartment ID onto the callee stack, allowing the sentry to identify the correct caller in the return path. The return path is handled differently to the call path, because the sentry does not push a return address onto the stack. To distinguish between a call and a return, the sentry pushes an 8-byte magic 6
Figure 8. Example Sentry Read-Only Data Layout as shadow stacks [9]. Next, the sentry pushes the caller return address onto the callee stack (Line 21). Thus, when the callee issues a ret, a #VE will be triggered, leading control back to the sentry. Finally, the sentry sets up a #VE frame on the callee stack, with the RIP pointing to the address of the callee function, and the RSP pointing to the caller return address on the callee stack (Line 22). This allows the sentry to redirect control flow to the callee function using the iret instruction. Finally, the sentry clears the #VE mask in the Virtualization Exception information area, re-arming #VE, before executing the iret instruction to ”return” to the callee function (Line 24). Figure 9 shows the stack manipulations performed by the sentry in the call path. When the callee returns, the return address on its stack is popped into RIP. Since this address points to an instruction in the caller compartment, a #VE occurs, invoking the sentry with interrupts disabled and a #VE frame pushed onto the callee stack. Once the sentry inspects the magic number, it pops the #VE frame and the magic number (Line 3) before clearing the #VE mask. This ensures that the #VE handler is invoked at the next EPT violation. The sentry then pops the caller compartment ID from the callee stack (Line 5) for use in the vmfunc instruction. Now, all state information pushed onto the callee stack is fully cleared. This clean callee stack pointer is saved in the sentry read-write region (Line 6). This ensures that, in case of nested call-returns, we do not run out of stack memory. The sentry then uses the vmfunc instruction to switch back to the caller EPT (Line 7). The caller’s stack state is restored from the sentry read-write region (Line 8). The sentry restores the callee-saved registers, but preserves RAX, which holds the return value in the 64-bit System V ABI. Finally, the sentry modifies the #VE frame that was pushed in the call path to ensure an iret diverts control to the caller. The RIP in the #VE frame is changed to the return address in the caller function, and 8 is added to the RSP in the #VE frame (Line 10). Thus, when the sentry issues an iret, the resulting RSP skips the caller return address, and control returns to the correct instruction in the caller. The iret instruction restores the previous state of the RFLAGS. Figure 10 shows the stack manipulations performed by the sentry in the return path. While the sentry relies on the callee stack to determine the caller compartment ID, it must be noted that the underlying EPT mappings ensure that a corrupted compartment ID
Procedure 1: Sentry: Call-Return Path Data: #VE stack frame (SS, RSP, RFLAGS, CS, RIP) 1 key = value at #VE stack frame RSP; 2 if key = MAGIC_CALL then // Return Path 3 4 5 6 7 8 9 10 11 12
Pop #VE frame and MAGIC_CALL; Clear #VE mask; Pop caller cmpt_id; Save clean callee stack ptr; vmfunc to caller EPT; Restore caller stack ptr from sentry R/W region; Pop callee-saved registers; Fix #VE frame instruction and stack pointers; iretq ; // Return to Caller else // Call Path
13 14 15 16 17 18 19 20 21
22 23 24
Read callee RIP, return addr from #VE frame; if curr compt is 𝐶 0 then read eUID; Look up callee RIP (and eUID) in can_call list; if no match then vmcall to gate manager; Save callee-saved registers on caller stack; Save caller stack ptr in sentry R/W region; vmfunc to callee EPT; Load callee stack ptr from sentry R/W region; Push caller cmpt_id, MAGIC_CALL and return addr; Set up #VE frame on callee stack; Clear #VE mask; iretq ; // Go to Callee
number, MAGIC_CALL. When the sentry is invoked, it reads the value that the RSP in the #VE frame is pointing to (Line 1). The RSP in the #VE frame always points to the last value that was pushed onto the stack before the #VE occurred. In the case of the call, this is the return address in the caller. In the case of a return, this is the existing stack contents. Thus, if this value is equal to the MAGIC_CALL, the sentry determines that this is a return path, and if not, it treats the value as the caller return address. This magic number approach ensures that the return address on the stack is consistent with typical call-return semantics, making Pomegranate compatible with hardware features such 7
Figure 9. Sentry Stack Manipulation: Call Path
Figure 10. Sentry Stack Manipulation: Return Path by itself cannot result in an access violation. While we currently do not include can_return permissions in the compartment definition, the design of the sentry lends itself to be easily extended to support that case, with a check performed in the return path as well as the call path. The sentry thus handles all compartment transitions without requiring instrumentation of the compartmentalized code. 5.2
1
INT
Interrupt Sentry
2
JMP
Interrupted Compartment
Handler in C0 4 IRET
Sentry
3
IRET (#VE)
Interrupt Handling Figure 11. Interrupt Path in Non-Default Compartments
Interrupts in 𝐶 0 do not require special handling because 𝐸𝑃𝑇0 already contains all the top half handlers. However, for compartments 𝐶 1 𝑡𝑜 (𝑛−1) , mapping all interrupt handlers 8
and their dependencies into every EPT makes it difficult to establish compartment boundaries. The sentry is capable of disabling interrupts in specific compartments by modifying the RFLAGS in the #VE frame, but to follow the semantics of the guest, we support interrupts in all compartments. Pomegranate uses special interrupt sentries to redirect interrupts efficiently to their handlers. Figure 11 shows the flow of control when an interrupt occurs in compartments 𝐶 1 𝑡𝑜 (𝑛−1) .
summarizes the working of the interrupt sentry, which handles the interrupt path, and Procedure 3 shows the interrupt return path, which is handled by the #VE handler, i.e., sentry. When an interrupt occurs in 𝐶 1 𝑡𝑜 (𝑛−1) , the interrupt sentry is invoked with interrupts disabled and an interrupt frame pushed onto the current stack. The interrupt sentry switches to the top of the shared interrupt stack, pushes the interrupted compartment ID and a magic number (distinct from the one used for cross-compartment returns), and sets up the interrupt frame, with the RIP pointing to an instruction in the interrupted compartment. After switching to 𝐶 0 , using a vmfunc, the interrupt sentry finally jumps to the original handler. Since the interrupt sentry always starts from the top of the interrupt stack, any previous values pushed onto it are overwritten. Hence, any modifications to the interrupt stack by other compartments will have no effect. When the handler in 𝐶 0 completes and issues an iret, a #VE is triggered as the return address is not mapped into 𝐶 0 . This invokes the sentry with interrupts disabled and a #VE frame is pushed onto the current (interrupt) stack. If the value pointed to by the RSP in the #VE frame matches the interrupt magic number, the sentry determines that this is an interrupt return path. It then reads the interrupted compartment ID and uses it to switch to the right EPT, before issuing an iret to resume execution (Figure 11). Two magic numbers are chosen from the range of non-canonical addresses to handle the case where the hardware pushes an error code, and the case where it does not. Thus, the sentry handles returns from interrupts in addition to calls and regular returns. Figures 13 and 14 show the stack manipulations performed by the interrupt sentry in the interrupt path and the sentry in the iret path, respectively.
Procedure 2: Interrupt Sentry Input: Interrupt stack frame (SS, RSP, RFLAGS, CS, RIP, optionally Error Code) 1 Push clobbered GP registers onto the current stack; 2 Save stack ptr in sentry R/W region; 3 Switch to interrupt stack; 4 Push interrupted cmpt_id; 5 Push MAGIC_INT_NOERR or MAGIC_INT_ERR; 6 Set up interrupt stack frame; 7 if exception has error code then 8 Push error code vmfunc to 𝐶 0 (EPT index 0); 10 jmp to appropriate interrupt handler; 9
Procedure 3: Interrupt Return (Sentry Path) Input: #VE frame (SS, RSP, RFLAGS, CS, RIP) 1 key = value at #VE frame RSP; 2 if key = MAGIC_INT_NOERR or MAGIC_INT_ERR then 3 Read source cmpt_id from interrupt stack; 4 Clear #VE mask; 5 Save RIP from #VE frame // handler may modify it 6 vmfunc to interrupted cmpt_id; 7 Restore stack ptr from sentry R/W region; 8 Update RIP on exception frame to the saved RIP; 9 Pop saved GP registers from the current stack; 10 if key = MAGIC_INT_ERR then 11 RSP ← RSP + 8 ; // Skip error code 12
iretq ;
6
Experimental Evaluation
In this section, we evaluate the Pomegranate framework. All experiments were conducted on a Supermicro X13SRN-H motherboard equipped with a 13th Generation Intel Core i7-1370PE processor. The system runs Quest-V hosting a 64bit Yocto Linux Scarthgap guest using the kernel version 6.8.1. To eliminate interference from concurrent execution and isolate the overhead attributable to our framework, all experiments were performed with a single active core. First, we evaluate the overhead of the Pomegranate framework and then apply it to a network driver in the Linux kernel to analyze its impact on the network performance of the system.
// Return to interrupted code
The guest virtual and physical addresses of the Interrupt Descriptor Table (IDT) are left unchanged, but in 𝐸𝑃𝑇1 𝑡𝑜 (𝑛−1) the underlying host physical page points to interrupt sentries instead of the original handlers, as shown in Figure 12. We allocate a dedicated interrupt stack page and map it read-writeable in all compartments so that the interrupt frame is accessible after an EPT switch. Procedure 2
6.1
Compartment Switch Overhead
Compartment switch overhead is based on the cost of four operations: (1) a Virtualization Exception (#VE) that traps into the sentry, (2) a vmfunc instruction that switches the 9
Figure 12. Interrupt Descriptor Table Setup in Pomegranate
Figure 13. Interrupt Sentry Stack Manipulation
Figure 14. Sentry Stack Manipulation: Interrupt Return Path
10
Operation One-way Round-trip #UD (reference) 448 790 Virtualization Exception 938 1286 EPT Violation (reference) 990 1318 vmfunc 142 Full Sentry Transition 1782 3592 Interrupt Handling 642 2958 Table 1. Median Latency (cycles) of Operations in Pomegranate
Function igc_msix_ring napi_schedule_prep __napi_schedule igc_features_check igc_xmit_frame dma_map_page_attrs igc_poll dma_unmap_page_attrs napi_consume_skb __memcpy napi_alloc_skb eth_get_headlen skb_add_rx_frag eth_type_trans gro_receive_skb napi_complete_done
EPT, (3) the sentry logic that checks whether the transition is allowed, and (4) the interrupt delivery path that redirects interrupts to the correct compartment. We measure each operation separately by executing 10,000 iterations in a kernel loop with interrupts disabled and record cycle counts with rdtsc. Table 1 reports the median costs, along with the cost of an undefined opcode exception (#UD) and an EPT violation trapping into the hypervisor for comparison. The #VE is 490 cycles more expensive than #UD as a result of the EPT page-walk and Virtualization Exception information area write, but is still cheaper than a full EPT violation (990 cycles one-way). To measure the full sentry cost, we write a kernel module whose device write function calls another function test_mod_func 10,000 times. We then define two compartments: a test compartment 𝐶 1 consisting only of the test_mod_func, and the default compartment 𝐶 0 which consists of the rest of the kernel and can call test_mod_func. The “Full Sentry Transition” row in Table 1 shows the median one-way (call path) and round-trip (call and return path) cost incurred by the device write function, excluding rdtsc overhead and the execution time of test_mod_func itself. The “Interrupt Handling” row shows the one-way cost of redirecting an interrupt that arrives while the CPU is not in 𝐶 0 (Procedure 2), and the roundtrip cost of interrupt handling (Figure 11). To measure the costs of interrupt handling, we define a custom #UD handler in 𝐶 0 . 6.2
Direction kern → drv drv → kern drv → kern kern → drv kern → drv drv → kern kern → drv drv → kern drv → kern drv → kern drv → kern drv → kern drv → kern drv → kern drv → kern drv → kern
Data Path ISR ISR ISR TX submission TX submission TX submission NAPI poll entry TX cleanup TX cleanup RX RX RX (>256 B only) RX (>256 B only) RX RX Poll exit
Table 2. Compartment Switches in the igc Data Path the other runs the same Yocto Linux on bare metal. Since Pomegranate does not currently support heap object isolation, we identify the heap region of the Linux kernel and map it as read-writeable in both the igc and the default compartments. First, we analyze the igc data path to determine the number of cross-compartment switches. Then, we refine the compartment boundary and amortize the remaining crossings to reduce overhead. Finally, we evaluate the performance of the unrefined and the refined versions of the igc driver under the Pomegranate framework and compare them against the baseline performance of the driver without compartmentalization. We disable the driver’s periodic watchdog timer during benchmarking to eliminate infrequent timer-driven crossings that are irrelevant to data-path performance. 6.2.1 Kernel Analysis of the igc Data Path. We statically analyze the igc driver data path in Linux 6.8.1 to identify every function that crosses the compartment boundary. Table 2 lists all crossing points. Several functions that appear in the driver source are static inline wrappers defined in kernel headers and do not cross the boundary. We verify each wrapper to determine whether it ultimately invokes an exported symbol. For example, napi_gro_receive() is static inline but calls the exported gro_receive_skb(). Each call to an exported function and its return constitutes two compartment switches. The igc driver uses NAPI, so ISR and poll entry/exit costs are amortized across a batch of packets. Transmit (TX) submission incurs 6 switches per packet: igc_features_check (called by the kernel on every outgoing packet before igc_xmit_frame) and dma_map_page_attrs each contribute a call– return pair in addition to igc_xmit_frame itself. TX cleanup incurs 4 switches per packet. On the receive (RX) path, four exported functions cross the boundary per
Isolated NIC Driver
To evaluate the overhead of our framework, we partition the kernel into two compartments: one containing the Intel I225-V NIC driver (igc) and a default compartment, containing the remainder of the kernel. We choose the igc driver because it is the NIC present on our test platform and because NIC drivers produce a high frequency of cross-compartment calls on the data path. All code in drivers/net/ethernet/intel/igc/ constitutes the driver compartment, while the core kernel, including the networking stack, DMA subsystem, and memory allocators, constitutes the kernel compartment. All experiments in this subsection use two Supermicro X13SRN-H machines connected via Intel I225-V NICs (2.5 Gbps). One machine is designated as the test machine running Pomegranate-compartmetalized Yocto Linux while 11
packet: __memcpy, napi_alloc_skb, eth_type_trans, and gro_receive_skb. Packets larger than 256 bytes on the wire additionally invoke eth_get_headlen and skb_add_rx_frag. Including the amortized poll overhead (napi_complete_done), the per-packet totals are 10 switches for small RX packets and 14 for large ones (Table 3, Baseline column). When the CPU is in the driver compartment, an additional 2 switches are needed for interrupt delivery (Figure 11).
Out of the driver: The top-half interrupt handler performs only an MMIO write to the NIC’s interrupt throttle register and a call to schedule NAPI—trivial work wrapped in 6 handler gate crossings costing ∼10,776 cycles. We replace it with a kernel-side trampoline that performs both operations without entering the driver compartment, eliminating all 6 handler crossings. The interrupt delivery overhead (2 switches when the CPU is in the driver compartment; Figure 11) remains, as it is part of our framework’s interrupt handling mechanism. Similarly, the kernel calls igc_features_check on every outgoing packet to validate offload features—pure skb header inspection with no driverprivate state. We replace it with a kernel-side stub, eliminating 2 switches per TX packet. The security trade-off is that the kernel compartment gains write access to the NIC’s ITR register, which controls only interrupt coalescing timing, and read access to skb headers that it already constructed. Batch amortization: Three per-packet kernel calls remain after boundary refinement: napi_alloc_skb (RX allocation), napi_consume_skb (TX cleanup), and napi_gro_receive (RX delivery to the protocol stack). We replace each with a kernel-side batch helper. A preallocation helper allocates an entire poll budget’s worth of sk_buffs in a single crossing at the start of each NAPI poll; a second helper frees any unused remainder. For TX cleanup, we collect completed sk_buff pointers during the cleanup loop and pass them to a batch helper in one crossing, amortizing the per-packet cost from 2 switches to ∼0.02 switches (2 switches per 128 packets). For RX delivery, we defer napi_gro_receive calls, collecting completed sk_buffs during the receive loop and delivering the entire batch in a single crossing at the end of the poll—amortizing from 2 switches per packet to ∼0.03 switches per packet (2 switches per budget of 64 packets). The deferral is safe: the batch helpers perform the same operations as the individual calls, touch no descriptor ring state, and the driver does not access an sk_buff after handing it to the helpers. We also skip the RX batch allocation during TX-only polls by checking the first RX descriptor before crossing. In total, the amortized RX path incurs ∼0.06 switches per packet. Table 3 shows the compartment switch counts before and after applying boundary refinement and batch amortization.
6.2.2 Reducing Compartment Crossings. The baseline compartment boundary between the igc driver module and the rest of the kernel is determined by the module’s symbol table: every call to or from an exported kernel symbol crosses the boundary. Many of these crossings are unnecessary—the called function either performs trivial work that could execute in the caller compartment, or is a platform-specific no-op. Each call-return pair nonetheless incurs two compartment switches (∼3592 cycles round-trip, Table 1). We reduce crossings by applying two strategies. First, we refine the compartment boundary by relocating functions across it: small, stateless kernel functions are moved into the driver compartment, while driver entry points that perform only kernel-relevant work are replaced with kernelside stubs. Second, we amortize the remaining crossings by batching per-packet kernel calls into per-poll helpers that cross the boundary once per batch. It is possible to implement both refinement strategies by modifying the underlying EPT mappings to refer to replacement code in host memory. However, our prototype approach modifies the Linux source code directly, for simplicity. Boundary Refinement: We relocate functions across the compartment boundary in both directions. Into the driver: On x86 with direct DMA and no IOMMU, dma_map_page_attrs reduces to a single arithmetic operation that returns the physical address as the DMA address, and dma_unmap_page_attrs is a complete noop; we compile the mapping logic into the driver and remove the unmapping call, eliminating 2 switches per TX submission and 2 per TX cleanup. On the RX path, skb_add_rx_frag (shared-heap only), eth_type_trans (pure header parsing), and the kernel’s __memcpy are each moved into the driver compartment. eth_get_headlen depends on the flow dissector __skb_flow_dissect; we copy the dissector into the driver after removing unused BPF code paths, mapping one read-only kernel global into the driver EPT. These relocations eliminate the size-dependent RX crossing split: after refinement, the RX path incurs the same crossing count regardless of packet size. The driver’s TCB grows, but all relocated code is stateless packet processing or arithmetic on shared-heap data; the driver already has access to page structures through the shared heap, so the driver gains no new capability. 12
Path Frequency Baseline Reduced ISR per interrupt 6 0 TX submission per packet 6 2 TX cleanup per packet 4 ∼0.02 RX (≤256 B) per packet 10 ∼0.06 RX (>256 B) per packet 14 ∼0.06 Poll entry/exit per poll 2–4 2–4 Table 3. Compartment Switch Counts Before and After Crossing Reduction
2250
2500
Baseline Unrefined Refined
Throughput (Mbps)
Throughput (Mbps)
2500 2000 1750 1500 500 0
27.2
54.0
64
128
108
214
256
512
Payload Size (bytes)
494
1024
764
1472
Figure 15. Single-Flow TX Throughput
2000
Baseline Unrefined Refined
1500 1000 500 10 5 0
0.28
0.56
64
128
1.00
1.99
256
512
Payload Size (bytes)
3.95
1024
5.70
1472
Figure 16. Single-Flow RX Throughput
After boundary refinement and batch amortization, the only remaining per-packet crossing is igc_xmit_frame entry/return (2 switches per TX packet); all RX per-packet crossings are fully amortized into per-poll batches.
matches the line rate from 512 bytes onward. The RX throughput exceeds TX throughput at small packet sizes because batch amortization reduces the number of compartment switches in the reception path to ∼0.06 switches/packet (Table 3), while the transmission path retains two irreducible per-packet switches. The RX throughput dip at 256 bytes in the refined configuration corresponds to the IGC_RX_HDR_LEN threshold, where the driver switches from a linear copy to a page-fragment mode with flow dissection. The additional per-packet cost of this path is negligible in the baseline, but accounts for a ∼40% throughput reduction under compartmentalization at that packet size. Larger payloads amortize this cost.
6.2.3 Experiment 1: Single-Flow Throughput. To understand the compartmentalization overhead on the igc driver data path, we perform the netperf [17] UDP_STREAM test for 30 seconds at different message sizes. We test three configurations of the test machine: (1) a baseline kernel without compartmentalization running on Quest-V, (2) an unrefined compartmentalized kernel with all original boundary crossings, and (3) a refined compartmentalized kernel with the boundary refinement and batch amortization described in Section 6.2.2. For TX, the netperf client runs on the test machine and the server on an unmodified machine. For RX, the client runs on the unmodified machine and the test machine acts as the server. We disable interrupts in the igc compartment using the sentry to minimize compartment crossings in both compartmentalized configurations. We vary the netperf message size (-m) from 64 bytes to 1472 bytes (the Maximum Transmission Unit, MTU), spanning six sizes: 64, 128, 256, 512, 1024, and 1472. TX throughput: Figure 15 shows TX throughput for all three configurations. When the payload is 1472 bytes, the refined compartmentalized configuration reaches the line rate (∼2393 Mbps), matching the baseline, while the unrefined configuration only achieves 764 Mbps (almost 70% reduction). At 64 bytes, where per-packet overhead dominates, the unrefined configuration delivers only 27 Mbps (vs. 359 Mbps baseline, a 92% reduction). In comparison, the refined configuration reaches 108 Mbps (70% reduction). The remaining TX overhead is dominated by the two irreducible igc_xmit_frame switches per packet (Table 3). RX throughput: Figure 16 shows the RX throughput. The unrefined configuration delivers <6 Mbps at all packet sizes, due to the large number of compartment switches in the typical RX path. The refined configuration reaches 304 Mbps at 64 bytes (84% of the 360 Mbps baseline) and
6.2.4 Experiment 2: Multi-Compartment Evaluation. Next, we measure how throughput changes with the number of active compartments, by setting up a multi-threaded TX test. Each igc compartment shares the same driver code and heap; only the per-compartment stack and execution context, in this case the thread ID, differ. This experiment evaluates the cost of the sentry policy lookup and any TLB or cache pressure from maintaining multiple EPT contexts. We use the refined compartmentalized configuration (Section 6.2.2), which incurs 2 switches per TX submission. We launch 𝑁 concurrent netperf UDP_STREAM instances on the test machine, each assigned to a distinct compartment, with a per-packet payload size of 1472 bytes. All instances send to the same port of the same unmodified receiver. A baseline system without compartmentalization is run with the same number of 𝑁 threads, for comparison. We only evaluate packet transmission as it runs in a process context with each sendto carrying the calling thread’s compartment identity. Hence, 𝑁 concurrent senders present 𝑁 distinct identities to the sentry. Packet reception, in contrast, runs in NAPI softirq context—the driver’s igc_poll dequeues all packets from a shared receive ring regardless of destination flow, and demultiplexing by 13
𝑁 2–32
Baseline (Mbps) 2392.77–2395.78
Compartmentalized (Mbps) 2392.67–2394.55
surface. Any pointers passed across compartment boundaries should preferably be sized such that their values fit within registers exchanged between caller and callee. Sub-Page Isolation: EPTs enforce permissions at page granularity. Such permissions fail when functions or static data from different compartments share a physical page. We currently use compiler directives applied to guest kernel source code to page-align compartmentalized objects (Section 4). Alternatively, source code annotations are unnecessary if the hypervisor allocates a separate copy of each shared page per compartment, containing only that compartment’s code and data at their original offsets. The remaining addresses are filled with trapping values (for code) or zeroes (for data). This allows sub-page isolation without guest kernel modification. Compartment Switching Mechanism: Pomegranate uses Virtualization Exceptions to facilitate compartment switches without source code or binary modification. However, source code annotations or binary rewriting has the potential to reduce the overhead of compartment switching. Implicit vs. Explicit Gate Calls: Pomegranate uses a sentry to implement gate calls that mediate transitions between compartments. We consider this approach an implicit gate calling mechanism because it does not require source code or binary modification of the guest. An explicit gate call would require guest code annotations to jump to a gate manager that validates compartment switches. One may wonder why it makes sense to support implicit gate calls when the establishment of compartments and policies requires source code analysis. We believe implicit gate calls are beneficial when the policy generation and enforcement are handled by separate entities. Moreover, policy updates are easily handled with an implicit gate calling mechanism, while they would require potential source code changes with an explicit approach. Heap Object Isolation: All kernel code shares a common heap. As stated above, EPT permissions apply only at page granularity, so heap objects from different compartments that share a page cannot easily be isolated using only EPTs. TME-MK [15] provides sub-page read isolation by assigning a different key ID per compartment in each EPT. The encryption engine operates at cache-line granularity, so accesses with an incorrect key ID return garbled data. TDX [16] adds write protection using Message Authentication Code (MAC) verification. However, TME-MK cannot protect objects smaller than a cache line (64 bytes), and TDX’s behavior on a MAC mismatch varies across platforms [38]. An alternative approach that does not require such esoteric hardware features is to implement percompartment heap-memory allocation. Each compartment is then assigned a dedicated heap region unmapped from other compartments’ EPTs. All original dynamic memory allocation requests are then redirected to compartmentspecific allocators using the same interposition techniques
Table 4. Multi-Compartment TX Aggregate Throughput
port occurs in the kernel’s transport layer above the compartment boundary. Additional compartments do not affect the RX data path. Table 4 shows that the aggregate throughput remains at line rate (∼2393 Mbps) for all tested values of 𝑁 , with less than 2 Mbps difference between the baseline and the compartmentalized configuration. All threads achieve approximately the same throughput in all cases. Increasing the number of igc compartments from 2 to 32 introduces no measurable aggregate throughput difference, consistent with no TLB or cache contention from maintaining multiple EPT contexts at this packet size.
7
Discussion
Resilience to Attacks: There is nothing to stop an attacker from inserting arbitrary vmfunc calls into an instruction sequence. However, as they do not divert control flow by themselves, the instruction immediately after a vmfunc will trigger an EPT violation, unless it is mapped into the target compartment. Control-Flow Enforcement Technology (CET) prevents an attacker from jumping to an arbitary sentry address, to avoid the can_call check, or corrupt the return address in the callee stack. With CET, legitimate indirect calls or jumps are followed by the endbr instruction. The technology also uses shadow stacks to verify valid return addresses. Compartment-Aware Kernel Drivers: While drivers are a prominent source of vulnerabilities in Linux [7], our analysis shows that the design of the igc NIC driver does not easily lend itself to compartmentalization, due to significant interaction with core kernel code. Other complications relate to the handling of RX and TX cleanup using the same softirq handler, making it difficult to (a) provide differentiated quality-of-service [34], and (b) isolate the handling of received data for different users. Our approach to refine compartment boundaries requires altering the logic within separate compartments using interposed code that is mapped by EPTs to the original guest physical addresses. Similar interposition techniques are able to improve service quality and isolation between users. Policy Generation: The granularity of compartments needs to be determined by carefully considering the tradeoff between security and cost of enforcement. There is little utility in splitting each function into its own compartment if the resultant overhead is prohibitive. Another factor that should be considered when determing compartment size is the amount of global data shared between compartments. Shared data should be minimized to ensure a small attack 14
described earlier. This avoids kernel source code or binary modifications. Multi-core Support: We enabled only one core to avoid interference from impacting the experimental results, but Pomegranate is runnable on a multi-core system as long as the tasks in non-default compartments remain on the cores where they began execution. The VMCS tracks the current EPT on a per-core basis. Hence, to support task migration, the hypervisor needs to create the same EPTP list on all cores. In addition to calls, returns and interrupt returns, the sentry should also handle #VEs that occur when a task migrates to a core where the current EPT does not map its data or code.
8
achieve the same goal. The sentry logic maintains a centralized access control matrix that ensures all compartment crossings adhere to legitimate execution paths. Microsoft’s Virtualization-Based Security [26] uses Hyper-V to protect sensitive Windows components but does not support mutual distrust between arbitrary kernel components. xMP [32] provides memory protection primitives within a VM, allowing guests to isolate sensitive data in disjoint domains using EPT subspaces. While xMP also uses Virtualization Exceptions and vmfunc, it targets individual data structures (e.g., page tables, credentials) rather than general compartmentalization, and does not provide a sentry mechanism for compartment transitions with access control. Other Isolation Mechanisms: ERIM [39] uses Intel Memory Protection Keys (MPK) for in-process isolation on x86. BULKHEAD [12] applies Protection Keys for Supervisor (PKS) access to kernel compartmentalization, but requires binary rewriting to neutralize wrpkru and other instructions that could bypass protection keys. Connor et al. [8] and Voulimeneas et al. [40] use MPK for user-space sandboxing. Schrammel et al. [35] use memory tagging with cryptographic integrity on commodity x86. PKS and MPK operate at the kernel privilege level, so a compromised kernel can execute privileged instructions to bypass protection enforcement. All of these approaches therefore require binary analysis or rewriting to neutralize such instructions. In Pomegranate, enforcement is in the hypervisor: no guest instruction is able to modify EPT permissions regardless of privilege level, so no instruction scanning or binary rewriting is needed. EKC [46] provides a portable kernel compartment that enforces isolation but, like other methods that do not use a higher privilege level, requires code analysis to eliminate instructions that subvert the protection mechanism. In other work, Mondrix [44] applies Mondriaan memory protection to the Linux kernel. SVA [10] defines a virtual instruction set that enforces memory safety and control-flow integrity for system code.
Related Work
Lefeuvre et al. [19] classify compartmentalization approaches into sandbox, safebox, and mutual- distrust models. Lim et al. [23] survey kernel compartmentalization techniques along similar lines. Our work follows the mutual- distrust model, where all kernel components are treated as equally untrusted. Pomegranate differs from prior kernel compartmentalization systems in that compartment boundaries are enforced from outside the guest, at a privilege level no guest code can subvert. The guest kernel executes unmodified and is unaware of compartmentalization; boundaries are defined by the hypervisor’s EPT configuration and a generic sentry that handles all compartment transitions without per-compartment glue code. Roessler et al. [33] present 𝜇SCOPE, a methodology for analyzing least-privilege compartmentalization in large codebases. Our work is compatible with such a mechanism that analyzes a monolithic kernel to determine the right compartment boundaries. Virtualization-based Approaches: EPTI [13] uses EPT switching as an alternative to kernel page table isolation for Meltdown mitigation but does not provide intra-kernel compartmentalization. LXDs [28] isolate kernel subsystems into lightweight execution domains. Per-module glue code is generated from an interface definition language (IDL) to handle data marshalling across compartment boundaries. LVD [29] builds upon LXDs by using vmfunc to make compartment transitions more efficient. KSplit [14] automates the generation of this IDL-based marshalling code for device drivers. All three systems require custom glue code compiled into the kernel for each compartment boundary, and compartmentalization is limited to kernel drivers. In contrast, Pomegranate uses a combination of Virtualization Exceptions and vmfunc to enable compartment switches without source code or binary modification. These features are used to implement a sentry mechanism that interposes on function calls that cross compartment boundaries. Pomegranate uses HLAT to ensure that the guest does not modify its own page tables in a way that violates the system security policy, avoiding the need for glue code to
9
Conclusion and Future Work
This work presents Pomegranate, a framework that uses hardware-assisted virtualization to compartmentalize a monolithic kernel. By enforcing isolation through EPTs, Pomegranate is able to compartmentalize a monolithic guest without modifying its source code or binary image. Pomegranate uses a generic sentry mechanism that handles all compartment transitions in the guest using Virtualization Exceptions and EPTs. Sentry execution avoid traps into a hypervisor-based gate manager, and allows direct access to guest state. This avoids expensive guest page table walks to perform compartment switching, if it were performed within the hypervisor. However, Virtualization Exceptions are shown to incur more overhead than other 15
exceptions, as the CPU must walk the EPT table and write to a special information area the reason for the exception. This is an area where future hardware optimizations would be beneficial to virtualization-based compartmentalization approaches such as Pomegranate. Pomegranate is evaluated on a compartmentalized Linux network stack using the igc NIC driver. With up to 32 separate execution contexts, results show that compartmentalization incurs negligible overhead for MTU-sized packets. By analyzing compartment crossings and redrawing boundaries, Pomegranate is able to maintain the throughput of a baseline monolithic Linux system. Future work will extend Pomegranate to enforce protection against return- and jump-oriented attacks using Control-flow Enforcement Technology (CET) [9]. Other research directions include more fine-grained isolation of dynamically allocated objects, boot-time compartment initialization, and dynamic compartment creation.
[9] Intel Corporation. 2025. Combined Volume Set of Intel® 64 and IA-32 Architectures Software Developer’s Manuals. Intel Corporation. https://www.intel.com/content/www/us/en/developer/articles/ technical/intel-sdm.html [10] John Criswell, Andrew Lenharth, Dinakar Dhurjati, and Vikram Adve. 2007. Secure Virtual Architecture: A Safe Execution Environment for Commodity Operating Systems. In Proceedings of Twenty-First ACM SIGOPS Symposium on Operating Systems Principles (Stevenson, Washington, USA) (SOSP ’07). Association for Computing Machinery, New York, NY, USA, 351–366. doi:10.1145/1294261.1294295 [11] Úlfar Erlingsson, Martín Abadi, Michael Vrable, Mihai Budiu, and George C. Necula. 2006. XFI: Software Guards for System Address Spaces. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (Seattle, Washington) (OSDI ’06). USENIX Association, USA, 75–88. https://www.usenix.org/legacy/event/osdi06/ tech/full_papers/erlingsson/erlingsson.pdf [12] Yinggang Guo, Zicheng Wang, Weiheng Bai, Qingkai Zeng, and Kangjie Lu. 2025. BULKHEAD: Secure, Scalable, and Efficient Kernel Compartmentalization with PKS. In 32nd Annual Network and Distributed System Security Symposium, (NDSS). The Internet Society. doi:10.14722/ndss.2025.230328 [13] Zhichao Hua, Dong Du, Yubin Xia, Haibo Chen, and Binyu Zang. 2018. EPTI: Efficient Defence against Meltdown Attack for Unpatched VMs. In Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, Haryadi S. Gunawi and Benjamin C. Reed (Eds.). USENIX Association, Berkeley, CA, USA, 255–266. https://www.usenix.org/system/files/conference/ atc18/atc18-hua.pdf [14] Yongzhe Huang, Vikram Narayanan, David Detweiler, Kaiming Huang, Gang Tan, Trent Jaeger, and Anton Burtsev. 2022. KSplit: Automating Device Driver Isolation. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 613–631. [15] Intel Corporation. 2023. Intel® Architecture Memory Encryption Technologies Specification. Specification Revision 1.4. Intel Corporation. https://cdrdv2-public.intel.com/679154/multi-key-totalmemory-encryption-spec-1.4.pdf [16] Intel Corporation. 2023. Intel® Trust Domain Extensions (Intel® TDX) Module Base Architecture Specification. Specification. Intel Corporation. https://www.intel.com/content/www/us/en/contentdetails/853294/intel-trust-domain-extensions-intel-tdx-modulebase-architecture-specification.html [17] Rick Jones. 2021. Netperf: A Network Performance Benchmark. https: //github.com/HewlettPackard/netperf Hewlett Packard Enterprise. [18] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. 2009. seL4: Formal Verification of an OS Kernel. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP ’09). ACM, Big Sky, MT, USA, 207–220. doi:10.1145/1629575. 1629596 [19] Hugo Lefeuvre, Nathan Dautenhahn, David Chisnall, and Pierre Olivier. 2025. SoK: Software Compartmentalization. In 2025 IEEE Symposium on Security and Privacy (SP). IEEE, USA, 3107–3126. doi:10. 1109/SP61157.2025.00075 [20] Henry M. Levy. 1984. Capability-Based Computer Systems. Butterworth-Heinemann, USA. https://homes.cs.washington.edu/ ~levy/capabook/ [21] Ye Li, Richard West, and Eric Missimer. 2014. A Virtualized Separation Kernel for Mixed Criticality Systems. In Proceedings of the 10th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (Salt Lake City, Utah, USA) (VEE ’14). Association for Computing Machinery, New York, NY, USA, 201–212. doi:10.1145/2576195.2576206
References [1] Arm. 2019. Armv8.5-A Memory Tagging Extension: White Paper. White paper. Arm Ltd. https://developer.arm.com//media/Arm%20Developer%20Community/PDF/Arm_Memory_ Tagging_Extension_Whitepaper.pdf [2] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the Art of Virtualization. SIGOPS Oper. Syst. Rev. 37, 5 (Oct. 2003), 164–177. doi:10.1145/1165389.945462 [3] B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. 1995. Extensibility Safety and Performance in the SPIN Operating System. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, USA) (SOSP ’95). Association for Computing Machinery, New York, NY, USA, 267–283. doi:10.1145/224056.224077 [4] William Bugden and Ayman Alahmar. 2022. Rust: The Programming Language for Safety and Performance. arXiv:2206.05503 [cs.PL] [5] Burroughs Corporation. 1961. The Descriptor – A Definition of the B 5000 Information Processing System. Technical Manual 5000-20002-P. Burroughs Corporation. http://www.bitsavers.org/pdf/burroughs/ LargeSystems/B5000_5500_5700/5000-20002-P_The_Descriptor__A_Definition_of_the_B_5000_Information_Processing_System_ 196102.pdf Archived at Computer History Museum, bitsavers.org. [6] Miguel Castro, Manuel Costa, Jean-Philippe Martin, Marcus Peinado, Periklis Akritidis, Austin Donnelly, Paul Barham, and Richard Black. 2009. Fast Byte-Granularity Software Fault Isolation. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (Big Sky, Montana, USA) (SOSP ’09). Association for Computing Machinery, New York, NY, USA, 45–58. doi:10.1145/1629575.1629581 [7] Haogang Chen, Yandong Mao, Xi Wang, Dong Zhou, Nickolai Zeldovich, and M. Frans Kaashoek. 2011. Linux Kernel Vulnerabilities: State-Of-The-Art Defenses and Open Problems. In Proceedings of the Second Asia-Pacific Workshop on Systems (Shanghai, China) (APSys ’11). Association for Computing Machinery, New York, NY, USA, Article 5, 5 pages. doi:10.1145/2103799.2103805 [8] Emma Connor, Tyler McDaniel, Jared M. Smith, and Max Schuchard. 2020. PKU Pitfalls: Attacks on PKU-based Memory Isolation Systems. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 1409–1426. https://www.usenix.org/ conference/usenixsecurity20/presentation/connor 16
[22] J. Liedtke. 1995. On Micro-kernel Construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, USA) (SOSP ’95). Association for Computing Machinery, New York, NY, USA, 237–250. doi:10.1145/224056.224075 [23] Soo Yee Lim, Sidhartha Agrawal, Xueyuan Han, David Eyers, Dan O’Keeffe, and Thomas Pasquier. 2024. Securing Monolithic Kernels using Compartmentalization. arXiv:2404.08716 [cs.CR] [24] Yandong Mao, Haogang Chen, Dong Zhou, Xi Wang, Nickolai Zeldovich, and M. Frans Kaashoek. 2011. Software Fault Isolation with API Integrity and Multi-Principal Modules. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (Cascais, Portugal) (SOSP ’11). Association for Computing Machinery, New York, NY, USA, 115–128. doi:10.1145/2043556.2043568 [25] Derrick McKee, Yianni Giannaris, Carolina Ortega, Mathias Payer, Hamed Okhravi, and Nathan Burow. 2022. Preventing Kernel Hacks with HAKCs. In 32nd Annual Network and Distributed System Security Symposium, (NDSS). doi:10.14722/ndss.2022.24026 [26] Microsoft. [n. d.]. Virtualization-based Security (VBS). https: //learn.microsoft.com/en-us/windows-hardware/design/deviceexperiences/oem-vbs [27] Glenford J. Myers and David L. Budde. 1982. The Architecture of the Intel 432. ACM SIGARCH Computer Architecture News 10, 2 (1982), 10–16. doi:10.1145/1067649.801717 [28] Vikram Narayanan, Abhiram Balasubramanian, Charlie Jacobsen, Sarah Spall, Scott Bauer, Michael Quigley, Aftab Hussain, Abdullah Younis, Junjie Shen, Moinak Bhattacharyya, and Anton Burtsev. 2019. LXDs: Towards Isolation of Kernel Subsystems. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, 269–284. [29] Vikram Narayanan, Yongzhe Huang, Gang Tan, Trent Jaeger, and Anton Burtsev. 2020. Lightweight Kernel Isolation with Virtualization and VM Functions. In Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (Lausanne, Switzerland) (VEE ’20). Association for Computing Machinery, New York, NY, USA, 157–171. doi:10.1145/3381052.3381328 [30] R. M. Needham and R. D.H. Walker. 1977. The Cambridge CAP Computer and its Protection System. SIGOPS Oper. Syst. Rev. 11, 5 (Nov. 1977), 1–10. doi:10.1145/1067625.806541 [31] Gerald J. Popek and Robert P. Goldberg. 1974. Formal Requirements for Virtualizable Third Generation Architectures . Commun. ACM 17, 7 (July 1974), 412–421. doi:10.1145/361011.361073 [32] Sergej Proskurin, Marius Momeu, Seyedhamed Ghavamnia, Vasileios P. Kemerlis, and Michalis Polychronakis. 2020. xMP: Selective Memory Protection for Kernel and User Space. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 563–577. doi:10.1109/SP40000.2020.00041 [33] Nick Roessler, Lucas Atayde, Imani Palmer, Derrick McKee, Jai Pandey, Vasileios P. Kemerlis, Mathias Payer, Adam Bates, Jonathan M. Smith, Andre DeHon, and Nathan Dautenhahn. 2021. 𝜇SCOPE: A Methodology for Analyzing Least-Privilege Compartmentalization in Large Software Artifacts. In Proceedings of the 24th International Symposium on Research in Attacks, Intrusions and Defenses (San Sebastian, Spain) (RAID ’21). Association for Computing Machinery, New York, NY, USA, 296–311. doi:10.1145/3471621.3471839 [34] Zhiyuan Ruan, Anton Njavro, and Richard West. 2024. USB Interrupt Differentiated Service for Bandwidth and Delay-Constrained Input/Output. In 2024 IEEE 30th Real-Time and Embedded Technology and Applications Symposium (RTAS). 42–54. doi:10.1109/RTAS61025. 2024.00012 [35] David Schrammel, Samuel Weiser, Richard Sadek, and Stefan Mangard. 2022. Jenny: Securing Syscalls for PKU-based Memory Isolation Systems. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 936–952. https://www.usenix. org/system/files/sec22-schrammel.pdf
[36] Udo Steinberg and Bernhard Kauer. 2010. NOVA: A Microhypervisorbased Secure Virtualization Architecture. In Proceedings of the 5th European Conference on Computer Systems (Paris, France) (EuroSys ’10). Association for Computing Machinery, New York, NY, USA, 209–222. doi:10.1145/1755913.1755935 [37] Satoshi Tanda. 2023. Intel VT-rp - Part 1. remapping attack and HLAT. https://tandasat.github.io/blog/2023/07/05/intel-vt-rp-part-1.html [38] Martin Unterguggenberger, Lukas Lamster, David Schrammel, Martin Schwarzl, and Stefan Mangard. 2025. TME-Box: Scalable In-Process Isolation through Intel TME-MK Memory Encryption. In 32nd Annual Network and Distributed System Security Symposium, (NDSS). The Internet Society. doi:10.14722/ndss.2025.240277 [39] Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, and Deepak Garg. 2019. ERIM: Secure, Efficient In-process Isolation with Protection Keys (MPK). In Proceedings of the 28th USENIX Conference on Security Symposium (Santa Clara, CA, USA) (SEC’19). USENIX Association, USA, 1221–1238. https: //www.usenix.org/system/files/sec19-vahldiek-oberwagner_0.pdf [40] Alexios Voulimeneas, Jonas Vinck, Ruben Mechelinck, and Stijn Volckaert. 2022. You Shall Not (by)Pass! Practical, Secure, and Fast PKU-based Sandboxing. In Proceedings of the Seventeenth European Conference on Computer Systems (Rennes, France) (EuroSys ’22). Association for Computing Machinery, New York, NY, USA, 266–282. doi:10.1145/3492321.3519560 [41] Robert Wahbe, Steven Lucco, Thomas E. Anderson, and Susan L. Graham. 1993. Efficient Software-based Fault Isolation. In Proceedings of the Fourteenth ACM Symposium on Operating Systems Principles (Asheville, North Carolina, USA) (SOSP ’93). Association for Computing Machinery, New York, NY, USA, 203–216. doi:10.1145/168619. 168635 [42] Robert N. M. Watson, Simon W. Moore, Peter Sewell, and Peter G. Neumann. 2019. An Introduction to CHERI. Technical Report UCAMCL-TR-941. University of Cambridge, Computer Laboratory. https: //www.cl.cam.ac.uk/techreports/UCAM-CL-TR-941.pdf [43] Richard West, Ye Li, Eric Missimer, and Matthew Danish. 2016. A Virtualized Separation Kernel for Mixed-Criticality Systems. ACM Trans. Comput. Syst. 34, 3, Article 8 (June 2016), 41 pages. doi:10.1145/ 2935748 [44] Emmett Witchel, Junghwan Rhee, and Krste Asanović. 2005. Mondrix: Memory Isolation for Linux using Mondriaan Memory Protection. In Proceedings of the Twentieth ACM Symposium on Operating Systems Principles (Brighton, United Kingdom) (SOSP ’05). Association for Computing Machinery, New York, NY, USA, 31–44. doi:10. 1145/1095810.1095814 [45] W. Wulf, E. Cohen, W. Corwin, A. Jones, R. Levin, C. Pierson, and F. Pollack. 1974. HYDRA: The Kernel of a Multiprocessor Operating System. Commun. ACM 17, 6 (June 1974), 337–345. doi:10.1145/355616. 364017 [46] Jiaqin Yan, Qiujiang Chen, Shuai Zhou, Yuke Peng, Guoxing Chen, and Yinqian Zhang. 2025. EKC: A Portable and Extensible Kernel Compartment for De-Privileging Commodity OS. In 34th USENIX Security Symposium (USENIX Security 25). 7487–7506. https://www.usenix. org/system/files/usenixsecurity25-yan-jiaqin.pdf
17