Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap Between C Retrofits and Native Safety

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap Between C Retrofits and Native Safety Andrew Laramore

Joseph Spracklen

Murtuza Jadliwala

andrew.laramore2@my.utsa.edu The University of Texas at San Antonio San Antonio, Texas, USA

joe.spracklen@my.utsa.edu The University of Texas at San Antonio San Antonio, Texas, USA

Murtuza.Jadliwala@utsa.edu The University of Texas at San Antonio San Antonio, Texas, USA

arXiv:2608.15486v1 [cs.PL] 16 Aug 2026

Abstract While decades of research have produced numerous retrofitted memory-safety protections for C, these mechanisms are almost exclusively evaluated in isolation, targeting specific vulnerability classes. This siloed evaluation paradigm leaves practitioners without a clear understanding of the cumulative performance costs, interoperability conflicts, and protection gaps that arise when layering defenses to achieve comprehensive safety. This paper presents a new evaluation paradigm that benchmarks natively memory-safe languages like Rust and Go against compounded C retrofits. Using standardized cross-language tasks, we evaluate the performance and protection tradeoffs of state-of-the-art mechanisms when deployed in combination. Our results demonstrate that layered C defenses incur compounding and workload-dependent performance penalties, can suffer from fundamental architectural incompatibilities, and fall short of the protection scope provided by native memory-safe languages. These findings expose a critical engineering decision gap where the true cost of backporting safety to C remains hidden from practitioners. We argue for a fundamental shift in memory-safety research: moving away from isolated evaluation toward holistic, comparative frameworks that inform the high-stakes choice between retrofitting legacy codebases and migrating to modern, safe languages.

CCS Concepts • Security and privacy → Software security engineering; • Software and its engineering → General programming languages; Software performance; Compilers; Memory management.

Keywords Memory Safety, Performance Analysis, Runtime Protection Mechanisms, C Language, Cross-language Performance Comparison

1

Introduction

C and C++ remain critically important in modern software infrastructure, and both languages consistently rank among the top ten most popular on GitHub [43]. These languages underpin foundational systems such as the Linux and Windows NT kernels, Google Chrome, ffmpeg and countless device drivers that are actively maintained and developed today. Although these technologies drive contemporary computing, they remain vulnerable to persistent memory safety flaws inherent to C and C++. The 2024 Common Weakness Enumeration (CWE) Top 25 explicitly highlights this enduring challenge, listing "Out-of-bounds read" (CWE-125) as the sixth most critical vulnerability and "Use-after-free" (CWE-416) as the eighth [32].

This prevalence underscores a fundamental tension: despite decades of engineering effort, memory-unsafe languages continue to dominate performance-critical domains while contributing significantly to security breaches. Traditional mitigation techniques, such as Address Space Layout Randomization (ASLR) [42], Control Flow Integrity (CFI) [11], and compiler-based checks such as -fstack-protector [18] address specific attack vectors, but fail to provide holistic memory safety. Crucially, these protections operate in isolation, creating a fragmented defense landscape where adversaries can exploit gaps between disjointed mechanisms. Concurrently, memory-safe languages such as Rust and Go have demonstrated that eliminating entire vulnerability classes (e.g., dangling pointers and buffer overflows) is achievable without sacrificing performance. However, wholesale migration from C/C++ ecosystems is often impractical due to legacy codebases, toolchain dependencies, and domain-specific performance constraints. This raises the question: Can synergistic integration of state-of-the-art memory protections for C achieve memory safety guarantees comparable to native memory-safe languages, while preserving compatibility and performance? We investigate whether a unified framework that combines these techniques can bridge the safety gap between C and memory-safe alternatives such as Rust and Go. Evaluating C/C++ memory safety retrofits against baseline C alone creates a false sense of progress. Without benchmarking combinatorial defense stacks against native memory-safe languages’ inherent safety guarantees and performance profile, practitioners cannot determine whether retrofitting legacy code is more cost-effective than strategic migration. Native memory-safe languages, particularly Rust, establish a practical baseline for acceptable security guarantees and performance overhead, often delivering comprehensive memory safety with negligible measurable slowdown[39]. For retrofit C defenses to be a viable alternative, they must therefore demonstrate comparable protection breadth and predictable performance costs, rather than merely outperforming unprotected C. Absent such a comparison, claims of progress in C/C++ memory safety risk optimizing incomplete defenses whose cumulative cost and coverage remain poorly understood. Taken together, these observations expose a fundamental security gap: practitioners lack empirical guidance on whether layered C/C++ defenses can serve as a practical alternative to migration to memory-safe languages. Addressing this gap requires a systematic evaluation of protection coverage and performance costs across languages. Given that motivation, this research addresses three critical questions:

Laramore, Spracklen, and Jadliwala

(1) Can a combination of C/C++ memory safety mechanisms achieve Rust-like coverage without prohibitive overhead? (2) Does the composition of these mechanisms introduce safety regressions relative to the benchmarks established in their respective original papers? (3) What performance penalties arise from layered memory safety defenses versus native safe languages? Through a thorough evaluation of runtime performance and stability using the Checked C compiler[31], HWASAN compiler-based sanitizer [34], and the memory allocators MarkUs [3] and FFMalloc [50], both individually and in combination with established benchmarks, we reveal the following key findings: (1) Safety Completeness Gap: Even when combined, stateof-the-art (SOTA) retrofits, specifically Checked C paired with security-focused allocators like MarkUs or FFMalloc, fail to match Rust’s formal completeness, covering 5 out of 6 memory safety categories. WHile these 5 categories account for 99.8% of CVEs in the evaluated CWEs, the remaining uncovered class highlights a fundamental gap between retrofitting and native memory safety. (2) Performance Superiority: Rust consistently outperforms optimized C retrofits in 7 out of 9 benchmarks. While retrofitting aims to bridge the security gap, we observed significant performance penalties ranging from 1.5x to 4.5x slowdowns compared to Rust, demonstrating that native memory safety is often substantially more efficient than cumulative C protection layers. (3) Interoperability and Maintenance Failures: We identified critical interoperability failures that undermine the practical viability of C retrofits. For example, combining HWASAN with Checked C triggered immediate runtime crashes due to conflicting memory handling. These failures reduce the breadth of protection and significantly increase maintenance overhead, making such combinations difficult to deploy in production environments.

2

Background

The history of memory errors is nearly as long as the history of computing itself. A 1972 USAF-funded security analysis identified ‘Scavenging’ (uninitialized memory exploitation) and ‘Implied Sharing’ (memory isolation failures) as active attack vectors, with technical descriptions aligning with modern concerns like use-after-free and cross-process memory leaks. The document noted the performance overhead of countermeasures, revealing the early tension between efficiency and memory integrity [4]. The rise of C and C++, with arbitrary pointer arithmetic, exacerbated memory safety errors. The 1988 Morris worm became the first large-scale memory-safety incident, exploiting a buffer overflow and resulting in a conviction [15]. Initial mitigations were operating system features: the non-executable stack was proposed for Linux in 1997, followed by canaries in 1999 and ASLR in 2001 [49]. The C and C++ languages also implemented minor changes, such as replacing functions such as gets() and strcpy() with bounds-checked alternatives such as fgets(), gets_s() or strlcpy(). C++ introduced smart pointers for automatic deallocation to combat use-after-free errors. Although

these changes help, they are insufficient for making C/C++ memory safe, prompting research into more robust measures.

2.1

The Addition of New Systems-Level Languages

C has long dominated systems programming, underpinning foundational infrastructure including the Linux [9] and Windows NT kernels [30] as well as widely deployed webservers such as Apache httpd and Nginx[19]. This dominance reflects C’s unmatched performance and control, but also entrenches its well-known memory safety risks. Beginning in 2013, several languages emerged explicitly to challenge C/C++ in systems-level domains, including Go, Rust, D, and Nim each aiming to provide memory safety guarantees without sacrificing performance [10]. Among these, Rust has seen the most significant adoption. Released in 2015, Rust 1.0 introduced a novel ownership and borrowing model that enforces memory safety at compile time while preserving low-level control comparable to C [47].Since it’s release, Rust has moved beyond experimental use and into production systems contexts. It has been adopted for operating systems components and device drivers [17], and has enabled the development of fully memory-safe software such as the Redox operating system [38]. This growing deployment signals a broader shift in systems engineering: rather than retrofitting safety onto memory-unsafe languages, developers are increasingly willing to adopt languages that enforce memory safety by construction.

2.2

Rust Memory Safety Mechanisms

Rust enforces memory safety through a statically checked ownership and borrowing model that prevents spatial and temporal memory erros at compile time. The borrowing method provides temporary access via immutable (&T) or exclusive mutable (&mut T) references. The borrow checker validates reference lifetimes, implementing an affine type system that prevents use-after-free, double-free, and data races without runtime overhead. Safety guarantees exclude logical errors and integer overflows in release builds. Unsafe blocks enable raw pointer operations, FFI, and inline assembly through programmer-verified invariants, necessary for systems programming while maintaining localized safety boundaries. Rust’s ownership-based memory safety design establishes it as a practical baseline for memory safety, demonstrating that guaranteed protection can be achieved with minimal performance cost. Accordingly, Rust provides a natural point of comparison for evaluating whether layered C/C++ retrofits can deliver similar safety and performance.

2.3

Go Memory Safety Mechanisms

Go ensures memory safety via automatic garbage collection [20], strong static typing, runtime checks, and controlled pointers [21]. Its concurrent mark-sweep garbage collector tracks object references to reclaim unused memory, preventing leaks and use-afterfree errors. Compile-time type enforcement eliminates memory corruption bugs by catching type mismatches early. Runtime checks trigger panics for unsafe operations like out-of-bounds array access or null pointer dereferencing. Although the unsafe package permits low-level pointer manipulation, it explicitly warns against bypassing safety guarantees. These mechanisms create a secure yet efficient memory model suitable for systems programming,

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

reflecting Go’s philosophy of balancing simplicity with performance. Together, these mechanisms position Go as an alternative memory-safe baseline, enabling direct comparison between garbage-collected safety guarantees and the performance and composability of retrofitted C/C++ defenses evaluated in this work.

2.4

Programming Language Performance

Cross-language performance comparison is notoriously difficult, and most existing evaluations have been informal, ad-hoc, and largely disconnected from security considerations. The Computer Programming Language Benchmarks Game [46] provides one of the most widely referenced comparisons by measuring equivalent tasks in languages, consistently showing that C, C++, and Rust have closely matched performance profiles. Independent practitioner studies, such as Niklas Heer’s /dev/night analyses, reach similar conclusions, reporting only marginal performance differences between Rust and C across diverse workloads [25]. Although these results suggest that memory safety need not come at a substantial performance cost, they stop short of evaluating how retrofitted safety mechanisms in C compare to native safety guarantees in modern languages.

3

Problem Definition

To compare state-of-the-art memory safety in C with that of natively memory-safe languages, we first characterize the classes of memory safety vulnerabilities and the extent to which each language addresses them. Typically, C++ memory safety protections need to be explicitly enabled using a blacklist approach towards memory safety, but this requires developers to read each available protection mechanism and explicitly enable that mechanism one by one ("blacklisting" that memory safety bug). Forgetting or choosing not to explicitly enable protections available through standard tooling often leaves vulnerabilities that would otherwise have been avoided. Native memory safe languages operate with a whitelisting approach to memory safety bugs where the compiler assumes the program must be memory safe and requires the developer to explicitly disable protections.

3.1

Adding Safety to C

Microsoft Research introduced Checked C in 2015, which enhances the C programming language with static and dynamic verification mechanisms to identify and prevent frequent coding mistakes, including buffer overflows and memory access beyond allocated boundaries [31]. Checked C is meant to be a set of checks at both the compile time and the runtime that enable spatial safety for the C codebases [44] to protect against errors such as buffer overflows. It works on both regular C codebases and those that compile into LLVM [23], and is designed to be added to the C code incrementally over time to increase adoption. Despite this work, Checked C does not protect against all safety concerns. Notably, Microsoft states that Checked C is not designed to handle use-after-free errors [31]. Separate research efforts have attempted to address memory safety issues through specialized allocators with distinct protection mechanisms. Fast-Forward Allocation [50] prevents use-after-free exploits by implementing one-time memory allocation, ensuring that each

address is permanently retired after deallocation to eliminate memory reuse vulnerabilities. Freeguard [41] offers a high-performance secure heap allocator that mitigates multiple heap-related attacks through randomized memory layout and guard pages. MarkUs [3] counters use-after-free errors by quarantining freed memory blocks and delaying reallocation until it is confirmed that no dangling pointers reference the reclaimed space. In all cases, these features add overhead to the runtime performance of the C programs to which they are applied. The most performant, Fast-Forward Allocation, added a performance overhead (runtime) of 2.3% and a memory overhead of 61% for SPEC CPU2006 benchmarks, but added negligible overhead to the performance of ChakraCore and Nginx during testing of real-world applications.

3.2

Protection Breadth

Attempts to make C memory safe tend to target specific known memory safety problems. While this targeted approach is required to maintain a reasonable scope of work and get a detailed analysis on the level of protection, it also means no single solution provides complete memory safety. Each solution only solves one problem, or at most one category, of memory safety. For security-critical applications, this makes C and C++ inferior to native memory-safe languages. Prior works have evaluated individual mechanisms in isolation and proven they are effective at mitigating specific vulnerabilities. However, such isolated evaluations do not reveal how protections interact when combined, not whether their cumulative coverage and performance costs justify retrofitting compared to migration. Combinatorial evaluation therefor becomes essential to address the engineering decision gap: Is a layered C/C++ defense stack more cost-effective than adopting Rust or Go?

3.3

Research Questions

In line with the above, our work attempts to address the following research questions: • RQ1: Can combinations of existing C/C++ memory safety mechanisms achieve coverage comparable to Rust’s protection scope? Current state-of-the-art C/C++ protections (Section 3.1) typically target isolated memory safety flaws (e.g., Checked C for spatial errors and MarkUs for temporal errors), but remain fragmented in scope. Crucially, “comprehensive coverage” here is operationally defined as matching native memory-safe languages’ systematic prevention of spatial and temporal memory safety errors. Without confirming whether combinatorial approaches can collectively cover these categories before investing in performance/efficacy testing, we risk optimizing solutions for inherently incomplete protection models. • RQ2: Can individual safety methods be combined to give more robust protection to C programs? Due to state-of-the-art protections being spread across many different solutions, getting the same protection provided by native memory safe languages will require combining these solutions to address the various weaknesses that may be present. The protections provided by these solutions should complement each other rather than interfere. To confirm whether a combination of these protections is plausible, it

Laramore, Spracklen, and Jadliwala

needs to be tested that their existing functionality is not adversely impacted when working in combination with other protections. • RQ3: What performance penalties arise from deploying individual memory safety mechanisms, layered combinations of mechanisms, and different protection stacks, relative to natively memory-safe languages? Individual protections incur performance overheads, but the cost of combining multiple mechanisms cannot be inferred from their isolated effects and must be evaluated directly. These combined overheads are then compared against native memory-safe languages to determine whether retrofitted defenses impose similar or fundamentally different performance costs.

4 Methodology 4.1 Threat Model This work formalizes a threat model for memory safety protections in C programs, focusing on defenses against spatial and temporal memory violations under the constraint that adversaries possess only the ability to craft malicious input data while lacking privileges to modify executable binaries, runtime environments, or underlying hardware. We consider an adversary whose objective is to achieve arbitrary code execution or information disclosure via memory corruption vulnerabilities (e.g., buffer overflows, use-after-free, or type confusion), operating within the standard attack surface of unprivileged user-space applications. Crucially, the threat model assumes that the adversary cannot subvert the compilation toolchain, tamper with runtime libraries, or exploit side channels beyond the memory safety boundaries enforced by the protection mechanisms. This scope explicitly excludes threats involving physical access, kernel exploits, or compiler backdoors, thereby isolating the evaluation to software-only memory corruption vectors originating from untrusted input.

4.2

Protection Mechanisms

Checked C [44] is a Microsoft Research extension to the C language designed to enforce spatial memory safety while maintaining backward compatibility with existing C code. Its core innovation lies in introducing three distinct checked pointer types with explicit bounds declarations, enabling both static verification and optimized runtime checks: • _Ptr<T>: This type is used for single-object pointers where dereferencing occurs without arithmetic. The compiler inserts null checks before dereference. Pointer arithmetic is not performed on these pointers, as they are intended solely for dereference operations (e.g., output parameters such as _Ptr<int> out in functions). • _Array_ptr<T>: This type supports pointer arithmetic for arrays. Programmers declare bounds expressions (e.g., count(n) and byte_count(s*n)) specifying valid memory ranges. Bounds expressions consist of non-modifying C expressions and can involve variables, parameters, and struct field members. The compiler either statically proves accesses are safe or inserts runtime bounds checks before dereference. Bounds are stored separately from the pointer itself (avoiding “fat pointers”).

• _Nt_array_ptr<T>: This type extends _Array_ptr<T> for null terminated arrays (e.g., strings). Types without explicit bounds default to bounds of count(0). It enforces zero-termination invariants by restricting writes beyond the current bounds unless the terminating null is verified. Checked C enables incremental adoption through two mechanisms: • Checked regions: Annotated via #pragma, function prototypes, or blocks such as _Checked { ... }, these regions enforce strict spatial safety. Within the checked regions: (1) explicit casts from unchecked to checked pointers are disallowed, (2) reads/writes to unchecked pointers are restricted to ensure spatial safety, and (3) all pointer operations undergo mandatory null/bounds checks. • Bounds-safe interfaces: Allow unchecked legacy code (e.g., standard libraries) and as-yet unconverted code to interoperate safely with checked regions. For instance, fwrite can be annotated as void *p byte_count(s*n), enabling the compiler to treat p as _Array_ptr<void> in checked contexts while preserving C compatibility elsewhere. Checks are inserted only when static verification fails. All valid C remains valid Checked C, as unchecked pointers (T*) coexist with checked types, enabling gradual conversion—developers can start with isolated checked regions and expand them incrementally. Hardware-assisted AddressSanitizer (HWASAN) [45] uses ARM v8’s Top-Byte Ignore (TBI) to detect memory errors (buffer overflows, use-after-free, stack use-after-return) with 10-35% RAM overhead [34], significantly lower than ASAN. It embeds 8-bit tags in the pointer top bytes and shadow memory, validating matches during memory access. Collisions cause a 0.39% per-access error miss probability, but eliminate ASAN’s redzones/quarantine. HWASAN reliably detects stack use-after-return and supports ARM64 (full TBI) and x86_64 (limited page table aliasing). MarkUs [3] is a prototype memory allocator that prevents useafter-free (UAF) vulnerabilities in C through a two-phase memory reclamation process decoupling programmer-initiated freeing from actual memory reuse. When free() is called, MarkUs places the freed object onto a quarantine list rather than immediately returning it to free lists. Verification occurs through periodic “marking” inspired by garbage collection but optimized for C/C++ constraints. MarkUs initiates conservative graph traversal from root pointers, treating any word within heap bounds as a potential pointer, and recursively identifying all live objects accessible from program roots, flagging quarantined memory still reachable via dangling pointers. This marking tolerates false positives, prioritizing safety over precision. After marking, MarkUs walks the quarantine list: unmarked objects are confirmed unreachable and migrated to size-categorized free lists for future allocation, while marked objects remain under quarantine for re-evaluation. This process ensures that UAF exploits become unexploitable by guaranteeing freed memory cannot be reallocated while any live pointer could reference it, with reallocation occurring strictly after empirical validation of pointer absence via marking, and allowing early physical memory reclamation for large objects while retaining metadata under quarantine. FFMalloc [50] is an experimental memory allocator that eliminates UAF vulnerabilities in C programs through one-time allocation, ensuring that each virtual memory address is assigned to exactly one

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Table 1: Comparison of memory safety mechanisms across vulnerability types. ⊕ indicates complete protection, ⊖ partial protection, ⃝ no protection. CVE counts from NVD API [33]. Vulnerability (CWE) Buffer Overflow (120) Buffer Overread (126) Use-after-free (416) Double Free (415) Uninitialized Var (457) Null Pointer Deref (476)

CVE Count

Rust

Go

Checked C

HWASAN

MarkUs

FFMalloc

2,795 74 5,121 574 19 3,723

⊕ ⊕ ⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⃝ ⃝ ⃝ ⊕

⊕ ⊕ ⊕ ⊕ ⊕ ⊕

⃝ ⃝ ⊕ ⊕ ⃝ ⃝

⃝ ⃝ ⊕ ⊕ ⊖ ⃝

allocation request during execution and never reused, though freed memory regions are eventually returned to the operating system when sufficient contiguous pages accumulate. This prevents attackers from reclaiming freed memory to manipulate the contents, removing exploitation of dangling pointers through content control. To maintain performance, FFMalloc combines two specialized allocators: a binning allocator for requests up to 2048 bytes prevents small persistent allocations from blocking page release, while a continuous allocator for larger requests minimizes alignment-related waste. Memory is organized into per-core 4MB pools to reduce contention, with freed pages batch-released to the kernel via munmap only after contiguous blocks of at least eight pages accumulate. Address tracking uses a three-level tree structure mapping addresses to pool metadata, with status information stored differently per allocator type: the continuous allocator encodes states in the least significant bits of 8-byte aligned addresses, while the binning allocator uses per-page bitmaps. This metadata system enforces the one-time allocation guarantee, while inherently enabling detection of double-free and invalid-free conditions. Table 1 provides a visual summary of the claimed protections for the various protection mechanisms to illustrate the coverage they provide and the intent when combining these mechanisms.

4.3

Protection Validation

We validation the functional correctness of each protection mechanism with respect to its claimed security guarantees, following the evaluation methodology described in the original works. For Checked C, we evaluate the reference programs provided by the authors [13], which demonstrate protection against spatial memory errors such as buffer overflows, out-of-bounds accesses, and null pointer dereferences when code is properly annotated. For FFMalloc, we rely on the exploitation and validation examples included in its original evaluation, which demonstrate prevention of temporal memory errors including use-after-free and double-free vulnerabilities. MarkUs does not provide a standalone validation suite but because its protections closely mirror FFMalloc, and FFMalloc explicitly cites MarkUs, we evaluate MarkUs using the same test cases. To assess whether these mechanisms remain effective when combined, we consider a protection stack valid if each constituent mechanism preserves its expected behavior under composition. This validation criterion aligns with prior work evaluating the layering of security mechanisms, which treats preservation of individual guarantees under combination as the baseline requirement for practical deployment.

4.4

Performance Evaluation

To create an initial comparison, we selected experimental features to create combinations: HWASAN, MarkUs, and FFMalloc. These features were tested individually and in various configurations with standard compiler protections. The features can be seen in Table B. This allowed us to measure the performance trade-offs required for more robust memory security. Native compiler protections were included to provide a comparison point to how a program might run in production today, and added to the combinations to show what performance might look like in a production system with added memory safety mechanisms.

4.5

Platform

The benchmarks were conducted on an Apple M1 MacBook Air [7]. Apple Silicon was chosen as the desired platform due to its desktopclass performance and the availability of ARM 8.5 instructions for testing. ARM architecture was selected as the primary platform for three reasons: (1) it represents the only architecture with full HWASAN support, enabling evaluation of hardware-assisted memory safety; (2) ARM’s increasing adoption in cloud computing (AWS Graviton, Azure Cobalt) makes it relevant for future production deployments; and (3) it provides a consistent platform for comparing relative performance impacts across protection mechanisms. While x86_64 remains dominant in current deployments, the relative performance penalties observed here provide valuable insights for any architecture. To run the benchmark programs, an instance of Linux was required to minimize possible compatibility issues. At the time of testing, the only usable Linux distribution available for Apple Silicon was Asahi Linux [8], specifically version 40 with kernel version 6.12.1.

4.6

Software

Addressing existing memory safety issues in C is crucial, and any proposed mitigations should be evaluated for their impact on overall program performance, if they are to be considered feasible. To effectively benchmark these memory safety mitigations, it is essential to establish a baseline of expected performance across various tasks. The Computer Language Benchmark Games (CLBG) [46], hosted by the Debian project, served as an initial reference for benchmarking due to their popular methodology in comparing the performance of different programs and different programming languages. The CLBG is conducted as a public competition among different programming languages, featuring a range of predefined

Laramore, Spracklen, and Jadliwala

tasks. Programmers are challenged to write solutions in any language they choose, aiming for the lowest runtime, with the fastest solution being recognized as the “winner” for that task. Notably, C and Rust frequently compete for the title of the fastest language, making them excellent foundations for this analysis. We also used hyperfine to manage benchmark execution and measurement, in accordance with CLBG recommendations [12].

4.7

Benchmarking

We utilize all nine benchmarking tasks featured in CLBG for our study: (1) binarytrees - Constructs perfect binary trees in their entirety prior to any garbage collection of tree nodes. (2) pidigits - Employs arbitrary precision arithmetic alongside a consistent step-by-step, single-threaded algorithm to generate digits of Pi. (3) regex-redux - Manipulates FASTA format data using identical regex patterns and actions. (4) fannkuch-redux - Implements a custom algorithm as delineated in “Performing Lisp Analysis of the FANNKUCH Benchmark” [5] (5) n-body - Models the orbits of Jovian planets utilizing a straightforward symplectic integrator. (6) spectral-norm - Computes the spectral norm of an infinite matrix 𝐴 with specified entries. (7) mandelbrot - Plots the Mandelbrot set [-1.5-i,0.5+i] on an N-by-N bitmap (8) fasta - Generates DNA sequences by replicating a given sequence and through weighted random selection from two alphabets. (9) k-nucleotide - Utilizes a built-in or library hash table to aggregate count values, allowing for key lookups and updates to the corresponding counts within the hash table. Our investigation evaluates all available candidates written in Rust, Go, and C for the tasks described above. During this assessment, certain benchmark candidates were observed to utilize x86_64 hardware acceleration, leading to their exclusion from the testing, as the evaluation was conducted on an AArch64 platform. To facilitate the execution of tests, a JSON configuration file, accompanied by a Python script, was developed to automate the process. The JSON configuration encompassed each candidate code across the different programming languages, along with their compilation prerequisites as delineated in the original CLBG. This configuration served as the control group for performance comparisons. Subsequently, a series of modifications were identified to alter the execution parameters for the C language candidates: • Compiler Selection: Candidates were compiled using gcc, clang, or checked-c, with the control group using gcc. Both gcc and clang are highly regarded within the C development community, with clang offering features that are particularly relevant to this study, such as Hardware AddressSanitizer (HWASAN). Checked C is a fork of clang 12 and required some slight modifications to the programs. The Checked C 3c tool [28] was used to automatically annotate the existing C code to enable the extra features that Checked C provides. However, not all benchmark candidates worked

with the default configuration. Of the 33 total benchmark candidates, 14 (42%) failed to compile with Checked C annotations and were excluded. All failing candidates utilized OpenMP for multi-threading parallelization. This consistent failure pattern suggests limitations in the 3c annotation tool’s handling of OpenMP pragmas or fundamental incompatibilities between Checked C’s type system and OpenMP’s threading model-a determination beyond the scope of this study. This limitation restricts generalizability to programs compatible with Checked C’s current alpha implementation. Those programs that did not compile with annotations were omitted from the performance analysis. • Compiler Hardening Flags: The study incorporated recommended compiler hardening flags, as outlined in the “Compiler Options Hardening Guide for C and C++” provided by OpenSSF [24]. The options included either a basic selection or an extended set of production-use flags. These flags enable compiler protections that influence the behavior of the final binary, guided by OpenSSF’s recommendations for robust security. • AddressSanitizer and Hardware-assisted AddressSanitizer: The configurations allowed for the enabling of Address Sanitization in the compiler, or alternatively, the activation of ARM’s HWASAN for hardware-assisted ASAN, utilizing the top-byte ignore feature. ASAN serves as a memory debugging tool in C compilers, significantly increasing both memory usage and CPU overhead, but facilitating the resolution of obscure memory issues. Although this overhead limits ASAN’s practicality to debugging scenarios, its capabilities hold potential merit for enhancing security. HWASAN, conversely, effectively leverages ARM’s topbyte-ignore feature to mitigate ASAN’s memory overhead, representing a potential advancement toward employing ASAN-like capabilities in production environments. • Memory Allocation Alternatives: The study considered two proof-of-concept replacements for the standard malloc function, namely MarkUs and FFMalloc, both of which are designed to address use-after-free vulnerabilities. Each modification set was selectively incorporated into the control group configuration. For instance, a configuration could utilize clang, integrate the tl;dr hardening flags, exclude ASAN, and implement FFMalloc. However, it was not permissible to combine both ASAN and HWASAN within the same configuration due to their inherent conflicts as modifications of the same type. Details on the different modifications tested in this study can be found in Appendix B.

5 Results 5.1 RQ1: Comparison of claimed memory protections 5.1.1 Checked C. Checked C focuses on protections on bounds checking and null pointer checking with the addition of new pointer and array types. The added types will run checks when used and cause program stoppage when attempting to access memory that is out of bounds of a checked variable or upon dereferencing a null pointer [44]. Use of Checked C along with it’s new types should

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

address buffer overflows, buffer overreads, and null pointer dereference problems. Checked C does not address temporal memory safety issues; however, the addition of these protections has been suggested by the authors of Checked C [16]. 5.1.2 HWASAN. HWASAN is intended to be a lower resource intensive alternative to ASAN. ASAN and HWASAN are designed to debug memory problems in C programs and thus cause failure for all of the memory safety flaws. HWASAN does, however, come with some caveats [45]: • HWASAN is limited to use on AArch64 architectures due to its use of ARM’s hardware address tagging. Intel CPUs on x86_64 can use HWASAN in a limited capacity so long as the CPU supports Intel’s Linear Address Masking. • Detection of buffer overflows or use-after-free is probabilistic with a 6.25% chance of missing a bug with four-bit tags and 0.39% chance with eight-bit tags [45] • Can cause interference with other programs that may make use of high bits in an address. • As a diagnostic tool, HWASAN is unsuitable for production security because its debugging features may require disabling or bypassing standard security mitigations to facilitate error reporting. HWASAN was included despite its ARM-specific design and current unsuitability for production software to evaluate whether hardware-assisted approaches could provide production-viable alternatives to ASAN. While ASAN explicitly states that it is not suitable for production, as does HWASAN, HWASAN promises lower overhead. Testing revealed fundamental compatibility issues that suggest that hardware-assisted sanitizers require further development before they can be deployed in production. Despite these caveats, testing the combination of memory tagging with other memory safety features using HWASAN will be helpful as a preliminary look at what a solution like ARM MTE may be able to accomplish. 5.1.3 MarkUs. Despite protecting from use-after-free vulnerabilities, MarkUs does not completely protect against double-free issues [3]. Although the target vulnerability for MarkUs is use-after-free, the MarkUs paper [3] mentions that an attacker may be able to use an existing double free vulnerability to edit the quarantine list, but editing the list would not result in any useful privilege escalation, thereby giving some protection for double-free as well [3]. We chose to count the weak protection since the double-free would not be usefully exploitable, despite it technically still being possible according to the original study. 5.1.4 FFMalloc. FFMalloc is an alternative memory allocator that uses Fast-Forward allocation to fully prevent use-after-free and double-free vulnerabilities. FFMalloc provides only partial mitigation for the Uninitialized Variables category; while it successfully handles invalid-free errors, it does not detect the use of uninitialized pointers. 5.1.5 Summary. As demonstrated in Table 1, current state-of-theart solutions, excluding HWASAN, fail to provide a combination of protections that matches the comprehensive safety guarantees of Rust. Combining Checked C with either FFMalloc or MarkUs

covered only five of the six categories of memory safety issues. That combination still greatly increases memory safety to give both spatial and temporal memory safety, even if incomplete in comparison. Based on vulnerability CWE classification, 12287 CVEs are potentially mitigated, covering 99.8% of the CVEs in the analyzed CWE categories. The benefit of using mechanisms like Checked C, FFMalloc, and MarkUs is the ability to retrofit existing code with these features to avoid the time and effort required for a complete rewrite. However, it should be noted that should one choose to incorporate Checked C, there are changes required to ensure the use of the custom types offered by Checked C. This analysis acknowledges two critical limitations: (1) CVE records reflect reported vulnerabilities rather than absolute exploit prevalence, and (2) coverage claims derive from theoretical protection scope (e.g., Checked C for spatial errors, MarkUs for temporal errors) rather than empirical validation against each CVE. Crucially, we frame these findings as indicative of vulnerability type coverage, not definitive prevention counts, aligning with security community standards that emphasize vulnerability class elimination over raw CVE metrics. Our threshold for meaningful progress (≥5/6 categories) remains justified by the concentration of high-impact risks in the covered categories, while acknowledging that individual CVE mitigation would require casespecific validation. The fundamental takeaway of RQ1 is that a protection scope comparable to Rust is indeed achievable through a combination of C retrofits; by mitigating 99.8% of analyzed vulnerability classes, these combinations prove to be a practically significant alternative that justifies a shift from isolated tool testing toward the holistic evaluation of integrated defenses. As of 2026 there is no established method for this holistic evaluation!

5.2

RQ2: Effectiveness of the Combination of Safety Methods

5.2.1 Methodology. Combining these methods of memory safety is only useful if the protections hold in combination. To validate whether protections still hold in combination, we used samples directly from the referenced literature. To test the memory protections of Checked C, sample programs are used from the Checked C GitHub repository and behavior is compared before and after adding HWASAN, MarkUs, and FFMalloc. To test for protection against use-after-free vulnerabilities, the Capture-the-Flag tests used to test FFMalloc are compiled using combinations of Checked C, HWASAN, MarkUs, and FFMalloc. The programs are initially exploitable with no protections enabled, so exploitation should not be possible after introducing temporal memory safety protections. The vulnerabilities are tested with the same combinations and results noted in Table 4.3. 5.2.2 Results. The experimental evaluation reveals critical insights regarding the interoperability of memory safety frameworks with Checked C. When executing the standardized Checked C example suite (Table 2), HWASAN exhibited complete incompatibility with the Checked C compiler toolchain, manifesting as immediate failure upon attempting to allocate memory. This failure occurs because HWASAN’s instrumentation pass conflicts with Checked C’s extended type system semantics, which alter pointer representation in ways that violate HWASAN’s memory tagging assumptions.

Laramore, Spracklen, and Jadliwala

Table 2: A filled circle (⊕) signifies the expected behavior was unchanged, and the program received the expected exit code(s) noted under the exit code column with the additional modification. An empty circle (⃝) indicates that the program’s behavior changed after adding the mechanism in combination. Checked C

HWASAN

MarkUs

FFMalloc

avoid-warning.c [13] echo-args.c [13] echo-args-buggy.c [13] exit-on-failure.c [13] hello-world.c [13]

⃝ ⃝ ⃝ ⃝ ⃝

⊕ ⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⊕ ⊕ ⊕

Table 3: Vulnerability test results comparing behavior during use-after-free exploit. A filled circle (⊕) confirms expected behavior. Our tests validated that all combined protections resulted in expected behavior. Test Case ghostparty [37] uaf [36] Python Issue 24613 [27] Python Issue 39421 [14]

Checked C

HWASAN

MarkUs

FFMalloc

⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⊕ ⊕

⊕ ⊕ ⊕ ⊕

Conversely, runtime memory allocators demonstrated seamless integration with Checked C’s compile-time protections. Both MarkUs (a metadata-enriched allocator) and FFMalloc (a fencebased allocator) maintained identical behavioral profiles when processing Checked C-annotated binaries compared to their operation with standard C code. This compatibility was empirically validated through two canonical test cases: • The echo-args-buggy.c example, which intentionally triggers an off-by-one error, consistently generated an exception regardless of allocator implementation • The exit-on-failure.c test case, designed to showcase Checked C’s bounds checking, produced identical termination behavior under all allocator configurations These results confirm that Checked C’s bounds enforcement operates orthogonally to heap memory management layers, as the allocator swap neither suppressed nor altered the spatial safety violations detected by Checked C’s type system. Further validation through Capture-the-Flag security challenges demonstrated robust compatibility across protection layers. When compiled with Checked C annotations (automatically inserted via the 3c conversion tool), all test binaries maintained full exploit resistance under MarkUs and FFMalloc. Notably, the protections remained effective even when Checked C annotations were applied to legacy codebases originally developed without spatial safety considerations. Architectural constraints necessitated methodological adaptation for Python Issue #24613 testing. The absence of aarch64-compatible HWASAN tooling prevented direct evaluation on Apple Silicon platforms. Consequently, we migrated testing to x86_64 infrastructure,

where both MarkUs and FFMalloc successfully mitigated exploitation attempts even when Python was compiled with Checked C annotations. 5.2.3 Summary. The empirical analysis demonstrates that allocatorbased memory safety mechanisms exhibit full interoperability with Checked C, as evidenced by Table 3. However, a critical exception to this interoperability principle emerges in the interaction between HWASAN and the Checked C extension. The tests reveal that HWASAN consistently fails during runtime validation when processing Checked C-annotated codebases, incorrectly producing segmentation faults in all cases. This incompatibility stems from fundamental architectural conflicts: Checked C introduces extended pointer types (_Ptr<T>, _Array_ptr<T>) with compiler-enforced spatial and temporal safety guarantees, while HWASAN relies on precise memory tagging aligned with standard C type semantics. Specifically, Checked C’s custom type management system modifies pointer representation and metadata handling in ways that violate HWASAN’s assumptions about memory layout expectations. This incompatibility represents a significant barrier to comprehensive adoption of memory safety in legacy code modernization efforts using memory sanitization functionality. Three possible directions to resolve this conflict are: • Development of a unified metadata schema that accommodates Checked C’s extended type annotations within HWASAN’s tagging framework • Creation of a translation layer in LLVM’s middle-end to reconcile Checked C’s type transformations with HWASAN’s instrumentation passes • Formal verification of the combined system using separation logic to prove the absence of semantic conflicts Such work would enable synergistic deployment of compile-time (Checked C) and runtime (HWASAN) protections, closing a critical gap in the memory safety toolchain.

5.3

RQ3: Performance Evaluation

A total of 2,316 benchmark tests were conducted, including control baselines, with each modification combination tested over 30 independent runs to support statistical comparison. Normality of execution-time distributionswas assessed using the Shapiro-Wilk test [40]. At a significance level of 0.05, 740 sample sets did not exhibit evidence of non-normality, while 1,576 deviated significantly from a normal distribution. Given this prevalence of non-normality, median values were used as measures of central tendency and non-parametric statistical tests were employed. To compare execution times between Rust and C implementations, we applied the Mann-Whitney U test [29], testing the null hypothesis that the two languages produce execution-time distributions with equal central tendency.To compare unmodified C baselines against C binaries compiled with memory-safety mechanisms, we used the Wilcoxon signed-rank test [51], testing the null hypothesis that applying a given protection mechanism does not change execution time relative to the baseline. Across all comparisons, 24,903 test pairs yielded sufficient evidence to reject their respective null hypotheses at 𝛼 = 0.05, while 895 comparisons showed no evidence of a statistically significant performance difference.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 1: Massif memory usage graph for mandelbrot displayed using Massif Visualizer

Figure 2: Massif memory usage graph for k-nucleotide displayed using Massif Visualizer

5.3.1 Stability. The application of various modification sets did not consistently lead to successful executions. Many of the errors encountered were attributed to memory mismanagement, as identified by ASAN and HWASAN. In the case of “binarytrees.c”, the function NewTreeNode() allocates memory using malloc to create a new treeNode; however, this memory is not deallocated, resulting in a memory leak upon program termination. Similar issues arise in other instances, such as in fasta.gcc-7.gcc at line 146. Discovering memory leaks of this nature is not unexpected, given the context of the programs being assessed. These programs are small, simple, and designed to complete specific tasks for a benchmarking competition. They execute once and can depend on the operating system to reclaim any memory not otherwise deallocated. Although the memory leak does not significantly impact the tasks at hand under the benchmark parameters, it underscores the utility of ASAN in debugging. It suggests that a more performant ASAN implementation could provide greater protection in production environments, considering that other modifications did not detect the leak. The error code charts feature data points from ASAN and HWASAN that completed successfully but ultimately exited due to memory not being deallocated. Once again, another significant source of errors stemmed from the combination of ASAN/HWASAN with Checked C. When compiled with both, the binaries failed to allocate memory and terminated immediately. The failure occurred at line 54 of sanitizer_ common.cpp in the Checked C compiler, with error code 22 indicating an "Invalid Argument." This behavior was consistently observed across the benchmarks, and given that the Checked C code is considered an alpha implementation, it seems reasonable to speculate that this may be a bug. Finally, the error figures in Appendix 18 when split by compiler show that clang failed more often than gcc. While the two compilers are technically different, it was a bit surprising to see clang having more errors than gcc. Further investigation found that all errors have to do with HWASAN. A core dump was triggered on all benchmarks using clang with HWASAN after running atoi() with argv given as an argument. An example is shown in Figure 3. With significantly more failures, it shows that HWASAN, while more performant, may not be ready for production use (at least when using clang).

5.3.2 Workload Classification. Differences in performance are not consistent across the tests. The fundamental difference lies in each program’s memory access patterns and computational characteristics. Mandelbrot’s execution is predominantly CPU-bound with minimal heap activity, where approximately 92% of its peak memory usage (87 MB out of 94.6 MB) consists of thread stacks for OpenMP parallelization rather than dynamic heap allocations (Figure 1). This workload profile means that the program spends most of its time performing computations rather than memory operations, making it relatively insensitive to the overhead introduced by different security mechanisms. Fasta performs similarly with relatively little change across various combinations of memory protections. Conversely, k-nucleotide demonstrates relatively intensive heap activity with frequent allocation/deallocation cycles, where 57% of its peak memory usage (268 MB out of 467 MB) consists of program data allocated through standard malloc operations, including significant realloc activity accounting for 130 MB of memory usage (Figure 2). This memory-bound workload can be hampered depending on the mechanism of protection. For example, this directly conflicts with FFMalloc’s design principle, where "any given virtual memory address is only returned to the calling application once" [50], forcing new allocations for operations that would normally reuse memory in standard allocators. These contrasting behaviors highlight a critical insight: the performance penalty of memory safety protections is not inherent to the protection mechanism itself, but rather emerges from the interaction between the protection strategy and the application’s memory usage patterns. As noted in FFMalloc research [50], it could more aggresively release pages, but this tradeoff only becomes performance-critical for applications like k-nucleotide that frequently reuse memory locations. For CPU-bound applications such as mandelbrot with minimal memory reuse patterns, security benefits can be obtained with negligible performance impact.

5.3.3 Individual Protection Mechanisms. (HW)ASAN Empirical analysis reveals that while HWASAN generally offers comparable or better runtime than ASAN in many scenarios, certain workload characteristics can reverse this expectation. When evaluating two implementations of the same task, binarytrees.gcc-3.gcc and

Laramore, Spracklen, and Jadliwala

Program received signal SIGSEGV, Segmentation fault. 0x0000fffff7b75814 in __GI_____strtol_l_internal ... at ../stdlib/strtol_l.c:304 304 while (ISSPACE (*s)) #0 #1 #2 #3

0x0000fffff7b75814 in __GI_____strtol_l_internal ... at ../stdlib/strtol_l.c:304 0x0000fffff7b7575c [PAC] in __strtol ... at ../stdlib/strtol.c:117 0x00000000004670c0 in atoi (__nptr=0x0) at /usr/include/stdlib.h:483 main (argc=<optimized out>, argv=<optimized out>) at fannkuchredux.gcc-5.gcc:30

Figure 3: GDB output from debugging (ellipses indicate truncated text for space) binarytrees.gcc-5.gcc under identical conditions on Asahi Linux running on Apple Silicon hardware, it was observed that binarytrees.gcc3.gcc exhibited a significant runtime penalty with HWASAN compared to ASAN with the HWASAN median time being 42% slower than the ASAN median time, while binarytrees.gcc-5.gcc the HWASAN median time was 36% faster than the ASAN median time. Massif profiling with –time-unit=B revealed that binarytrees.gcc-5.gcc had 64 snapshots recording stack changes (Figure 7) and 71 snapshots recording heap changes (Figure 6) while binarytrees.gcc-3.gcc had 42 snapshots recording stack changes (Figure 5) and 10 snapshots recording heap changes (Figure 4). This evidence strongly suggests that HWASAN’s runtime characteristics are particularly sensitive to programs with a higher ratio of stack operations. The technical explanation lies in the fundamental differences between the stack protection mechanisms of the two sanitizers: while ASAN primarily relies on redzone padding with compiler instrumentation, HWASAN leverages the top-byte-ignore feature of aarch64 to store metadata in the top byte of pointers [45]. This hardware-assisted approach enables HWASAN to detect additional error classes including stack-use-after-return that ASAN may miss [35], but requires additional operations during each stack frame management event-tag generation, storage, and consistency checking. However, this methodology is much more efficient than ASAN’s when applied to the heap. Consequently, in workloads with intensive stack allocation ratios like binarytrees.gcc-3.gcc, the overhead of these additional operations on the stack results in runtimes that are worse than ASAN. While ASAN also monitors the stack, it utilizes a less computationally demanding approach of redzone poisoning and shadow memory bit-flipping to detect overflows. In contrast, HWASAN implements a more intensive stack protection protocol. While this methodology enables HWASAN to detect complex error classes, it introduces a significant performance penalty in environments characterized by high-frequency stack frame management. This finding demonstrates that memory sanitizer selection must consider not only the target architecture and the desired error coverage, but also the specific memory access patterns of the application if the feature is intended to be implemented in a production capacity, as the use of HWASAN does not necessarily translate into a runtime advantage despite lower RAM overhead (10%-35%) [45]. Memory Allocators FFMalloc and MarkUs exhibited workloaddependent performance characteristics that diverged from their published benchmarks. Although the original papers reported overheads of 2.3% (FFMalloc[50]) and 5% (MarkUs[3]) on SPEC CPU2006, our testing revealed penalties ranging from negligible (<5%) for CPU-bound workloads to substantial (>70%) for allocation-intensive

programs such as k-nucleotide. This discrepancy likely stems from differences in workload characteristics: SPEC CPU2006 emphasizes computational tasks with predictable memory patterns, while the CLBG benchmarks include allocation-intensive scenarios that stress the allocators’ quarantine mechanisms. The k-nucleotide benchmark’s extensive use of realloc operations particularly conflicts with FFMalloc’s forward-only allocation strategy, as each reallocation requires entirely new address space rather than in-place expansion. Compiler Selection Analysis of mean runtimes across successful protection combinations reveals nuanced compiler performance characteristics that challenge conventional expectations about compiler age and optimization. Due to the high failure rate of Checked C configurations (absent entirely from knucleotide and spectralnorm benchmarks), mean values provide more representative comparisons than medians for successful runs. Surprisingly, Checked C (based on clang-12) outperformed modern compilers in several benchmarks, despite its outdated foundation. For computeintensive workloads like fannkuchredux (21.5s vs 25.1s clang, 32.1s gcc) and pidigits (427ms vs 653ms clang, 798ms gcc), Checked C demonstrated superior optimization. However, Checked C showed significant degradation in others, particularly regexredux (2.56s vs 1.17-1.34s) and mandelbrot when compared to modern clang (14.2s vs 4.7s). The absence of Checked C results for knucleotide and spectralnorm, due to compilation or runtime failures, underscores the fragility of the Checked C toolchain rather than performance concerns. These findings suggest that Checked C’s performance penalties are not uniform but highly workload-dependent, with the primary limitation being compatibility rather than raw performance. The selective success of Checked C configurations creates a survivorship bias in the performance data: only simpler, less-protected configurations succeed, potentially explaining the unexpectedly competitive performance in successful runs.

5.4

RQ3: Memory Protections in Combination

Non-linear Scaling of Protection Mechanisms The empirical data from Tables E and D demonstrate conclusively that combining memory protection mechanisms produces non-linear performance degradation patterns that cannot be predicted from individual component analysis. Examining the knucleotide benchmark runtime data reveals striking non-linearity. The baseline configuration shows a median runtime of 1.93s (Tables D.1). When FFMalloc is added individually [0,0,0,0,1,0], the median drops to 1.86s—actually improving performance by 3.7%. Several combinations of memory allocator and OpenSSF flags stay below the runtime of the control by as much as

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 4: Massif heap memory usage graph for binarytrees.gcc-3.gcc displayed using Massif Visualizer

Figure 5: Massif stack memory usage graph for binarytrees.gcc-3.gcc displayed using Massif Visualizer

5.4%. However, when FFMalloc combines with ASAN [0,0,1,0,1,0], the median jumps to 3.99s, representing a 69.4% degradation from baseline. If the effects were linear, we would expect approximately 65.7% overhead (69.4% from ASAN alone minus the 3.7% improvement from FFMalloc), but instead observe the full 69.4%—demonstrating that FFMalloc’s optimization completely disappears when combined with ASAN. The binarytrees benchmark provides another compelling example of non-additive overhead. From Tables E.2, the baseline shows a 4.43s mean runtime. ASAN alone adds an overhead of 106.2%. The combination of OpenSSF production flags along with FFMalloc adds an overhead of 9.4%. Combine all of those to enable OpenSSF production flags, ASAN, and FFMalloc, and the overhead compared to control is 106.78%, negligibly different than ASAN alone. The mean runtime analysis (Tables E.2) further confirms non-linearity. For fannkuchredux, the OpenSSF production flags alone add minimal overhead, while HWASAN alone adds 78.7% overhead. Yet their combination shows 79.0% overhead—virtually identical to HWASAN alone, indicating the production flags become negligible when combined with HWASAN’s instrumentation. Comparing Runtimes to Rust and Go The experimental objective was to achieve memory-safety guarantees comparable to natively memory-safe languages while addressing equivalent classes of memory vulnerabilities within C programs. Performance evaluation across nine standardized benchmarks revealed Rust’s computational efficiency relative to the implemented mitigation techniques. As documented in Tables D.3, Rust demonstrated the lowest median runtime in seven of nine benchmarks. Meanwhile, Go had a lower median runtime than at least one C compiler in six out of nine benchmarks. Similarly, Tables E.3 indicates Rust achieved the lowest mean runtime in eight cases, whereas Go had a lower mean runtime than at least one C compiler in four out of nine benchmarks. A notable exception occurred in minimum runtime analysis (Tables F.3), where Rust ranked lowest in only four benchmarks and Go only outperformed at least one C compiler in two out of nine benchmarks. These results indicate that existing methodologies for mitigating memory-safety violations in C introduce measurable runtime overhead, resulting in Rust’s predominant performance advantage across the majority of evaluated metrics. This trade-off

between safety enforcement and execution efficiency warrants consideration in systems programming contexts requiring stringent memory protection. While Go generally exhibits lower runtime performance compared to Rust, the language offers greater approachability for novice programmers through features such as automatic memory management, built-in concurrency primitives, and an extensive package ecosystem. Excluding statistical outliers (binarytrees, regexredux), median Go runtimes remain within 17400% of Rust and C implementations. Although these performance differences are quantitatively significant, in practical application contexts, such differences are often acceptable and can be mitigated by the substantially longer latency inherent in human interaction cycles.

6

Limitations

This work’s primary contribution lies not in prescribing specific memory safety configurations for C, but in advocating for a paradigm shift in how the academic community approaches memory safety research for C. Specifically, it argues that future work must: • Systematically compare memory safety modifications for C against the comprehensive protection models of native memory-safe languages like Rust and Go, and • Design and evaluate compound memory safety mechanisms that collectively approximate the robustness of native solutions. By demonstrating this dual approach through empirical analysis of combined protections and cross-language benchmarking, this paper serves as a methodological exemplar for advancing C memory safety research beyond isolated mechanism evaluation. The intent is not to recommend specific implementation strategies, but to establish a necessary framework for honest comparative analysis that accounts for the modern programming language landscape. This conceptual contribution precedes the technical limitations of the current implementation, which focus on narrower empirical constraints. Given the intent of this paper, several limitations within this study should be kept in mind when reviewing the results of these performance benchmarks. First, Apple Silicon’s unified memory

Laramore, Spracklen, and Jadliwala

Figure 6: Massif heap memory usage graph for binarytrees.gcc-5.gcc displayed using Massif Visualizer

Figure 7: Massif stack memory usage graph for binarytrees.gcc-5.gcc displayed using Massif Visualizer

architecture (UMA) fundamentally differs from traditional x86_64 it lacks full compiler integration beyond experimental clang forks systems by integrating the CPU, GPU, and neural engine into a sin(now obsolete), breaks compatibility with standard C libraries, and gle system-on-a-chip (SoC) with shared high-bandwidth LPDDR5X this paper demonstrated it has breaking incompatibilities, such as memory. This design eliminates data copying between discrete HWASAN feature production-grade compilers don’t have issues components, as all processors access the same memory pool via with, as well as 42% of the benchmark candidates which failed to ultra-fast on-package interconnects. Memory-bound applications compile. MarkUs and FFMalloc are both very young compared to may appear exceptionally efficient on Apple hardware but reveal sigproduction memory allocators present in libraries like glibc or musl, nificant slowdowns on DIMM-based systems where bandwidth conand have only been tested in limited and very controlled circumstraints dominate. Developers must isolate compute-bound metrics stances. Much more testing would be required to begin including from memory-dependent operations when cross-platform benchthem in production workloads. marking, as Apple’s architectural advantages in memory hierarchy can mask underlying algorithmic inefficiencies that only manifest on conventional hardware. 7 Ethical Considerations Second, benchmarks on Asahi Linux require cautious interpretaThis research evaluates the efficacy of C retrofits and native safety tion due to its experimental reverse-engineered approach, which mechanisms through the analysis of memory safety vulnerabilirelies on community-driven trial-and-error to interface with Apple ties. We have considered the ethical implications of this work in Silicon. Without Apple’s documentation, developers reconstruct accordance with the ACM CCS 2026 [1] guidelines and the USENIX hardware behavior through incomplete driver implementations Security Ethics Policy [48]. [8]. Key subsystems like memory management, GPU acceleration, and power efficiency often operate suboptimally compared to macOS or mature x86_64 Linux distributions. Critical optimizations 7.1 Vulnerability Analysis and Disclosure (e.g., cache coherence, thermal throttling) remain partially reverseOur evaluation utilizes publicly available CVEs and established engineered, causing benchmarks to underrepresent hardware pobenchmark suites to assess security defenses. No new vulnerabilities tential. Performance tests may reflect software-layer inefficiencies were discovered in production software during this study. Had any rather than hardware capabilities, skewing comparisons with wellpreviously unknown vulnerabilities been identified, we would have documented platforms. Results lack consistency across versions followed the standard responsible disclosure process, notifying the and platforms due to rapid, unstable development cycles. Updates relevant vendors and maintainers before public dissemination. can introduce significant performance shifts as drivers evolve from approximations to accurate control, rendering longitudinal comparisons unreliable. Unlike x86_64 Linux distributions with decades of vendor-collaborative integration, Asahi’s inferred behavior creates 7.2 Impact and Dual Use transient performance profiles. Apple Silicon’s unified memory While our findings highlight specific limitations and "engineering architecture, which Asahi utilizes less effectively than macOS’s nagaps" in existing security retrofits, we believe the defensive benefits tive optimizations, exacerbates discrepancies in memory-intensive of this research outweigh the risks. By providing a systematic frameworkloads. work for comparing C retrofits against native safety (e.g., Rust), we Lastly, academic tools like Checked C, FFMalloc, and MarkUs repempower engineers to make informed security decisions. We do not resent valuable but inherently limited research prototypes designed provide exploit code or actionable information that would simplify to explore specific memory safety or allocation concepts—not productionthe creation of new attacks; rather, we focus on the architectural grade solutions. Checked C is a proof-of-concept with critical gaps: and performance trade-offs of defensive mechanisms.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

7.3

Human Subjects and Data

This study does not involve human subjects, nor does it utilize private or sensitive user data. All experiments were conducted in controlled environments using open-source software and synthetic or historical vulnerability data. Consequently, Institutional Review Board (IRB) approval was not required for this work.

8

Generative AI Usage

Generative AI (Claude Opus 4.2 [6], Grammarly [2], and Google Gemini 3 [22]) was systematically used to refine grammar, academic tone, and technical phrasing throughout this paper. All AIgenerated suggestions underwent rigorous manual verification for accuracy in programming language concepts and memory safety principles and followed university guidelines on the use of generative AI in academia. The author retains full scholarly ownership of all analytical content, with AI serving strictly as an editing tool under human oversight [26].

References [1] ACM SIGSAC. 2026. ACM CCS 2026 Call for Papers. https://www.sigsac.org/ ccs/CCS2026/call-for/call-for-papers.html. Accessed: 2026-01-14. [2] Grammarly AI. 2025. Grammarly.com. [Grammarly AI LLM features such as spell checking, grammar checking, and prose suggestions were used in the writing of this thesis.]. [3] Sam Ainsworth and Timothy M. Jones. 2020. MarkUs: Drop-in use-after-free prevention for low-level languages. In 2020 IEEE Symposium on Security and Privacy (SP). 578–591. doi:10.1109/SP40000.2020.00058 [4] James Anderson. [n. d.]. Computer Security Technology Planning Study. https: //seclab.cs.ucdavis.edu/projects/history/papers/ande72.pdf. [Accessed 01-012025]. [5] Kenneth R. Anderson and Duane Rettig. 1994. Performing Lisp analysis of the FANNKUCH benchmark. SIGPLAN Lisp Pointers VII, 4 (Oct. 1994), 2–12. doi:10.1145/382109.382124 [6] Anthropic. 2025. Claude Opus 4.2. Large language model. https://claude.ai Used for academic editing on September 15, 2025. [7] Apple. 2020. MacBook Air (M1, 2020) - Technical Specifications. https://support. apple.com/en-us/111883. Accessed: 2025-09-21. [8] Asahi Linux Project. 2025. About Asahi Linux. https://asahilinux.org/about/. Community-developed Linux distribution for Apple Silicon hardware, created via reverse engineering. [9] Arnd Bergmann. 2022. Kbuild: move to -std=gnu11. [10] Andrew Binstock. 2013. The Rise And Fall of Languages in 2013. Dr. Dobbs (2013). https://www.drdobbs.com/jvm/the-rise-and-fall-of-languages-in-2013/ 240165192 [11] Clang. [n. d.]. Control Flow Integrity — Clang 22.0.0git documentation. https: //clang.llvm.org/docs/ControlFlowIntegrity.html. [12] Debian Benchmark Game Team. [n. d.]. Bencher - Benchmark Runner for The Computer Language Benchmarks Game. https://salsa.debian.org/ benchmarksgame-team/benchmarksgame/-/tree/master/bencher. Accessed: 2025-09-21. [13] Checked C Developers. [n. d.]. checkedc/samples at main · checkedc/checkedc. https://github.com/checkedc/checkedc/tree/main/samples [14] dk0n9. [n. d.]. Issue39421: Use-after-free in heappushpop() of heapq module Python tracker. https://bugs.python.org/issue39421 [15] Joshua Dressler. 2007. Cases and materials on criminal law (4th ed ed.). Thomson/West, St. Paul, MN. [16] duneroadrunner. [n. d.]. use-after-free, complementary library · Issue #57 · checkedc/checkedc. https://github.com/checkedc/checkedc/issues/57 [17] Paul Emmerich, Simon Ellmann, Fabian Bonk, Alex Egger, Esaú García SánchezTorija, Thomas Günzel, Sebastian Di Luzio, Alexandru Obada, Maximilian Stadlmeier, Sebastian Voit, and Georg Carle. 2019. The Case for Writing Network Drivers in High-Level Programming Languages. In ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS 2019). [18] Free Software Foundation, Inc. 2024. Instrumentation Options (Using the GNU Compiler Collection (GCC)). Free Software Foundation, Inc. https://gcc.gnu.org/ onlinedocs/gcc/Instrumentation-Options.html Accessed: 2025-01-13. [19] Matthias Gelbmann. 2017. Nginx reaches 33.3% web server market share while Apache falls below 50%. W3Techs (2017). [20] Google. 2025. Go Garbage Collector Guide. https://go.dev/doc/gc-guide. Accessed: 2025-10-07.

[21] Google. 2025. The Go Programming Language Specification. https://go.dev/ref/ spec. Accessed: 2025-10-07. [22] Google DeepMind. 2026. Gemini 3 Flash. https://gemini.google.com/. [23] Mandeep Signh Grang and Katherine Kjeer. 2020. Checked C: Adding Memory Safety to LLVM. https://llvm.org/devmtg/2020-09/slides/Checked_C_Adding_ Memory_Safety_to_LLVM.pdf. [24] Open Source Security Foundation (OpenSSF) Best Practices Working Group. [n. d.]. Compiler Options Hardening Guide for C and C++. https://best.openssf.org/Compiler-Hardening-Guides/Compiler-OptionsHardening-Guide-for-C-and-C++.html [25] Niklas Heer. 2023. Speed Comparison. https://niklas-heer.github.io/speedcomparison/ [26] Mohammad Hosseini, David B Resnik, and Kristi Holmes. 2023. The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts. Research ethics 19, 4 (Oct. 2023), 449–465. doi:10.1177/17470161231180449 [27] John Leitch. [n. d.]. Issue 24613: array.fromstring Use After Free - Python tracker — bugs.python.org. https://bugs.python.org/issue24613. [Accessed 29-07-2025]. [28] Aravind Machiry, John Kastner, Matt McCutchen, Aaron Eline, Kyle Headley, and Michael Hicks. 2022. C to checked C by 3c. Proc. ACM Program. Lang. 6, OOPSLA1, Article 78 (April 2022), 29 pages. doi:10.1145/3527322 [29] H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50 – 60. doi:10.1214/aoms/1177730491 [30] Microsoft. 2014. Lesson 2 - Windows NT System Overview. https://learn. microsoft.com/en-us/previous-versions//cc767881(v=technet.10) [31] Microsoft. 2022. Checked C. https://github.com/microsoft/checkedc. [32] MITRE. [n. d.]. CWE - 2024 CWE Top 25 Most Dangerous Software Weaknesses. https://cwe.mitre.org/top25/archive/2024/2024_cwe_top25.html [33] National Institute of Standards and Technology. 2025. National Vulnerability Database (NVD) API. https://nvd.nist.gov/developers/vulnerabilities. Programmatic query for CWE-120, CWE-126, CWE-416, CWE-415, CWE-457, and CWE-476 via REST API (2.0); data retrieved 2023-10-04. [34] Android Open Source Project. 2026. https://source.android.com/docs/security/ test/hwasan [35] Android Open Source Project. 2026. https://source.android.com/docs/security/ test/asan [36] PWNABLE.KR. 2014. Sh3ll we play a game? https://pwnable.kr. [Accessed 27-07-2025]. [37] PWNABLE.TW. [n. d.]. Pwnable.tw — pwnable.tw. https://pwnable.tw/challenge/. [Accessed 26-07-2025]. [38] Redox Developers. 2015. Redox OS. https://www.redox-os.org/ [39] Guangyong Shang, Guangpeng Qi, Jianing Ren, Xianqi Jin, Wanjiang Shen, Junchao Li, Xiuzhen Cheng, and Runyu Pan. 2025. LwRustIP: Memory-safe and efficient embedded networking stack with ownership semantics. High-Confidence Computing (2025), 100351. doi:10.1016/j.hcc.2025.100351 [40] S. S. SHAPIRO and M. B. WILK. 1965. An analysis of variance test for normality (complete samples)†. Biometrika 52, 3-4 (12 1965), 591– 611. arXiv:https://academic.oup.com/biomet/article-pdf/52/3-4/591/962907/52-34-591.pdf doi:10.1093/biomet/52.3-4.591 [41] Sam Silvestro, Hongyu Liu, Corey Crosser, Zhiqiang Lin, and Tongping Liu. 2017. FreeGuard: A Faster Secure Heap Allocator. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). 2389–2403. [42] Brad Spengler. [n. d.]. The Guaranteed End of Arbitrary Code Execution. https: //grsecurity.net/PaX-presentation.pdf. [Accessed 30-07-2025]. [43] GitHub Staff. 2024. Octoverse: AI leads Python to top language as the number of global developers surges. https://github.blog/news-insights/octoverse/ octoverse-2024/ [44] David Tarditi, Archibald Samuel Elliott, Andrew Ruef, and Michael Hicks. 2018. Checked C: Making C Safe by Extension. In IEEE Cybersecurity Development Conference 2018 (SecDev). IEEE, 53–60. https://www.microsoft.com/enus/research/publication/checkedc-making-c-safe-by-extension/ [45] The Clang Team. [n. d.]. Hardware-assisted AddressSanitizer Design Documentation - Clang 21.0.0git. https://clang.llvm.org/docs/ HardwareAssistedAddressSanitizerDesign.html [46] The Computer Language Benchmark Games. 2022. Measure "Which programming language is fastest?". https://benchmarksgame-team.pages.debian.net/ benchmarksgame/index.html [47] The Rust Core Team. 2015. Announcing Rust 1.0. https://blog.rust-lang.org/ 2015/05/15/Rust-1.0.html [48] USENIX Association. 2026. USENIX Security Ethics Policy. https://www.usenix. org/conference/usenixsecurity26/call-for-papers#ethics. Adopted by ACM CCS 2026. [49] Victor van der Veen, Nitish dutt Sharma, Lorenzo Cavallaro, and Herbert Bos. 2012. Memory Errors: The Past, the Present, and the Future. In Research in Attacks, Intrusions, and Defenses, Davide Balzarotti, Salvatore J. Stolfo, and Marco Cova (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 86–106.

Laramore, Spracklen, and Jadliwala

[50] Brian Wickman, Hong Hu, Insu Yun, DaeHee Jang, JungWon Lim, Sanidhya Kashyap, and Taesoo Kim. 2021. Preventing Use-After-Free Attacks with Fast Forward Allocation. In 30th USENIX Security Symposium (USENIX Security 21). 2453–2470. [51] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. http://www.jstor.org/stable/3001968

A

Open Science

To support the transparency and reproducibility of our findings, we provide the complete benchmarking framework and datasets used in this study. The artifacts include: (1) Automated Benchmarking Framework: Includes cbench.py, and benchmark.py used to orchestrate the build and execution of compounding defenses. (2) Benchmark Task Suite: The complete source code for the nine benchmark tasks taken from the Computer Language Benchmark Games. The dataset for running the knucleotide benchmarks was compressed to knucleotide-input25000 000.txt.zstd using the zstd compression tool to fit within Github’s 100 megabyte limit. The file knucleotide-input1 000.txt is available as a toy example without compression. (3) Analysis and Visualization Scripts: Python scripts (including dataframe.py and shapiro.py) used to process raw telemetry and generate the statistical analyses in Section 6. (4) Replication Environment: A Dockerfile and configuration scripts that define the exact compiler toolchain and library versions used for our evaluation. (5) Raw Experimental Data: The full dataset of performance and security metrics generated during our evaluation in results. These artifacts are publicly available at the following URL: https://github.com/red-avalanche/compounding-defenses-evaluation (commit 59411c3f9e254738bed66a48cfff0241daf83377) The repository for the forked ffmalloc, ffmalloc-16k, is available at: https://github.com/red-avalanche/ffmalloc-16k (commit 9661ccacf9ace0ffcae3071f9e74249cadd8738c) No artifacts have been withheld.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

B

Mod Map Definitions

The combinations of various mechanisms are described using "mod maps" or Modification Maps. Resembling bitmaps, the mod maps had six binary digits each representing a set of compiler flags (specifics of the each shown in section C). The mod maps are as follows: [0, 0, 0, 0, 0, 0] - Control group (original compiler settings from benchmark games used) [1, 0, 0, 0, 0, 0] - OpenSSF tl;dr [0, 1, 0, 0, 0, 0] - OpenSSF Production [0, 0, 1, 0, 0, 0] - Memory Sanitizer ASAN [0, 0, 0, 1, 0, 0] - Memory Sanitizer HWASAN [0, 0, 0, 0, 1, 0] - Memory Allocator FFMalloc [0, 0, 0, 0, 0, 1] - Memory Allocator MarkUs When the benchmarks were conducted all test samples had either one or multiple mechanisms enabled as an addition to the original compiler settings from the benchmark games. Which mechanisms were enabled are described with these mod maps. Notes on the some patterns of mod maps: • OpenSSF tl;dr is a subset or OpenSSF Production. All mod maps with OpenSSF will have either one or both digits enabled. The production flags are not enabled without the tl;dr flags. • Only one memory sanitizer can be enabled at a time. They are two different methods of achieving the same end goal so there are no benchmarks where they are both enabled. • Only one memory allocator can be enabled at a time. They use two different methods of allocating memory and both replace the malloc() function present in the standard library (glibc for Asahi). There are no benchmarks where they are both enabled.

C

Compiler Flags Provided

Laramore, Spracklen, and Jadliwala

Compiler Flags Provided OpenSSF

tl;dr

Production

-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -D_GLIBCXX_ASSERTIONS -fstrict-flex-arrays=3 -fstack-clash-protection -fstack-protector-strong -Wl,-z,nodlopen -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,–as-needed -Wl,–no-copy-dt-needed-entries -fPIE -pie -fno-delete-null-pointer-checks -fno-strict-overflow tl;dr + -fno-strict-aliasing -ftrivial-auto-var-init=zero

Memory Sanitize ASAN

HWASAN

-L/usr/lib/gcc/aarch64-redhat-linux/14 -fsanitize=address -fsanitize-recover=all -static-libasan -L/usr/lib/gcc/aarch64-redhat-linux/14 -fsanitize=hwaddress -fsanitize-recover=all

Malloc FFMalloc (single thread)

FFMalloc (multi thread)

MarkUs

-L/usr/local/lib/ffmalloc -l:libffmallocnpst.so -Wl,-rpath=/usr/local/lib/ffmalloc -L/usr/local/lib/ffmalloc -l:libffmallocnpmt.so -Wl,-rpath=/usr/local/lib/ffmalloc -L/usr/local/lib/markus-allocator/lib -I /usr/local/lib/markus-allocator/include -l:libgc.so -l:libgccpp.so -Wl,-rpath-link,/usr/lib64

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

D Median Runtime Tables D.1 By mod_map including Rust and Go

binarytrees Median Durations mod_map

time_as_duration

[1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0]

2s 32861102ns 2s 49840569ns 2s 79303898ns 2s 130858809ns 2s 268153184ns 2s 333344353ns 2s 340136279ns 2s 370944604ns 2s 389668825ns 2s 437078371ns 2s 458268465ns 2s 492526755ns 2s 512938441ns 2s 535197836ns 2s 696562207ns 3s 107707060ns 3s 143455294ns 3s 393016507ns 3s 891049580ns 4s 213404207ns 4s 224625004ns 4s 258080924ns 4s 286361933ns 4s 289112387ns 4s 313419616ns 4s 381562379ns 4s 443435317ns

%_difference 70.744 70.015 68.76 66.591 60.982 58.401 58.135 56.936 56.212 54.398 53.595 52.308 51.547 50.723 44.906 31.236 30.119 22.613 9.0079 1.0547 0.78878 0.0 0.66197 0.72613 1.2912 2.8585 4.2603

fannkuchredux Median Durations mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1]

7s 716126922ns 23s 293798556ns 23s 353293507ns 23s 454567830ns 23s 455147355ns 23s 498811897ns 23s 586171255ns 23s 596873781ns 23s 600597766ns 27s 543939175ns 27s 683255040ns 27s 927590301ns 27s 928904639ns 28s 11496459ns 28s 217245161ns 29s 121002594ns 29s 494527037ns 30s 150331007ns 38s 407972725ns 38s 453165545ns 38s 526122665ns 39s 483923375ns 39s 712817557ns 39s 719572170ns 40s 291480631ns 40s 493011462ns 40s 576456760ns

0.0 100.47 100.66 100.98 100.98 101.12 101.4 101.43 101.44 112.47 112.81 113.41 113.41 113.61 114.11 116.21 117.05 118.49 133.08 133.15 133.25 134.61 134.92 134.93 135.71 135.98 136.09

Laramore, Spracklen, and Jadliwala

fasta Median Durations mod_map

time_as_duration

[1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0]

998396639ns 1s 2969940ns 1s 18738831ns 1s 19313116ns 1s 24363481ns 1s 29410536ns 1s 32781673ns 1s 59855670ns 1s 487245716ns 1s 599325078ns 1s 600811455ns 1s 667635260ns 1s 710305901ns 1s 746504903ns 1s 822675943ns 1s 888435437ns 2s 57137469ns 2s 263212505ns 3s 440671842ns 3s 466964829ns 3s 483451708ns 3s 488201486ns 3s 839671787ns 3s 883711551ns 4s 301688552ns 4s 443246207ns 4s 581203933ns

knucleotide Median Durations %_difference 39.334 38.894 37.391 37.337 36.859 36.384 36.068 33.559 0.0 7.2624 7.3552 11.436 13.952 16.035 20.268 23.769 32.157 41.38 79.28 79.921 80.319 80.433 88.322 89.238 97.235 99.688 101.97

mod_map

time_as_duration

[1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0]

1s 829948135ns 1s 838519189ns 1s 858599369ns 1s 862534748ns 1s 890772899ns 1s 898015128ns 2s 77741974ns 2s 80159397ns 3s 52112662ns 3s 537135748ns 3s 714023788ns 3s 878621573ns 3s 988719588ns 4s 51464255ns 4s 101552056ns 4s 116309892ns 4s 161069662ns 4s 250337017ns 6s 395152534ns 6s 420811965ns 6s 565219855ns 6s 583466351ns 6s 597831592ns 6s 646750275ns 7s 147984203ns 7s 234202114ns 7s 289109294ns

%_difference 50.068 49.629 48.609 48.41 46.99 46.629 37.988 37.876 0.0 14.722 19.565 23.851 26.605 28.137 29.34 29.691 30.748 32.817 70.773 71.123 73.058 73.298 73.487 74.125 80.31 81.314 81.944

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

mandelbrot Median Durations mod_map

time_as_duration

[0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0]

1s 372407075ns 1s 393161897ns 1s 407514057ns 1s 443017492ns 1s 452841981ns 1s 481397848ns 1s 508760576ns 1s 515155776ns 2s 9994371ns 2s 286067276ns 2s 299822659ns 2s 345449350ns 2s 360442193ns 2s 379142550ns 2s 404482069ns 2s 413834285ns 2s 430611386ns 3s 227565390ns 7s 850678912ns 7s 867542604ns 7s 877854707ns 7s 902111548ns 8s 21473295ns 8s 504150490ns 8s 512217793ns 8s 531445517ns 8s 556567954ns

nbody Median Durations

%_difference 80.66 79.399 78.534 76.417 75.836 74.164 72.58 72.212 46.494 34.152 33.569 31.657 31.035 30.265 29.229 28.849 28.17 0.0 83.463 83.64 83.748 84.001 85.232 89.954 90.03 90.21 90.444

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

2s 181703329ns 2s 328540828ns 2s 330787495ns 2s 347674287ns 2s 349521825ns 2s 356721997ns 3s 332593609ns 3s 354073712ns 3s 389477337ns 3s 412999738ns 3s 578430510ns 3s 863110869ns 3s 997347153ns 4s 32422369ns 4s 148217041ns 4s 189969306ns 4s 206688380ns 4s 254183789ns 7s 222710208ns 7s 230657368ns 7s 236632517ns 7s 238125354ns 7s 248889319ns 7s 268065019ns 7s 269158871ns 7s 276286200ns 7s 297401433ns

%_difference 0.0 6.5113 6.6076 7.3286 7.4072 7.7127 41.742 42.356 43.358 44.016 48.496 55.631 58.768 59.565 62.134 63.037 63.396 64.404 107.21 107.28 107.34 107.36 107.46 107.65 107.66 107.73 107.94

Laramore, Spracklen, and Jadliwala

pidigits Median Durations mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0]

381553778ns 383419608ns 388122005ns 397716155ns 460939768ns 526278898ns 561323987ns 584525213ns 634471212ns 699532269ns 713454343ns 740282575ns 740663350ns 747753456ns 773971333ns 779993970ns 785086581ns 789269493ns 902110904ns 904474342ns 909519325ns 917025782ns 919682564ns 920576343ns 1s 183110673ns 1s 184638125ns 1s 255696635ns

regexredux Median Durations %_difference 42.02 41.553 40.385 38.037 23.642 10.487 4.0498 0.0 8.1946 17.913 19.866 23.514 23.565 24.504 27.891 28.65 29.287 29.807 42.725 42.975 43.505 44.288 44.563 44.655 67.727 67.841 72.945

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 0, 1, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0]

858411278ns 905870468ns 912055543ns 925373857ns 944552726ns 945778246ns 947677190ns 950754267ns 969658660ns 977405338ns 1s 25465685ns 1s 54316078ns 1s 56256779ns 1s 121015496ns 1s 164709239ns 1s 170492119ns 1s 200109273ns 1s 207429959ns 1s 210736573ns 1s 266543323ns 1s 324648411ns 1s 356042980ns 1s 374394155ns 1s 393703246ns 1s 418592773ns 1s 524932749ns 1s 581160950ns

%_difference 10.208 4.835 4.1549 2.7057 0.65446 0.52475 0.32416 0.0 1.9687 2.7644 7.561 10.33 10.513 16.436 20.228 20.718 23.187 23.786 24.056 28.484 32.864 35.139 36.44 37.787 39.491 46.385 49.797

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

spectralnorm Median Durations mod_map [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0]

time_as_duration 308246580ns 311494232ns 317158343ns 328716001ns 337458389ns 342899642ns 343479043ns 346147126ns 350465804ns 352857345ns 353096263ns 355705622ns 359260899ns 364807555ns 374396822ns 384178648ns 387803137ns 405966248ns 585031608ns 633501678ns 650040817ns 650556592ns 655668970ns 680693910ns 680977303ns 685844156ns 694538396ns

D.2

By mod_map excluding Rust and Go

%_difference 0.0 1.0482 2.85 6.4274 9.0481 10.644 10.812 11.584 12.819 13.496 13.563 14.296 15.285 16.807 19.381 21.932 22.86 27.364 61.971 69.075 71.334 71.404 72.086 75.323 75.358 75.969 77.044

binarytrees Median Durations Only C programs shown

mod_map

time_as_duration

[1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0]

2s 32861102ns 2s 49840569ns 2s 79303898ns 2s 130858809ns 2s 268153184ns 2s 333344353ns 2s 340136279ns 2s 370944604ns 2s 389668825ns 2s 437078371ns 2s 458268465ns 2s 492526755ns 2s 512938441ns 2s 528213589ns 2s 535197836ns 2s 696562207ns 3s 107707060ns 3s 143455294ns 3s 393016507ns 3s 891049580ns 4s 213404207ns 4s 224625004ns 4s 286361933ns 4s 289112387ns 4s 313419616ns 4s 381562379ns 4s 443435317ns

%_difference 21.721 20.899 19.486 17.057 10.844 8.0167 7.7265 6.4202 5.6343 3.6709 2.8054 1.4216 0.60601 0.0 0.27586 6.4443 20.564 21.695 29.21 42.461 49.994 50.243 51.6 51.66 52.187 53.644 54.943

Laramore, Spracklen, and Jadliwala

fannkuchredux Median Durations

fasta Median Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1]

20s 852956479ns 23s 293798556ns 23s 353293507ns 23s 454567830ns 23s 455147355ns 23s 498811897ns 23s 586171255ns 23s 596873781ns 23s 600597766ns 27s 543939175ns 27s 683255040ns 27s 927590301ns 27s 928904639ns 28s 11496459ns 28s 217245161ns 29s 121002594ns 29s 494527037ns 30s 150331007ns 38s 407972725ns 38s 453165545ns 38s 526122665ns 39s 483923375ns 39s 712817557ns 39s 719572170ns 40s 291480631ns 40s 493011462ns 40s 576456760ns

0.0 11.058 11.312 11.743 11.746 11.931 12.301 12.346 12.362 27.65 28.145 29.006 29.011 29.3 30.015 33.089 34.328 36.458 59.247 59.354 59.527 61.756 62.279 62.294 63.582 64.03 64.215

mod_map

time_as_duration

[1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0]

998396639ns 1s 2969940ns 1s 18738831ns 1s 19313116ns 1s 24363481ns 1s 29410536ns 1s 32781673ns 1s 59855670ns 1s 599325078ns 1s 600811455ns 1s 667635260ns 1s 710305901ns 1s 720479643ns 1s 746504903ns 1s 822675943ns 1s 888435437ns 2s 57137469ns 2s 263212505ns 3s 440671842ns 3s 466964829ns 3s 483451708ns 3s 488201486ns 3s 839671787ns 3s 883711551ns 4s 301688552ns 4s 443246207ns 4s 581203933ns

%_difference 53.116 52.691 51.237 51.184 50.722 50.262 49.955 47.521 7.2989 7.2061 3.1194 0.5931 0.0 1.5013 5.7686 9.3078 17.824 27.248 66.659 67.335 67.755 67.876 76.228 77.201 85.724 88.348 90.792

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

knucleotide Median Durations

mandelbrot Median Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0]

1s 829948135ns 1s 838519189ns 1s 858599369ns 1s 862534748ns 1s 890772899ns 1s 898015128ns 1s 932941501ns 2s 77741974ns 2s 80159397ns 3s 537135748ns 3s 714023788ns 3s 878621573ns 3s 988719588ns 4s 51464255ns 4s 101552056ns 4s 116309892ns 4s 161069662ns 4s 250337017ns 6s 395152534ns 6s 420811965ns 6s 565219855ns 6s 583466351ns 6s 597831592ns 6s 646750275ns 7s 147984203ns 7s 234202114ns 7s 289109294ns

%_difference 5.4741 5.0072 3.9215 3.7101 2.2057 1.8234 0.0 7.2207 7.3369 58.653 63.081 66.959 69.432 70.801 71.874 72.186 73.125 74.957 107.16 107.45 109.02 109.21 109.37 109.88 114.86 115.66 116.16

mod_map

time_as_duration

[0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0]

1s 372407075ns 1s 375989431ns 1s 393161897ns 1s 407514057ns 1s 443017492ns 1s 452841981ns 1s 481397848ns 1s 508760576ns 1s 515155776ns 2s 9994371ns 2s 286067276ns 2s 299822659ns 2s 345449350ns 2s 360442193ns 2s 379142550ns 2s 404482069ns 2s 413834285ns 2s 430611386ns 7s 850678912ns 7s 867542604ns 7s 877854707ns 7s 902111548ns 8s 21473295ns 8s 504150490ns 8s 512217793ns 8s 531445517ns 8s 556567954ns

%_difference 0.26066 0.0 1.2402 2.2651 4.7554 5.4335 7.3779 9.205 9.6271 37.449 49.703 50.266 52.101 52.695 53.428 54.411 54.77 55.41 140.35 140.46 140.52 140.68 141.43 144.29 144.34 144.45 144.59

Laramore, Spracklen, and Jadliwala

nbody Median Durations

pidigits Median Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

2s 50706612ns 2s 328540828ns 2s 330787495ns 2s 347674287ns 2s 349521825ns 2s 356721997ns 3s 332593609ns 3s 354073712ns 3s 389477337ns 3s 412999738ns 3s 578430510ns 3s 863110869ns 3s 997347153ns 4s 32422369ns 4s 148217041ns 4s 189969306ns 4s 206688380ns 4s 254183789ns 7s 222710208ns 7s 230657368ns 7s 236632517ns 7s 238125354ns 7s 248889319ns 7s 268065019ns 7s 269158871ns 7s 276286200ns 7s 297401433ns

%_difference 0.0 12.689 12.785 13.504 13.582 13.886 47.625 48.23 49.218 49.867 54.279 61.294 64.372 65.155 67.673 68.559 68.91 69.897 111.54 111.62 111.68 111.69 111.79 111.98 111.99 112.05 112.25

mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0]

381553778ns 383419608ns 388122005ns 397716155ns 460939768ns 518926285ns 526278898ns 561323987ns 634471212ns 699532269ns 713454343ns 740282575ns 740663350ns 747753456ns 773971333ns 779993970ns 785086581ns 789269493ns 902110904ns 904474342ns 909519325ns 917025782ns 919682564ns 920576343ns 1s 183110673ns 1s 184638125ns 1s 255696635ns

%_difference 30.511 30.034 28.842 26.447 11.836 0.0 1.4068 7.8495 20.036 29.645 31.569 35.158 35.208 36.13 39.453 40.198 40.822 41.331 53.93 54.173 54.688 55.447 55.714 55.804 78.046 78.155 83.034

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

regexredux Median Durations

spectralnorm Median Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 0, 1, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0]

858411278ns 905870468ns 912055543ns 925373857ns 944552726ns 945778246ns 947677190ns 969658660ns 977405338ns 1s 25465685ns 1s 54316078ns 1s 56256779ns 1s 121015496ns 1s 164709239ns 1s 170492119ns 1s 200109273ns 1s 207429959ns 1s 210736573ns 1s 266543323ns 1s 324648411ns 1s 338728526ns 1s 356042980ns 1s 374394155ns 1s 393703246ns 1s 418592773ns 1s 524932749ns 1s 581160950ns

%_difference 43.722 38.569 37.913 36.514 34.527 34.401 34.207 31.976 31.201 26.501 23.77 23.589 17.702 13.902 13.409 10.92 10.314 10.041 5.5415 1.0573 0.0 1.285 2.6291 4.0239 5.7929 13.005 16.606

mod_map [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0]

D.3

time_as_duration 311494232ns 317158343ns 318509350ns 328716001ns 337458389ns 342899642ns 343479043ns 346147126ns 350465804ns 352857345ns 353096263ns 355705622ns 359260899ns 364807555ns 374396822ns 384178648ns 387803137ns 405966248ns 585031608ns 633501678ns 650040817ns 650556592ns 655668970ns 680693910ns 680977303ns 685844156ns 694538396ns

%_difference 2.227 0.42507 0.0 3.1541 5.7774 7.3752 7.5439 8.3165 9.5537 10.232 10.3 11.034 12.025 13.551 16.131 18.691 19.621 24.144 58.995 66.174 68.459 68.529 69.219 72.495 72.531 73.149 74.237

By compiler binarytrees Compiler Median Durations

lang

time_as_duration

go rust gcc clang checked-c

14s 283967536ns 631295242ns 2s 644193388ns 2s 748777936ns 8s 336653050ns

Laramore, Spracklen, and Jadliwala

fannkuchredux Compiler Median Durations

nbody Compiler Median Durations

lang

time_as_duration

lang

time_as_duration

go rust gcc clang checked-c

7s 634097602ns 6s 398731996ns 28s 840526586ns 26s 904184203ns 23s 464719732ns

go rust gcc clang checked-c

2s 583717322ns 2s 262851870ns 3s 303322607ns 4s 927316907ns 3s 294935837ns

pidigits Compiler Median Durations fasta Compiler Median Durations lang

time_as_duration

go rust gcc clang checked-c

1s 409343499ns 1s 199237012ns 2s 623590185ns 1s 17638204ns 1s 27606116ns

knucleotide Compiler Median Durations lang

time_as_duration

go rust gcc clang

3s 125889378ns 2s 317144819ns 3s 267298147ns 4s 23158505ns

mandelbrot Compiler Median Durations lang

time_as_duration

go rust gcc clang checked-c

4s 218654939ns 1s 247961384ns 2s 558363988ns 1s 91177433ns 14s 109587824ns

lang

time_as_duration

go rust gcc clang checked-c

686385733ns 578283342ns 714030256ns 696200911ns 387879231ns

regexredux Compiler Median Durations lang

time_as_duration

go rust gcc clang checked-c

16s 90826858ns 701207778ns 982293121ns 1s 171373332ns 2s 596073753ns

spectralnorm Compiler Median Durations lang

time_as_duration

go rust gcc clang

532665053ns 261998099ns 329510217ns 511795787ns

E Mean Runtime Tables E.1 By mod_map including Rust and Go Mean values shown for completeness and to capture the impact of performance outliers that median values might obscure; see Section D for median values which better represent non-normal distributions

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

binarytrees Average Durations

fannkuchredux Average Durations

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0]

4s 685017200ns 4s 873680474ns 5s 20459841ns 5s 36036719ns 5s 36333400ns 5s 43961317ns 5s 74913229ns 5s 98599682ns 6s 707310652ns 8s 665667154ns 8s 716940187ns 8s 818071722ns 9s 65903462ns 9s 71700833ns 9s 95883050ns 9s 112743143ns 9s 115974842ns 9s 221413116ns 14s 369719556ns 14s 419231091ns 14s 459510817ns 14s 519146620ns 14s 593761701ns 14s 677284952ns 14s 867354551ns 14s 931657656ns 15s 58214189ns

35.503 31.666 28.767 28.463 28.458 28.309 27.709 27.253 0.0 25.478 26.058 27.191 29.906 29.969 30.229 30.41 30.445 31.567 72.709 73.007 73.249 73.605 74.047 74.539 75.645 76.014 76.735

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0]

13s 696575241ns 19s 383956654ns 19s 450108956ns 19s 610149222ns 19s 638672685ns 19s 697723256ns 19s 731275513ns 19s 862497521ns 19s 988924145ns 26s 281667949ns 26s 459816181ns 26s 647459956ns 26s 716815089ns 26s 983284357ns 27s 92784966ns 27s 93777592ns 27s 369332365ns 27s 381995385ns 42s 970915867ns 42s 981672388ns 43s 66179108ns 43s 100639582ns 43s 167234945ns 43s 479050501ns 43s 484370078ns 44s 141222769ns 44s 619558539ns

0.0 34.385 34.716 35.51 35.651 35.941 36.106 36.747 37.359 62.96 63.568 64.202 64.435 65.323 65.685 65.688 66.589 66.63 103.32 103.34 103.48 103.54 103.65 104.18 104.19 105.28 106.05

Laramore, Spracklen, and Jadliwala

fasta Average Durations

knucleotide Average Durations

mod_map

time_as_duration

%_difference

[1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [1, 1, 0, 0, 1, 0] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0]

1s 956845793ns 1s 959510825ns 2s 1751097ns 2s 5618887ns 2s 8453615ns 2s 9876256ns 2s 304866298ns 2s 328782719ns 2s 344695232ns 2s 350664177ns 2s 354141890ns 2s 381314519ns 2s 382428369ns 2s 384970926ns 2s 390294µs 2s 434370850ns 2s 454421324ns 3s 193908211ns 3s 232022025ns 3s 256887756ns 3s 349352207ns 3s 350476576ns 3s 355425443ns 5s 770469328ns 15s 143566537ns 16s 914923863ns 16s 935585907ns

98.705 98.602 96.979 96.832 96.723 96.669 85.832 84.988 84.429 84.22 84.098 83.151 83.113 83.024 82.84 81.32 80.634 57.484 56.394 55.688 53.096 53.065 52.927 0.0 89.635 98.252 98.345

mod_map

time_as_duration

[1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 0, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1]

1s 917242154ns 1s 918771898ns 1s 922099484ns 1s 933371271ns 1s 937965793ns 1s 946218552ns 2s 193290601ns 2s 204447820ns 3s 691486572ns 3s 709516572ns 3s 724033328ns 3s 762961914ns 4s 6045906ns 4s 23554644ns 4s 124417509ns 4s 184478371ns 4s 188392395ns 4s 201140879ns 6s 760674730ns 6s 772429242ns 7s 131892522ns 7s 158187426ns 7s 173903007ns 7s 372111781ns 7s 375867317ns 7s 654846437ns 7s 741399756ns

%_difference 63.705 63.634 63.478 62.952 62.738 62.354 51.373 50.899 0.48723 0.0 0.39058 1.4304 7.6865 8.1219 10.592 12.034 12.127 12.429 58.283 58.442 63.135 63.466 63.664 66.102 66.147 69.433 70.42

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

mandelbrot Average Durations

nbody Average Durations

mod_map

time_as_duration

%_difference

[0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [0, 0, 0, 0, 0, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0]

4s 652852067ns 4s 686362885ns 4s 730226558ns 4s 734445014ns 4s 735657864ns 4s 746097096ns 4s 808975939ns 5s 320003945ns 5s 339569940ns 5s 388498349ns 6s 391740012ns 6s 472589724ns 6s 480713414ns 6s 505926533ns 6s 544223467ns 6s 621050447ns 6s 704873933ns 6s 730225241ns 20s 903195817ns 21s 366083780ns 21s 466326425ns 21s 566285920ns 21s 605736951ns 21s 673001347ns 21s 721173675ns 21s 821766098ns 24s 439192985ns

13.745 13.03 12.102 12.014 11.988 11.769 10.457 0.36711 0.0 0.91217 17.938 19.184 19.308 19.693 20.274 21.428 22.671 23.044 118.61 120.02 120.32 120.62 120.73 120.93 121.07 121.37 128.28

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0]

2s 389480883ns 2s 597632345ns 2s 599709530ns 2s 604647976ns 2s 610326306ns 2s 612963243ns 3s 174700480ns 3s 199109426ns 3s 225555717ns 3s 750048517ns 3s 846627551ns 3s 869616088ns 3s 887819332ns 3s 913919119ns 3s 940683986ns 4s 446815586ns 4s 523887069ns 4s 524647484ns 7s 446025729ns 7s 475444718ns 7s 488405796ns 7s 789392015ns 7s 818579495ns 7s 818811909ns 7s 823060388ns 7s 823100815ns 7s 949154292ns

%_difference 0.0 8.3476 8.4274 8.6168 8.8342 8.935 28.224 28.974 29.78 44.322 46.733 47.296 47.738 48.369 49.01 60.189 61.747 61.762 102.82 103.11 103.24 106.1 106.37 106.37 106.41 106.41 107.55

Laramore, Spracklen, and Jadliwala

pidigits Average Durations mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

428322868ns 430918796ns 435358138ns 466127263ns 467087937ns 487747766ns 513390828ns 593567406ns 710951047ns 724877297ns 727375346ns 731897894ns 735778002ns 739279779ns 742851035ns 757120129ns 768641766ns 771717011ns 927614729ns 929332275ns 932793907ns 936024777ns 936124283ns 990088905ns 1s 65932595ns 1s 92030269ns 1s 112508550ns

regexredux Average Durations %_difference 52.331 51.767 50.81 44.368 44.172 40.036 35.093 20.873 2.9034 0.96377 0.61976 0.0 0.52886 1.0036 1.4855 3.3879 4.8974 5.2966 23.587 23.77 24.136 24.477 24.487 29.988 37.16 39.49 41.272

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0]

908626149ns 923210553ns 928924048ns 965021062ns 988720085ns 988806608ns 997552595ns 1s 39780290ns 1s 45182913ns 1s 73896972ns 1s 79047954ns 1s 90168290ns 1s 99012093ns 1s 127575332ns 1s 131861880ns 1s 144052616ns 1s 157094302ns 1s 161664819ns 1s 494943796ns 1s 508455494ns 1s 519839594ns 1s 521528983ns 1s 539629127ns 1s 546196341ns 1s 651190542ns 1s 671870876ns 4s 22096005ns

%_difference 126.29 125.33 124.95 122.6 121.07 121.07 120.51 117.83 117.5 115.71 115.39 114.7 114.16 112.42 112.16 111.42 110.64 110.36 91.613 90.9 90.303 90.214 89.27 88.929 83.581 82.551 0.0

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

spectralnorm Average Durations mod_map

time_as_duration

[1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 0, 0, 0]

368389749ns 384004540ns 385018348ns 401841547ns 403824208ns 421556439ns 427955643ns 433736134ns 436260797ns 439093292ns 449544147ns 450156440ns 493633137ns 501844547ns 513059535ns 513105020ns 516786673ns 559574707ns 583805650ns 584950883ns 588891950ns 596598292ns 631409293ns 634665195ns 697937382ns 723514884ns 1s 97922977ns

E.2

By mod_map excluding Rust and Go

%_difference 99.506 96.35 96.147 92.826 92.439 89.026 87.814 86.728 86.256 85.728 83.799 83.686 75.937 74.521 72.61 72.602 71.98 64.959 61.142 60.964 60.354 59.17 53.953 53.476 44.545 41.111 0.0

binarytrees Average Durations Only C programs shown

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0]

4s 434497371ns 4s 685017200ns 4s 873680474ns 5s 20459841ns 5s 36036719ns 5s 36333400ns 5s 43961317ns 5s 74913229ns 5s 98599682ns 8s 665667154ns 8s 716940187ns 8s 818071722ns 9s 65903462ns 9s 71700833ns 9s 95883050ns 9s 112743143ns 9s 115974842ns 9s 221413116ns 14s 369719556ns 14s 419231091ns 14s 459510817ns 14s 519146620ns 14s 593761701ns 14s 677284952ns 14s 867354551ns 14s 931657656ns 15s 58214189ns

0.0 5.4942 9.4365 12.395 12.703 12.709 12.86 13.469 13.933 64.597 65.125 66.154 68.611 68.668 68.903 69.066 69.097 70.108 105.67 105.92 106.12 106.41 106.78 107.19 108.1 108.41 109.0

Laramore, Spracklen, and Jadliwala

fannkuchredux Average Durations

fasta Average Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0]

18s 714158040ns 19s 383956654ns 19s 450108956ns 19s 610149222ns 19s 638672685ns 19s 697723256ns 19s 731275513ns 19s 862497521ns 19s 988924145ns 26s 281667949ns 26s 459816181ns 26s 647459956ns 26s 716815089ns 26s 983284357ns 27s 92784966ns 27s 93777592ns 27s 369332365ns 27s 381995385ns 42s 970915867ns 42s 981672388ns 43s 66179108ns 43s 100639582ns 43s 167234945ns 43s 479050501ns 43s 484370078ns 44s 141222769ns 44s 619558539ns

0.0 3.5162 3.8567 4.6758 4.8211 5.1212 5.2912 5.9535 6.5874 33.636 34.293 34.978 35.23 36.191 36.582 36.586 37.563 37.608 78.647 78.668 78.834 78.902 79.032 79.639 79.649 80.907 81.806

mod_map

time_as_duration

%_difference

[1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [1, 1, 0, 0, 1, 0] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0]

1s 956845793ns 1s 959510825ns 2s 1751097ns 2s 5618887ns 2s 8453615ns 2s 9876256ns 2s 108045635ns 2s 304866298ns 2s 328782719ns 2s 344695232ns 2s 350664177ns 2s 354141890ns 2s 381314519ns 2s 382428369ns 2s 384970926ns 2s 390294µs 2s 434370850ns 2s 454421324ns 3s 193908211ns 3s 232022025ns 3s 256887756ns 3s 349352207ns 3s 350476576ns 3s 355425443ns 15s 143566537ns 16s 914923863ns 16s 935585907ns

7.4393 7.3034 5.1727 4.9798 4.8387 4.7679 0.0 8.9202 9.9502 10.629 10.883 11.03 12.174 12.221 12.327 12.549 14.368 15.184 40.961 42.096 42.828 45.491 45.523 45.663 151.12 155.67 155.72

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

knucleotide Average Durations

mandelbrot Average Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1]

1s 909915227ns 1s 917242154ns 1s 918771898ns 1s 922099484ns 1s 933371271ns 1s 937965793ns 1s 946218552ns 2s 193290601ns 2s 204447820ns 3s 691486572ns 3s 724033328ns 3s 762961914ns 4s 6045906ns 4s 23554644ns 4s 124417509ns 4s 184478371ns 4s 188392395ns 4s 201140879ns 6s 760674730ns 6s 772429242ns 7s 131892522ns 7s 158187426ns 7s 173903007ns 7s 372111781ns 7s 375867317ns 7s 654846437ns 7s 741399756ns

%_difference 0.0 0.3829 0.46261 0.63591 1.2206 1.4579 1.8829 13.812 14.317 63.612 64.4 65.33 70.864 71.245 73.397 74.644 74.725 74.986 111.89 112.01 115.51 115.75 115.9 117.69 117.73 120.13 120.84

mod_map

time_as_duration

%_difference

[0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0]

4s 652852067ns 4s 686362885ns 4s 730226558ns 4s 734445014ns 4s 735657864ns 4s 746097096ns 4s 808975939ns 5s 320003945ns 5s 388498349ns 5s 986831690ns 6s 391740012ns 6s 472589724ns 6s 480713414ns 6s 505926533ns 6s 544223467ns 6s 621050447ns 6s 704873933ns 6s 730225241ns 20s 903195817ns 21s 366083780ns 21s 466326425ns 21s 566285920ns 21s 605736951ns 21s 673001347ns 21s 721173675ns 21s 821766098ns 24s 439192985ns

25.076 24.369 23.451 23.363 23.337 23.12 21.821 11.795 10.52 0.0 6.5421 7.7974 7.9227 8.3103 8.8962 10.061 11.315 11.691 110.94 112.45 112.77 113.09 113.21 113.42 113.57 113.89 121.29

Laramore, Spracklen, and Jadliwala

nbody Average Durations

pidigits Average Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0]

2s 100303419ns 2s 597632345ns 2s 599709530ns 2s 604647976ns 2s 610326306ns 2s 612963243ns 3s 174700480ns 3s 199109426ns 3s 225555717ns 3s 750048517ns 3s 846627551ns 3s 869616088ns 3s 887819332ns 3s 913919119ns 3s 940683986ns 4s 446815586ns 4s 523887069ns 4s 524647484ns 7s 446025729ns 7s 475444718ns 7s 488405796ns 7s 789392015ns 7s 818579495ns 7s 818811909ns 7s 823060388ns 7s 823100815ns 7s 949154292ns

%_difference 0.0 21.172 21.251 21.439 21.654 21.754 40.735 41.469 42.256 56.398 58.73 59.274 59.702 60.311 60.93 71.681 73.174 73.188 112.0 112.27 112.38 115.05 115.3 115.3 115.34 115.34 116.4

mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

428322868ns 430918796ns 435358138ns 466127263ns 467087937ns 487747766ns 513390828ns 519237322ns 593567406ns 710951047ns 724877297ns 727375346ns 735778002ns 739279779ns 742851035ns 757120129ns 768641766ns 771717011ns 927614729ns 929332275ns 932793907ns 936024777ns 936124283ns 990088905ns 1s 65932595ns 1s 92030269ns 1s 112508550ns

%_difference 19.189 18.59 17.574 10.78 10.575 6.2543 1.1325 0.0 13.359 31.168 33.058 33.393 34.508 34.968 35.436 37.275 38.731 39.115 56.45 56.621 56.962 57.28 57.29 62.392 68.976 71.099 72.716

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

regexredux Average Durations

spectralnorm Average Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0]

908626149ns 923210553ns 928924048ns 965021062ns 988720085ns 988806608ns 997552595ns 1s 39780290ns 1s 45182913ns 1s 73896972ns 1s 79047954ns 1s 90168290ns 1s 99012093ns 1s 127575332ns 1s 131861880ns 1s 144052616ns 1s 157094302ns 1s 161664819ns 1s 465295199ns 1s 494943796ns 1s 508455494ns 1s 519839594ns 1s 521528983ns 1s 539629127ns 1s 546196341ns 1s 651190542ns 1s 671870876ns

%_difference 46.899 45.391 44.806 41.169 38.84 38.832 37.984 33.972 33.469 30.829 30.361 29.359 28.568 26.05 25.677 24.622 23.505 23.117 0.0 2.0031 2.9027 3.6544 3.7654 4.9475 5.3728 11.93 13.17

mod_map [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0]

E.3

time_as_duration

%_difference

317049296ns 368389749ns 384004540ns 385018348ns 401841547ns 403824208ns 421556439ns 427955643ns 433736134ns 436260797ns 439093292ns 449544147ns 450156440ns 493633137ns 501844547ns 513059535ns 513105020ns 516786673ns 559574707ns 583805650ns 584950883ns 588891950ns 596598292ns 631409293ns 634665195ns 697937382ns 723514884ns

0.0 14.98 19.101 19.363 23.59 24.075 28.299 29.773 31.084 31.65 32.281 34.567 34.699 43.564 45.133 47.225 47.234 47.908 55.332 59.223 59.402 60.013 61.194 66.289 66.746 75.053 78.124

By compiler binarytrees Compiler Mean Durations

lang

time_as_duration

go rust gcc clang checked-c

13s 231707860ns 696284628ns 9s 592546578ns 8s 506177192ns 7s 363063525ns

Laramore, Spracklen, and Jadliwala

fannkuchredux Compiler Mean Durations

nbody Compiler Mean Durations

lang

time_as_duration

lang

time_as_duration

go rust gcc clang checked-c

12s 730773182ns 8s 294200923ns 32s 113712335ns 25s 93407091ns 21s 475742842ns

go rust gcc clang checked-c

2s 570698558ns 2s 509650319ns 4s 637346804ns 4s 915890172ns 3s 340507594ns

pidigits Compiler Mean Durations

fasta Compiler Mean Durations lang

time_as_duration

go rust gcc clang checked-c

25s 378070706ns 1s 622239543ns 4s 221502945ns 1s 892301796ns 2s 145090742ns

lang

time_as_duration

go rust gcc clang checked-c

875481155ns 586505087ns 798015473ns 653201527ns 427023268ns

regexredux Compiler Mean Durations knucleotide Compiler Mean Durations lang

time_as_duration

go rust gcc clang

3s 258889358ns 4s 610044121ns 4s 170514860ns 3s 556927297ns

lang

time_as_duration

go rust gcc clang checked-c

14s 45691872ns 714831943ns 1s 173606681ns 1s 341165723ns 2s 561424756ns

spectralnorm Compiler Mean Durations

mandelbrot Compiler Mean Durations lang

time_as_duration

lang

time_as_duration

go rust gcc clang checked-c

5s 677370907ns 1s 370978892ns 17s 7296929ns 4s 699170884ns 14s 153630340ns

go rust gcc clang

2s 176211360ns 261364084ns 416878293ns 571815623ns

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

fannkuchredux Min Durations

F Minimum Runtime Tables F.1 By mod_map including Rust and Go

binarytrees Min Durations mod_map

time_as_duration

[1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0]

2s 32861102ns 2s 49840569ns 2s 79303898ns 2s 130858809ns 2s 268153184ns 2s 333344353ns 2s 340136279ns 2s 370944604ns 2s 389668825ns 2s 437078371ns 2s 458268465ns 2s 492526755ns 2s 512938441ns 2s 535197836ns 2s 696562207ns 3s 107707060ns 3s 143455294ns 3s 393016507ns 3s 891049580ns 4s 213404207ns 4s 224625004ns 4s 258080924ns 4s 286361933ns 4s 289112387ns 4s 313419616ns 4s 381562379ns 4s 443435317ns

%_difference 70.744 70.015 68.76 66.591 60.982 58.401 58.135 56.936 56.212 54.398 53.595 52.308 51.547 50.723 44.906 31.236 30.119 22.613 9.0079 1.0547 0.78878 0.0 0.66197 0.72613 1.2912 2.8585 4.2603

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1]

7s 716126922ns 23s 293798556ns 23s 353293507ns 23s 454567830ns 23s 455147355ns 23s 498811897ns 23s 586171255ns 23s 596873781ns 23s 600597766ns 27s 543939175ns 27s 683255040ns 27s 927590301ns 27s 928904639ns 28s 11496459ns 28s 217245161ns 29s 121002594ns 29s 494527037ns 30s 150331007ns 38s 407972725ns 38s 453165545ns 38s 526122665ns 39s 483923375ns 39s 712817557ns 39s 719572170ns 40s 291480631ns 40s 493011462ns 40s 576456760ns

0.0 100.47 100.66 100.98 100.98 101.12 101.4 101.43 101.44 112.47 112.81 113.41 113.41 113.61 114.11 116.21 117.05 118.49 133.08 133.15 133.25 134.61 134.92 134.93 135.71 135.98 136.09

Laramore, Spracklen, and Jadliwala

fasta Min Durations mod_map

time_as_duration

[1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0]

998396639ns 1s 2969940ns 1s 18738831ns 1s 19313116ns 1s 24363481ns 1s 29410536ns 1s 32781673ns 1s 59855670ns 1s 487245716ns 1s 599325078ns 1s 600811455ns 1s 667635260ns 1s 710305901ns 1s 746504903ns 1s 822675943ns 1s 888435437ns 2s 57137469ns 2s 263212505ns 3s 440671842ns 3s 466964829ns 3s 483451708ns 3s 488201486ns 3s 839671787ns 3s 883711551ns 4s 301688552ns 4s 443246207ns 4s 581203933ns

knucleotide Min Durations %_difference 39.334 38.894 37.391 37.337 36.859 36.384 36.068 33.559 0.0 7.2624 7.3552 11.436 13.952 16.035 20.268 23.769 32.157 41.38 79.28 79.921 80.319 80.433 88.322 89.238 97.235 99.688 101.97

mod_map

time_as_duration

[1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0]

1s 829948135ns 1s 838519189ns 1s 858599369ns 1s 862534748ns 1s 890772899ns 1s 898015128ns 2s 77741974ns 2s 80159397ns 3s 52112662ns 3s 537135748ns 3s 714023788ns 3s 878621573ns 3s 988719588ns 4s 51464255ns 4s 101552056ns 4s 116309892ns 4s 161069662ns 4s 250337017ns 6s 395152534ns 6s 420811965ns 6s 565219855ns 6s 583466351ns 6s 597831592ns 6s 646750275ns 7s 147984203ns 7s 234202114ns 7s 289109294ns

%_difference 50.068 49.629 48.609 48.41 46.99 46.629 37.988 37.876 0.0 14.722 19.565 23.851 26.605 28.137 29.34 29.691 30.748 32.817 70.773 71.123 73.058 73.298 73.487 74.125 80.31 81.314 81.944

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

mandelbrot Min Durations mod_map

time_as_duration

[0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0]

1s 372407075ns 1s 393161897ns 1s 407514057ns 1s 443017492ns 1s 452841981ns 1s 481397848ns 1s 508760576ns 1s 515155776ns 2s 9994371ns 2s 286067276ns 2s 299822659ns 2s 345449350ns 2s 360442193ns 2s 379142550ns 2s 404482069ns 2s 413834285ns 2s 430611386ns 3s 227565390ns 7s 850678912ns 7s 867542604ns 7s 877854707ns 7s 902111548ns 8s 21473295ns 8s 504150490ns 8s 512217793ns 8s 531445517ns 8s 556567954ns

nbody Min Durations %_difference 80.66 79.399 78.534 76.417 75.836 74.164 72.58 72.212 46.494 34.152 33.569 31.657 31.035 30.265 29.229 28.849 28.17 0.0 83.463 83.64 83.748 84.001 85.232 89.954 90.03 90.21 90.444

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

2s 181703329ns 2s 328540828ns 2s 330787495ns 2s 347674287ns 2s 349521825ns 2s 356721997ns 3s 332593609ns 3s 354073712ns 3s 389477337ns 3s 412999738ns 3s 578430510ns 3s 863110869ns 3s 997347153ns 4s 32422369ns 4s 148217041ns 4s 189969306ns 4s 206688380ns 4s 254183789ns 7s 222710208ns 7s 230657368ns 7s 236632517ns 7s 238125354ns 7s 248889319ns 7s 268065019ns 7s 269158871ns 7s 276286200ns 7s 297401433ns

%_difference 0.0 6.5113 6.6076 7.3286 7.4072 7.7127 41.742 42.356 43.358 44.016 48.496 55.631 58.768 59.565 62.134 63.037 63.396 64.404 107.21 107.28 107.34 107.36 107.46 107.65 107.66 107.73 107.94

Laramore, Spracklen, and Jadliwala

pidigits Min Durations mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0]

381553778ns 383419608ns 388122005ns 397716155ns 460939768ns 526278898ns 561323987ns 584525213ns 634471212ns 699532269ns 713454343ns 740282575ns 740663350ns 747753456ns 773971333ns 779993970ns 785086581ns 789269493ns 902110904ns 904474342ns 909519325ns 917025782ns 919682564ns 920576343ns 1s 183110673ns 1s 184638125ns 1s 255696635ns

regexredux Min Durations %_difference 42.02 41.553 40.385 38.037 23.642 10.487 4.0498 0.0 8.1946 17.913 19.866 23.514 23.565 24.504 27.891 28.65 29.287 29.807 42.725 42.975 43.505 44.288 44.563 44.655 67.727 67.841 72.945

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 0, 1, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0]

858411278ns 905870468ns 912055543ns 925373857ns 944552726ns 945778246ns 947677190ns 950754267ns 969658660ns 977405338ns 1s 25465685ns 1s 54316078ns 1s 56256779ns 1s 121015496ns 1s 164709239ns 1s 170492119ns 1s 200109273ns 1s 207429959ns 1s 210736573ns 1s 266543323ns 1s 324648411ns 1s 356042980ns 1s 374394155ns 1s 393703246ns 1s 418592773ns 1s 524932749ns 1s 581160950ns

%_difference 10.208 4.835 4.1549 2.7057 0.65446 0.52475 0.32416 0.0 1.9687 2.7644 7.561 10.33 10.513 16.436 20.228 20.718 23.187 23.786 24.056 28.484 32.864 35.139 36.44 37.787 39.491 46.385 49.797

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

spectralnorm Min Durations mod_map [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0]

time_as_duration 308246580ns 311494232ns 317158343ns 328716001ns 337458389ns 342899642ns 343479043ns 346147126ns 350465804ns 352857345ns 353096263ns 355705622ns 359260899ns 364807555ns 374396822ns 384178648ns 387803137ns 405966248ns 585031608ns 633501678ns 650040817ns 650556592ns 655668970ns 680693910ns 680977303ns 685844156ns 694538396ns

F.2

By mod_map excluding Rust and Go

%_difference 0.0 1.0482 2.85 6.4274 9.0481 10.644 10.812 11.584 12.819 13.496 13.563 14.296 15.285 16.807 19.381 21.932 22.86 27.364 61.971 69.075 71.334 71.404 72.086 75.323 75.358 75.969 77.044

binarytrees Min Durations Only C programs shown

mod_map

time_as_duration

[1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [0, 0, 1, 0, 1, 0] [0, 0, 0, 1, 1, 0] [0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0]

2s 32861102ns 2s 49840569ns 2s 79303898ns 2s 130858809ns 2s 268153184ns 2s 333344353ns 2s 340136279ns 2s 370944604ns 2s 389668825ns 2s 437078371ns 2s 458268465ns 2s 492526755ns 2s 512938441ns 2s 528213589ns 2s 535197836ns 2s 696562207ns 3s 107707060ns 3s 143455294ns 3s 393016507ns 3s 891049580ns 4s 213404207ns 4s 224625004ns 4s 286361933ns 4s 289112387ns 4s 313419616ns 4s 381562379ns 4s 443435317ns

%_difference 21.721 20.899 19.486 17.057 10.844 8.0167 7.7265 6.4202 5.6343 3.6709 2.8054 1.4216 0.60601 0.0 0.27586 6.4443 20.564 21.695 29.21 42.461 49.994 50.243 51.6 51.66 52.187 53.644 54.943

Laramore, Spracklen, and Jadliwala

fannkuchredux Min Durations

fasta Min Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

%_difference

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1]

20s 852956479ns 23s 293798556ns 23s 353293507ns 23s 454567830ns 23s 455147355ns 23s 498811897ns 23s 586171255ns 23s 596873781ns 23s 600597766ns 27s 543939175ns 27s 683255040ns 27s 927590301ns 27s 928904639ns 28s 11496459ns 28s 217245161ns 29s 121002594ns 29s 494527037ns 30s 150331007ns 38s 407972725ns 38s 453165545ns 38s 526122665ns 39s 483923375ns 39s 712817557ns 39s 719572170ns 40s 291480631ns 40s 493011462ns 40s 576456760ns

0.0 11.058 11.312 11.743 11.746 11.931 12.301 12.346 12.362 27.65 28.145 29.006 29.011 29.3 30.015 33.089 34.328 36.458 59.247 59.354 59.527 61.756 62.279 62.294 63.582 64.03 64.215

mod_map

time_as_duration

[1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [1, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0]

998396639ns 1s 2969940ns 1s 18738831ns 1s 19313116ns 1s 24363481ns 1s 29410536ns 1s 32781673ns 1s 59855670ns 1s 599325078ns 1s 600811455ns 1s 667635260ns 1s 710305901ns 1s 720479643ns 1s 746504903ns 1s 822675943ns 1s 888435437ns 2s 57137469ns 2s 263212505ns 3s 440671842ns 3s 466964829ns 3s 483451708ns 3s 488201486ns 3s 839671787ns 3s 883711551ns 4s 301688552ns 4s 443246207ns 4s 581203933ns

%_difference 53.116 52.691 51.237 51.184 50.722 50.262 49.955 47.521 7.2989 7.2061 3.1194 0.5931 0.0 1.5013 5.7686 9.3078 17.824 27.248 66.659 67.335 67.755 67.876 76.228 77.201 85.724 88.348 90.792

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

knucleotide Min Durations

mandelbrot Min Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[1, 0, 0, 0, 1, 0] [1, 1, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 0, 1] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 1, 0]

1s 829948135ns 1s 838519189ns 1s 858599369ns 1s 862534748ns 1s 890772899ns 1s 898015128ns 1s 932941501ns 2s 77741974ns 2s 80159397ns 3s 537135748ns 3s 714023788ns 3s 878621573ns 3s 988719588ns 4s 51464255ns 4s 101552056ns 4s 116309892ns 4s 161069662ns 4s 250337017ns 6s 395152534ns 6s 420811965ns 6s 565219855ns 6s 583466351ns 6s 597831592ns 6s 646750275ns 7s 147984203ns 7s 234202114ns 7s 289109294ns

%_difference 5.4741 5.0072 3.9215 3.7101 2.2057 1.8234 0.0 7.2207 7.3369 58.653 63.081 66.959 69.432 70.801 71.874 72.186 73.125 74.957 107.16 107.45 109.02 109.21 109.37 109.88 114.86 115.66 116.16

mod_map

time_as_duration

[0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0]

1s 372407075ns 1s 375989431ns 1s 393161897ns 1s 407514057ns 1s 443017492ns 1s 452841981ns 1s 481397848ns 1s 508760576ns 1s 515155776ns 2s 9994371ns 2s 286067276ns 2s 299822659ns 2s 345449350ns 2s 360442193ns 2s 379142550ns 2s 404482069ns 2s 413834285ns 2s 430611386ns 7s 850678912ns 7s 867542604ns 7s 877854707ns 7s 902111548ns 8s 21473295ns 8s 504150490ns 8s 512217793ns 8s 531445517ns 8s 556567954ns

%_difference 0.26066 0.0 1.2402 2.2651 4.7554 5.4335 7.3779 9.205 9.6271 37.449 49.703 50.266 52.101 52.695 53.428 54.411 54.77 55.41 140.35 140.46 140.52 140.68 141.43 144.29 144.34 144.45 144.59

Laramore, Spracklen, and Jadliwala

nbody Min Durations

pidigits Min Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 0, 0, 0, 0, 1] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 0]

2s 50706612ns 2s 328540828ns 2s 330787495ns 2s 347674287ns 2s 349521825ns 2s 356721997ns 3s 332593609ns 3s 354073712ns 3s 389477337ns 3s 412999738ns 3s 578430510ns 3s 863110869ns 3s 997347153ns 4s 32422369ns 4s 148217041ns 4s 189969306ns 4s 206688380ns 4s 254183789ns 7s 222710208ns 7s 230657368ns 7s 236632517ns 7s 238125354ns 7s 248889319ns 7s 268065019ns 7s 269158871ns 7s 276286200ns 7s 297401433ns

%_difference 0.0 12.689 12.785 13.504 13.582 13.886 47.625 48.23 49.218 49.867 54.279 61.294 64.372 65.155 67.673 68.559 68.91 69.897 111.54 111.62 111.68 111.69 111.79 111.98 111.99 112.05 112.25

mod_map

time_as_duration

[1, 0, 0, 0, 0, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 1, 0, 0, 1] [1, 0, 1, 0, 0, 0] [1, 1, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [1, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 1, 0] [1, 1, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 1] [1, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 0] [1, 1, 0, 1, 0, 0]

381553778ns 383419608ns 388122005ns 397716155ns 460939768ns 518926285ns 526278898ns 561323987ns 634471212ns 699532269ns 713454343ns 740282575ns 740663350ns 747753456ns 773971333ns 779993970ns 785086581ns 789269493ns 902110904ns 904474342ns 909519325ns 917025782ns 919682564ns 920576343ns 1s 183110673ns 1s 184638125ns 1s 255696635ns

%_difference 30.511 30.034 28.842 26.447 11.836 0.0 1.4068 7.8495 20.036 29.645 31.569 35.158 35.208 36.13 39.453 40.198 40.822 41.331 53.93 54.173 54.688 55.447 55.714 55.804 78.046 78.155 83.034

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

regexredux Min Durations

spectralnorm Min Durations

Only C programs shown

Only C programs shown

mod_map

time_as_duration

[0, 0, 1, 0, 0, 0] [1, 1, 1, 0, 1, 0] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 0, 1, 0, 1, 0] [1, 0, 0, 1, 1, 0] [1, 0, 1, 0, 0, 1] [0, 0, 0, 1, 0, 0] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 1, 0] [1, 1, 0, 1, 0, 0] [1, 0, 0, 0, 1, 0] [1, 1, 0, 1, 1, 0] [1, 1, 0, 1, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 0, 0] [1, 1, 0, 0, 1, 0] [0, 0, 0, 0, 1, 0] [0, 0, 0, 0, 0, 1] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0]

858411278ns 905870468ns 912055543ns 925373857ns 944552726ns 945778246ns 947677190ns 969658660ns 977405338ns 1s 25465685ns 1s 54316078ns 1s 56256779ns 1s 121015496ns 1s 164709239ns 1s 170492119ns 1s 200109273ns 1s 207429959ns 1s 210736573ns 1s 266543323ns 1s 324648411ns 1s 338728526ns 1s 356042980ns 1s 374394155ns 1s 393703246ns 1s 418592773ns 1s 524932749ns 1s 581160950ns

%_difference 43.722 38.569 37.913 36.514 34.527 34.401 34.207 31.976 31.201 26.501 23.77 23.589 17.702 13.902 13.409 10.92 10.314 10.041 5.5415 1.0573 0.0 1.285 2.6291 4.0239 5.7929 13.005 16.606

mod_map [1, 0, 1, 0, 0, 0] [0, 0, 1, 0, 0, 0] [0, 0, 0, 0, 0, 0] [1, 0, 1, 0, 0, 1] [1, 1, 1, 0, 0, 0] [1, 0, 1, 0, 1, 0] [1, 1, 1, 0, 0, 1] [0, 0, 1, 0, 0, 1] [1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0] [1, 0, 0, 0, 1, 0] [1, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 1] [1, 1, 1, 0, 1, 0] [0, 0, 1, 0, 1, 0] [1, 1, 0, 0, 0, 1] [1, 1, 0, 0, 1, 0] [1, 1, 0, 0, 0, 0] [1, 1, 0, 1, 0, 1] [0, 0, 0, 1, 0, 1] [0, 0, 0, 1, 0, 0] [1, 0, 0, 1, 0, 1] [1, 1, 0, 1, 0, 0] [1, 1, 0, 1, 1, 0] [1, 0, 0, 1, 0, 0] [1, 0, 0, 1, 1, 0] [0, 0, 0, 1, 1, 0]

F.3

time_as_duration

%_difference

311494232ns 317158343ns 318509350ns 328716001ns 337458389ns 342899642ns 343479043ns 346147126ns 350465804ns 352857345ns 353096263ns 355705622ns 359260899ns 364807555ns 374396822ns 384178648ns 387803137ns 405966248ns 585031608ns 633501678ns 650040817ns 650556592ns 655668970ns 680693910ns 680977303ns 685844156ns 694538396ns

2.227 0.42507 0.0 3.1541 5.7774 7.3752 7.5439 8.3165 9.5537 10.232 10.3 11.034 12.025 13.551 16.131 18.691 19.621 24.144 58.995 66.174 68.459 68.529 69.219 72.495 72.531 73.149 74.237

By compiler binarytrees Compiler Min Durations

lang

time_as_duration

go rust gcc clang checked-c

8s 220380080ns 550945501ns 435502090ns 603942472ns 4s 33227261ns

Laramore, Spracklen, and Jadliwala

fannkuchredux Compiler Min Durations

nbody Compiler Min Durations

lang

time_as_duration

lang

time_as_duration

go rust gcc clang checked-c

5s 363464276ns 3s 892555077ns 4s 202181310ns 4s 916285218ns 9s 579339016ns

go rust gcc clang checked-c

2s 87315669ns 1s 403163857ns 1s 820927166ns 1s 941024111ns 1s 893988924ns pidigits Compiler Min Durations

lang fasta Compiler Min Durations lang

time_as_duration

go rust gcc clang checked-c

1s 258664686ns 639131005ns 609830880ns 617215369ns 857610136ns

knucleotide Compiler Min Durations lang

time_as_duration

go rust gcc clang

2s 610700370ns 1s 109581068ns 1s 233266183ns 1s 270198179ns

time_as_duration

go rust gcc clang checked-c

389165808ns 357578477ns 336602774ns 338536112ns 365722236ns

regexredux Compiler Min Durations lang

time_as_duration

go rust gcc clang checked-c

7s 814214589ns 579498396ns 586368507ns 599702102ns 2s 193555490ns

spectralnorm Compiler Min Durations lang

time_as_duration

go rust gcc clang

264011338ns 245710903ns 240309303ns 190173426ns

mandelbrot Compiler Min Durations

G lang

time_as_duration

go rust gcc clang checked-c

2s 28205291ns 1s 78850238ns 1s 2286654ns 681409097ns 14s 95789194ns

Minimum Runtime Charts

Scatterplot of minimum runtimes. X-axis: modification combinations mod_map, Y-axis: runtime in seconds. Point colors represent different compilers and points have different protection combinations, with lower values indicating better performance.

H

Error Counts

Error counts compared to total number of benchmarks.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 8: Scatterplot of minimum runtimes of binarytree benchmarks.

Figure 9: Scatterplot of minimum runtimes of fannkuchredux benchmarks.

Laramore, Spracklen, and Jadliwala

Figure 10: Scatterplot of minimum runtimes of fasta benchmarks.

Figure 11: Scatterplot of minimum runtimes of knucleotide benchmarks.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 12: Scatterplot of minimum runtimes of mandelbrot benchmarks.

Figure 13: Scatterplot of minimum runtimes of nbody benchmarks.

Laramore, Spracklen, and Jadliwala

Figure 14: Scatterplot of minimum runtimes of pidigits benchmarks.

Figure 15: Scatterplot of minimum runtimes of regexredux benchmarks.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 16: Scatterplot of minimum runtimes of spectralnorm benchmarks.

Figure 17: Error count by compiler of binarytree benchmarks.

Laramore, Spracklen, and Jadliwala

Figure 18: Error count by compiler of fannkuchredux benchmarks.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 19: Error count by compiler of fasta benchmarks.

Figure 20: Error count by compiler of knucleotide benchmarks.

Laramore, Spracklen, and Jadliwala

Figure 21: Error count by compiler of mandelbrot benchmarks.

Figure 22: Error count by compiler of nbody benchmarks.

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap

Figure 23: Error count by compiler of pidigits benchmarks.

Figure 24: Error count by compiler of regexredux benchmarks.

Laramore, Spracklen, and Jadliwala

Figure 25: Error count by compiler of spectralnorm benchmarks.

Related documents

• Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap Between C Retrofits and Native Safety#616648
• Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap Between C Retrofits and Native Safety#616148
• CauSec: Unboxing the Causal Drivers of Static Vulnerability Analysis Performance#467075
• CRESS: Quantifying Vulnerabilities of Attack Scenarios in Hardware Reverse Engineering#259374
• Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection#238505