Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

arXiv:2611.03742v1 [cs.CR] 2 Nov 2026

JUNHUA TANG, University at Buffalo, SUNY, USA ZHENGSONG ZHANG, University at Buffalo, SUNY, USA HAIPENG CAI, University at Buffalo, SUNY, USA Static application security testing (SAST) tools are adopted on the expectation that a detector set up today will keep finding vulnerabilities in code written later, including previously unknown vulnerabilities. This expectation is rarely tested. In this paper, we test it on 20,000 vulnerable and fixed C/C++ cases from 1,116 open-source projects with source years 2016–2025, under three CodeQL releases, three Infer releases, and one Semgrep configuration. We conduct three studies in which we progressively control for changes in code composition to isolate SAST tool performance differences associated with code evolution over time: yearly populations, project- and label-matched year pairs, and within-project trajectories. We then trace the largest within-project drops to repair diffs, rule and query predicates, alert locations, and execution logs, and we account for failed, degraded, and costly runs. Under our changed-file case-label scoring, the overall F1 of these tools ranges from 0.066 to 0.478, yet every overall count-based AUROC lies between 0.495 and 0.526. For CodeQL, higher F1 in newer releases is accompanied by nearly parallel increases in recall and false-positive rate, rather than better vulnerable/fixed separation. Across the full project-matched year-pair analysis, the mean later-minus-earlier F1 for CodeQL R2024 is negative at every source-year gap, whereas Infer v1.1 and v1.3 are positive at every gap. In the selected drops, only 6 of 39 CodeQL findings and 2 of 620 Infer findings touch a changed repair line, and 78.96% of Semgrep cases carry parser or timeout diagnostics although every run reports success. Overall, our results show that apparent temporal changes in SAST effectiveness can reflect broader alerting, shifting analysis coverage, or incidental findings as well as genuine losses in defect coverage. Temporal sustainability should therefore be evaluated at the level of a fixed detector condition, with project controls, repair-level evidence, and explicit accounting of failed and degraded analyses. CCS Concepts: • Security and privacy → Software security engineering. Additional Key Words and Phrases: vulnerability detection, temporal sustainability

1

Introduction

Static application security testing (SAST) analyzes source or compiled code without running it, so it can flag a weakness before the code ships. The value of that early warning lies in the future. A detector adopted this year is expected to keep flagging the same classes of defects in code that will be written next year, and it matters most when a vulnerability has no identifier, exploit, or patch yet. In one large study, zero-day attacks went undisclosed for 312 days on average [12], and vulnerable code commonly survives in open-source projects for years [1]. A SAST tool cannot know a future zero-day by name, but its queries, abstract domains, and patterns may still cover the defect, provided that the tool can parse, build, and analyze the code in which the defect appears. Both sides of that interaction change. Programs evolve continuously [43]: languages gain idioms, build systems are replaced, and dependencies move. Historical snapshots can become unbuildable; in one study only 38% of the change history of 100 Java projects could still be compiled [57]. Analyzers change as well, through new engines, query packs, rule sets, and recovery behavior. Noirot Ferrand et al. ran 114 CodeQL versions on 3,993 CVEs and found that 21 detected CVEs were no longer detected after a version change [49]. Most SAST evaluations, however, are cross-sectional. They measure how many known vulnerabilities a fixed tool set finds at one point in time and report Authors’ Contact Information: Junhua Tang, University at Buffalo, SUNY, Buffalo, NY, USA, junhuata@buf falo.edu; Zhengsong Zhang, University at Buffalo, SUNY, Buffalo, NY, USA, zhengson@buffalo.edu; Haipeng Cai, University at Buffalo, SUNY, Buffalo, NY, USA, haipengc@buffalo.edu.

2

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

that much is missed [9, 34, 45, 47]. They do not say whether a fixed detector keeps working as the code it analyzes grows older. Answering that question is harder than it looks. Suppose a detector scores F1=0.30 on code from 2016 and 0.10 on code from 2024. The drop may reflect aging, or it may reflect that the 2024 sample contains different projects, a different label balance, larger files that the parser handles only partially, or more runs that timed out and were silently dropped. The same pitfalls are well documented for learning-based security systems, where random splits and mixed time periods inflate results [4, 51, 52]. SAST adds two more. An alert can disappear because the relevant file was never scanned, and a positive score can come from an alert unrelated to the labelled vulnerability. This paper studies the temporal sustainability of SAST: under a fixed, documented detector condition, do effectiveness, executability, and cost remain adequate as the analyzed code moves forward in time? We evaluate 20,000 C/C++ vulnerable and fixed cases from 1,116 projects and source years 2016–2025 under seven detector conditions: CodeQL releases from 2021, 2024, and 2026; Infer v1.1.0, v1.2.0, and v1.3.0; and one Semgrep configuration with 170 rules. Three complementary designs separate a change in the code population from a change over time: yearly populations (Study A), exactly project- and label-matched year pairs (Study B), and balanced within-project trajectories (Study C). We then examine the largest within-project drops at the level of repair diffs, alert locations, rule or query predicates, traces, and logs, and we analyze execution failures and cost on deduplicated targets. The main findings are as follows. • A higher score is not better detection (§4). F1 ranges from 0.066 (Infer v1.1) to 0.478 (Semgrep), yet every overall count-based AUROC lies between 0.495 and 0.526. Newer CodeQL releases more than double F1 by raising recall and FPR together, Infer’s 2024 release outperforms its 2026 release, and one generic Semgrep rule produces 63% of all qualifying findings. • Temporal degradation is detector-condition specific (§5). Fixed historical detector conditions exhibit later-year degradation where post-release observations are available. Across the broader projectand label-matched analysis, CodeQL R2024 has negative mean Δ𝐹 1 at all nine source-year gaps (up to −0.16), whereas Infer v1.1 and v1.3 are positive at every gap and Semgrep changes little at most gaps. Differences between year pairs are larger than most gap means, so no tool-wide “F1 lost per year” exists. • Aggregate effectiveness and within-project temporal trajectories need not agree (§6). Later CodeQL releases have higher F1 but more projects with negative descriptive F1 slopes (37, 74, and 85 of 202), and most flat Infer trajectories are flat only because F1 is zero in every year (144 to 176 of 202 projects). • Several selected extreme F1 drops are driven by off-repair or scope-misaligned findings (§7). In the selected drops, only 6 of 39 CodeQL findings and 2 of 620 Infer findings touch a changed repair line, no Semgrep positive reaches the labelled repair, and 11 of 20 Semgrep vulnerable/fixed groups are scored on disjoint files. The genuine misses we found arise from rule predicates that do not encode the defect and from code that never reaches the analysis. • A successful run can hide a degraded analysis (§8). Execution status changes in both directions across CodeQL releases (281 vs. 177 targets), Infer turns thousands of empty or failed captures into scored successes, 78.96% of Semgrep cases carry parser or timeout diagnostics under a successful status, and two range-analysis queries alone can exhaust a 12-hour budget. Contributions. This paper makes four contributions. • A definition of SAST sustainability that covers effectiveness, executability, and cost, and that ties the property to prospective vulnerability discovery (§2.2). • A three-estimand design that separates yearly ecosystems, matched populations, and withinproject trajectories, with explicit units of inference (§3.5).

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

3

Table 1. The three analyzers, their analysis style, and the components through which code evolution or tool evolution can change their results. Tool

Analysis style

CodeQL Relational database of the program; declarative QL queries over syntax, control and data flow [5, 24] Infer Compositional analysis with procedure summaries, rooted in separation logic and bi-abduction [16, 17, 50] Semgrep Structural pattern matching with constant propagation and intrafile taint rules [54]

Input it needs

Time-sensitive components

Extraction from a traced build or buildless extraction Captured compilation (build interception)

Extractor and compiler discovery; query packs and their models; query evaluation cost Build capture and dependency availability; checker set and models; analysis budget Parser coverage of constructs; rule definitions and severities; per-rule timeouts

Source files, no build

• A large longitudinal evaluation of seven detector conditions from three analyzer families over 20,000 cases, 45 year pairs, 100 resamples, and 202 projects (§4–§6). • A triangulation of scores with repair diffs, alert locations, rule and query predicates, traces, controlled local experiments, and execution ledgers, yielding concrete guidance for tool builders, practitioners, and benchmark designers (§7–§9). Paper organization. Section 2 explains what can age in a SAST tool and gives a motivating example. Section 3 describes the corpus, the seven detector conditions, the scoring rule, and the three studies in enough detail to reproduce them. Sections 4–8 answer the five research questions, one tool at a time, and report every per-year, per-gap, and per-cohort table we have. Each results section ends with a short boxed finding. Section 9 draws the results together, Section 9.5 discusses threats to validity, and Section 10 relates the study to prior work. 2 2.1

Background and Motivation SAST Families and What Can Age

SAST covers lightweight pattern matching, data-flow and taint analysis, abstract interpretation, and database-backed semantic queries [10, 28, 53]. Each family depends on interfaces that evolve over time. Table 1 summarizes the three analyzers we study and the components through which time can affect them. The table makes one point that the rest of the paper relies on. A SAST result is the product of an engine, a set of rules or queries, an extraction or parsing step, and an execution policy. When any of these components meets code it was not built for, the result can change even if nobody touched the defect-detection logic. We therefore treat the whole bundle, which we call a detector condition, as the object under study. 2.2

Sustainability and Zero-Day Relevance

Let 𝐷𝑟 be a detector condition (engine release, rules or queries, mode, and execution policy) and 𝐶𝑡 the population of code from source year 𝑡. We call 𝐷𝑟 effectiveness-sustainable when Perf (𝐷𝑟 , 𝐶𝑡 ) does not degrade with 𝑡 after known composition changes are controlled; reliability-sustainable when 𝐷𝑟 keeps parsing, building, extracting, and finishing on 𝐶𝑡 ; and cost-sustainable when its time and memory remain within practical budgets. Concept drift, a change in the relation between inputs and labels over time [32], is one cause of degradation. SAST can also degrade without any learned decision boundary, simply because the code no longer reaches the analysis. Our full longitudinal analyses characterize how a fixed detector condition responds to code from different source years. We use aging more specifically when later code follows the frozen detector

4

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

condition in time; comparisons involving code that predates the condition characterize temporal sensitivity but not prospective aging. The link of our study to zero-days is prospective rather than evidentiary. A zero-day has no case label when it is discovered, so no retrospective corpus measures zero-day discovery directly. Our corpus tests a necessary condition: whether detector logic and execution still apply to defectbearing code as that code moves away from the context in which the detector condition was built. A sustainable detector may still miss a zero-day. A detector that has lost applicability has one more, measurable reason to miss it. 2.3

Motivating Example

HAProxy illustrates why an aggregate score can mislead. In our corpus, the 2019 and 2023 HAProxy cells each contain one vulnerable and one fixed case per resampling repetition. Under CodeQL R2021, F1 falls from 1.0 in 2019 to 0.0 in 2023, which looks like a clear loss. The 2019 alerts, however, are in src/proto_htx.c, while the paired fix changes src/mux_h2.c; once only alerts in the repaired file are counted, the 2019 F1 is already zero. Under R2024 the 2023 cell has F1=2/3, which looks like a gain. Here an overflow-style warning on if (unlikely(cl + c < cl)) appears on both the vulnerable and the fixed snapshot (the line only moves from 57 to 64), because the repair leaves that check alone and instead rejects an empty Content-Length header. The cell therefore has one true positive and one false positive, and its FPR is 1.0. Under R2026 the same cell loses the true positive but keeps the false positive, so its F1 is zero again. None of these changes says anything about the CVE. Section 7 shows that such cases are common among the largest changes. 3 3.1

Study Design Research Questions

We ask five research questions. RQ1–RQ3 and RQ5 are quantitative; RQ4 is explanatory. • RQ1 (ecosystem effectiveness). How do effectiveness and coverage vary across source years and detector conditions? • RQ2 (project-controlled change). After project and label composition are matched, how does later-year performance differ from earlier-year performance as the year gap grows? • RQ3 (within-project change). As the same project evolves, how often and how strongly does detector performance change? • RQ4 (failure mechanisms). Which scope, rule or query, parsing, extraction, and execution mechanisms explain the largest within-project drops? • RQ5 (reliability and cost). How do execution failures, degraded analyses, and measured costs differ across conditions, and how often does the same target change status? 3.2

Corpus and Temporal Identity

The frozen corpus contains 20,000 C/C++ cases: 1,000 vulnerable and 1,000 fixed cases for each source year from 2016 to 2025, drawn from 1,116 canonical projects. The source year is the year of the analyzed snapshot. A case is a single commit snapshot of a C/C++ repository, with the files changed by the associated security fix and the fix diff. A fixed case is the fix commit named by a vulnerability record; a vulnerable case is the commit the record marks as vulnerable or, if none is given, the parent of the fix commit. Cases come from NVD (15,776), PrimeVul (3,854), and OSS-Fuzz/OSV (370) records. Each year holds 1,000 vulnerable and 1,000 fixed cases drawn at random from these candidates. A vulnerability record (vulnerability ID such as a CVE number, repository, and commit) listed by

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

5

Table 2. The seven detector conditions. A condition comprises an analyzer release and its associated rules, extraction, and execution/recovery policy; target-specific recovery differences are tracked as provenance. Condition

Engine

Release date / bundle

Rules or queries

Execution

CodeQL R2021 CodeQL R2024 CodeQL R2026 Infer v1.1 Infer v1.2 Infer v1.3 Semgrep

CLI 2.7.3 CLI 2.20.0 CLI 2.25.2 1.1.0 1.2.0 1.3.0 1.161.0, OSS mode

bundle 2021-12-08 bundle v2.20.0 — 2021-03-26 2024-06-21 2026-05-12 —

cpp queries 0.0.4 cpp queries 1.3.0 cpp queries 1.7.0 bundled defaults bundled defaults bundled defaults 170 local rules

traced (manual) build buildless (build-mode none) buildless, release-level recovery capture + analyze capture + analyze capture + analyze scan of changed files

several sources was kept only once among the vulnerable or fixed cases; different CVEs fixed by the same commit are kept as separate cases. Projects hosted under several URLs (e.g., GitHub and git.kernel.org copies of Linux) are merged into one canonical project, while related but distinct codebases such as ImageMagick and ImageMagick6 are kept separate. 3.3

Detector Conditions

We chose CodeQL, Infer, and Semgrep because they are widely used, open to inspection, and built on three different analysis architectures (Table 1), and we picked release points that span the source years of the corpus. Table 2 lists the seven conditions. Because engine, rules, extraction, and recovery change together, our comparisons across releases are descriptive. They do not isolate a pure engine-version effect, and we never read them as an estimate of what a version upgrade does. CodeQL. The 2021 condition uses traced compilation through a saved build wrapper. The 2024 condition uses buildless extraction, in which CodeQL’s autobuild step discovers a compiler and extracts the source without running the project’s build. R2026 is a single release-level condition. Its native security suite (security-extended in cpp queries 1.7.0) has 98 queries. During recovery, some R2026 targets were rerun with a different extraction scope, a reduced query set, or cached results; we keep these differences as provenance of one condition rather than treating them as extra releases. The query lists that each historical release actually resolved were saved, and they differ: for example, the R2021 list lacks cpp/missing-check-scanf and cpp/use-after-free, which the R2024 list contains, and neither list selects MissingNullTest or InconsistentNullnessTesting. These differences matter in §7. Infer. Each Infer condition uses the native binary, models, and default checkers shipped with its release. The command sequence is the same for all three. infer capture –keep-going runs over the full checked-out source tree into a version-specific output directory. infer analyze –keep-going then runs with eight jobs and the buffer-overrun analysis enabled explicitly; when the case’s changed-file index is nonempty, it is passed with –reactive, and an empty index is recorded as an explicit fallback to full analysis. Finally, infer report runs against the same output. The baseline stage budgets are 3,600 seconds for workspace setup, 1,800 for project setup, 7,200 each for capture and analysis, 120 for reporting, 600 for cleanup, and 21,600 end to end. Recovery configurations are versioned supplements with larger budgets and explicit repairs. Recovery rounds 1–9 allow 21,600 seconds for capture and 43,200 for analysis (72,000 end to end), and rounds 10–13 allow 43,200 and 86,400 (144,000 end to end). The later rounds supply 95 v1.1 and 215 v1.3 selected results. Several completed candidate runs can exist for one case. The selection rule, inherited from the original v1.2 workflow, keeps the candidate with the highest changed-file score within each identity-aware reuse group. Results are reused across cases only when commit, Git tree, input fingerprint, changed files, and labels all match; a shared commit alone is not enough.

6

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Semgrep. Semgrep runs in OSS mode with 170 C/C++ rules: 141 search rules and 29 taint rules. Their metadata marks 16 as community rules and 154 as Pro-authored, so rule provenance and engine mode are separate properties. By severity, 109 rules are INFO, 38 WARNING, 21 ERROR, and 2 CRITICAL; only the 61 rules at WARNING or above count toward the score. The configuration is not purely syntactic: it includes taint rules and semantic rules such as null-library-function and local-variable-new-delete. Each case runs semgrep scan over the directory of exported changed files, with JSON output and recorded cache reuse. All 20,000 raw outputs record version 1.161.0 and OSS mode. The runner supports per-rule time limits, a whole-scan deadline, and parallel jobs, but the historical records do not keep the effective values per case, so we do not treat the current script defaults as measured settings. 3.4

Scope, Prediction, and Metrics

All pipelines score findings inside a resolved changed-file scope. We score within the changed files because the vulnerable/fixed label refers to the code touched by a commit. Counting alerts anywhere in a repository would let any unrelated warning decide the label, and changed-file scanning resembles the diff-focused way SAST warnings are consumed in code review [19]. • For CodeQL, the score is the number of security-relevant findings whose primary location lies in the case’s changed-file set. • For Infer, the normalized report has one entry per issue, and an issue counts when its top-level file matches a path in the case’s changed-file list (the union of old and new paths, matched exactly or by a fixed suffix rule). The score counts findings, not files, traces, or procedures; the bug trace is kept as evidence but does not enter the score. • For Semgrep, the score is the number of WARNING-or-higher findings in the changed-file set selected for the case. The scope is resolved from target metadata and the before/after commit relation, never from the role alone: 7,801 vulnerable cases use the changed_before file set, and 2,062 vulnerable cases and all 9,881 primary fixed cases use the changed_after set. These names select a file set; they do not by themselves say whether the scanned code is pre-fix or post-fix. A score above zero predicts the case as vulnerable. We then compute standard binary-classification metrics—precision, recall (TPR), F1, and FPR—against the vulnerable/fixed case labels. Thus, a true positive means that a vulnerable case has at least one qualifying in-scope finding; it does not imply CWE agreement, repair overlap, or detection of the labeled vulnerability mechanism. We additionally report AUROC over the raw finding counts, using average ranks for ties [31], to measure whether vulnerable cases tend to receive more findings than fixed cases; AUROC is undefined when one class is absent. When a denominator is zero, precision, recall, F1, and FPR are recorded as zero, which matters in small cells (i.e., when an evaluation stratum contains few cases). We read F1 together with FPR and AUROC because F1 can rise when a tool reports more findings on both labels. A successful run with no findings (including an empty report) is an evaluated negative, whereas a failed or missing run has no prediction and enters only reliability denominators. Overall Study-A metrics are pooled over all cases rather than averaged over years. 3.5

Three Complementary Studies

Figure 1 summarizes the design. The three studies answer different questions and use different weights, so their numbers are not interchangeable. None of them reruns a detector: Studies B and C resample the stored Study-A results. Study A: observed ecosystem. For each condition, Study A evaluates the complete eligible population of each source year. It describes what a user would see on the corpus as it is, including changes in projects and defect types. It does not identify a temporal effect, because the project

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

Study A: observed ecosystem (RQ1) ··· 2016 2017 2018 2025

Study B: matched year pairs (RQ2) year 𝑥 year 𝑦 𝑝 : 𝑞𝑝 V + 𝑞𝑝 F 𝑝 : 𝑞𝑝 V + 𝑞𝑝 F

all eligible cases of each source year 𝑡

same projects, same 𝑞𝑝 per label

Perf (𝐷𝑟 , 𝐶𝑡 ) per year (19,344–19,744 cases per condition)

ΔF1 = F1(𝐶 𝑦 ) − F1(𝐶𝑥 ) 45 pairs × 100 repetitions

RQ4 Largest Study-C drops → repair diffs, alert and trace locations, rule and query predicates, location ablation, local controls

7

Study C: within-project paths (RQ3) OLS

one project, years with both labels

slope per project: ≥ 3 years (202) or ≥ 5 years (61, nested)

RQ5 Deduplicated execution targets → final status, status transitions, diagnostics, phase time, CPU, and memory

Fig. 1. Study design. Studies A–C answer RQ1–RQ3 with different estimands; the qualitative analysis (RQ4) starts from Study-C extremes, and the operational analysis (RQ5) uses execution ledgers. V/F: vulnerable/fixed; 𝑞𝑝 : matched capacity of project 𝑝.

distribution 𝑃 (project | 𝐶𝑡 ) changes with 𝑡: a difference between two years may come from different projects rather than from older or newer code. Study B: matched year pairs. Study B asks whether a detector still performs differently on code from two years once project composition is held fixed. For each of the 45 pairs (𝑥, 𝑦) with 2016 ≤ 𝑥 < 𝑦 ≤ 2025, a project is eligible if it has both labels in both years; it does not need to appear in any year in between, and it need not span the decade. Its matched capacity 𝑞𝑝 is the minimum count over its four project–year–label cells, 𝑞𝑝 = min(𝑛𝑝,𝑥,𝑉 , 𝑛𝑝,𝑥,𝐹 , 𝑛𝑝,𝑦,𝑉 , 𝑛𝑝,𝑦,𝐹 ). Each of 100 deterministic repetitions draws 𝑞𝑝 vulnerable and 𝑞𝑝 fixed cases without replacement from each year, so both sides contain exactly the same projects with the same number of cases per label, 𝑃 (project, label | 𝐶𝑥 ) = 𝑃 (project, label | 𝐶 𝑦 ) [55]. Within a repetition, the selected cases are pooled over projects in each year, so projects contribute in proportion to their matched capacity. We compute ΔF1 per repetition, average the repetitions within a pair, and weight pairs equally within each gap Δ𝑡 = 𝑦 − 𝑥. Across the 45 pairs there are 1,928 eligible project–pair combinations. Gap 1 has nine pairs, and each longer gap has one pair fewer, down to the single 2016→2025 pair at gap 9. Matching does not stop one project from dominating a pair: the largest project supplies between 15.0% and 92.8% of the matched capacity, depending on the pair. Study B deliberately does not restore 1,000 cases per label and year; it trades scale for comparability. Thus, Study B estimates a case-weighted temporal difference under identical project-and-label composition; it does not give every matched project equal influence. Study C: within-project trajectories. Study C asks whether a project’s own F1 changes as the project evolves. A usable year for a project is a source year in which it has at least one case of each label; usable years need not be consecutive. Study C keeps projects with at least three usable years (primary cohort: 202 projects and 869 project-year cells) and, as a stricter sensitivity check, projects with at least five (nested cohort: 61 projects and 396 cells). Each project-year draws min(𝑛𝑉 , 𝑛 𝐹 ) cases per label in each of 100 repetitions, which gives 15,598 selected cases per repetition in the primary cohort and 13,126 in the sensitivity cohort. We average repetitions per project-year and fit an ordinary least-squares F1 slope over the actual source years of each project, which weights projects equally; a slope needs at least two years with a defined metric. Slopes with absolute value at most 10−12 (10−10 for Semgrep) count as zero, to absorb floating-point noise. Annual cohort metrics are pooled over projects within each repetition and then averaged over the 100 repetitions, so they weight cases; the cohort summaries in this paper are equal-weight means of these ten annual values. We report both views. These slopes are descriptive summaries rather than per-project significance tests; some project-year cells are small, which is why we report the nested ≥5-year cohort and the underlying support alongside slope directions. Common rules for Studies B and C. Memberships were fixed from the corpus catalog before any detector result was compared, and the same memberships are used for all seven conditions. Balancing happens before the gates of §3.6, including the corpus-wide label-conflict gate, so the

8

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 3. Unique evaluable cases used by each study under the three Infer conditions. Studies B and C reuse already scored cases; the cohorts are nested, so rows must not be added. The full Study-B membership ledger holds 4,517,600 case references across repetitions and pairs. Study Study A Study B Study C, primary (≥3 years) Study C, sensitivity (≥5 years)

v1.1

v1.2

v1.3

Common intersection

19,461 17,368 15,669 13,108

19,700 17,557 15,828 13,234

19,581 17,464 15,759 13,197

19,450 17,357 15,658 13,097

Table 4. How each detector condition (a tool release or configuration) goes from 20,000 planned cases to its Study-A population. All counts are cases. Condition

Planned

Evaluable

CodeQL R2021

20,000

19,709

CodeQL R2024

20,000

19,563

CodeQL R2026 Infer v1.1

20,000 20,000

20,000 19,760

Infer v1.2 Infer v1.3

20,000 20,000

20,000 19,881

Semgrep

20,000

20,000

Study A

What is excluded

19,455 104 timeout, 182 database-failure, and 5 unmapped cases; 254 label conflicts 19,344 332 timeout, 100 database-failure, and 5 unmapped cases; 219 label conflicts 19,744 256 label conflicts; no execution failure after recovery 19,461 232 build failures, 8 timeouts; identity gate (47); 252 label conflicts 19,700 identity gate (48); 252 label conflicts 19,581 108 build failures, 11 timeouts; identity gate (48); 252 label conflicts 19,744 256 label conflicts; 109 zero-target cases kept

frozen memberships stay identical for all conditions. A case that a gate removes is dropped without replacement, so evaluated counts can be slightly unbalanced and can vary across repetitions. The 2.5–97.5% ranges we report describe resampling variation for a fixed pool of cases. They are not confidence intervals for a wider population of projects, because pairs and repetitions share projects and cases. The 100 repetitions reuse stored results, so they cannot measure run-to-run nondeterminism, and a case that appears in 100 repetitions is not 100 independent detections. Table 3 shows how many unique evaluable cases each study uses for Infer; the counts overlap and must not be added. 3.6

Evaluation Populations

Table 4 shows how each condition goes from 20,000 planned cases to its Study-A population. Because the gates differ by tool, we also report results on the common intersection of each tool’s releases: 19,169 cases for CodeQL and 19,450 for Infer. Effectiveness is counted on case rows and reliability on deduplicated targets (19,780–19,820 per CodeQL release for 20,000 cases). Label-conflict gate. Some CVEs are fixed by several commits, so an intermediate commit is both the fixed version of one patch and the vulnerable parent of the next. Using corpus metadata only, we find 256 records (137 vulnerable, 119 fixed) whose repository, vulnerability, and commit appear under both labels; 253 of them come from 2016–2021, with the most in 2020 (76; Figure 2(b)). These 256 ambiguous case identities are excluded from label-based evaluation for all tools. They were executed like every other case and stay in the execution and reliability counts. The gate removes 254, 219, and 256 cases from the CodeQL R2021, R2024, and R2026 populations and 252 from each Infer population; the remaining conflicting cases had already been removed by an execution failure or the identity gate. CodeQL. CodeQL registers 19,820 canonical targets for the 20,000 cases, because several cases can share one execution target. The two historical releases each connect to 19,780 of them. Of the 40

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

(a) Infer final failures Cases

v1.1 build failed v1.1 timeout

v1.3 build failed v1.3 timeout

75

(b) Label conflicts

50

50 25

25

0

0 '16

'18

'20

'22

'24

(c) Semgrep zero targets

76

75 49

19

'18

19

20 10 1 0 2 0

'16

31

30

44 49 16

'20

'22

9

16 11

6 8

5 6

2

5

0

'24

'16

'18

'20

'22

'24

Fig. 2. Cases removed or flagged per source year (x-axis). (a) Final Infer build failures and timeouts per 2,000 planned cases; v1.2 has none. (b) Corpus records excluded for all tools because the same repository, vulnerability, and commit occur under both labels. (c) Primary Semgrep cases in which no file was scanned; they stay in the population with score zero. Table 5. Semgrep confusion counts for the whole catalog and for the primary population, with all 61 qualifying rules and with security-category rules only. The four differences between catalog and primary counts (37, 31, 100, 88) sum to the 256 label-conflict records. Population

Cases

TP

FP

FN

TN

All rules, catalog All rules, primary Security-category rules, catalog Security-category rules, primary

20,000 19,744 20,000 19,744

4,626 4,589 4,550 4,515

4,764 4,733 4,671 4,642

5,374 5,274 5,450 5,348

5,236 5,148 5,329 5,239

registration differences, 35 are aliases of other targets and 5 cannot be resolved; the latter are the five unmapped cases in Table 4. Measured over cases, the coverage of the three releases is 98.55%, 97.82%, and 100%; most missing cases come from timeouts and database failures, and only five from unresolved registrations. After the label-conflict gate, the class sizes are 9,719 vulnerable and 9,736 fixed cases for R2021, 9,650 and 9,694 for R2024, and 9,863 and 9,881 for R2026. Infer. Only build failures and timeouts count as execution failures for Infer. The identity gate is a separate, version-independent rule inherited from the original workflow: it drops 40 catalog identities whose directory leaf does not follow the expected naming convention and 8 that are not the chosen representative of a truncated composite key. The gate removes 48 cases in v1.2 and v1.3 but 47 in v1.1, because one of the 48 identities is already a v1.1 build failure. After the label-conflict gate, the class sizes are 9,703 vulnerable and 9,758 fixed cases (v1.1), 9,823 and 9,877 (v1.2), and 9,761 and 9,820 (v1.3). Figure 2(a) shows that failures concentrate in a few years: v1.1 loses 70 of the 2,000 cases of 2016 to build failures, and all timeouts of both releases fall in 2020. In 2021–2024 the two releases have identical yearly failure counts. Semgrep. Every Semgrep case has a resolved scope and a result record. Apart from the shared label-conflict gate, Semgrep has no primary exclusion, leaving 9,863 vulnerable and 9,881 fixed cases. Diagnostics and timeouts never remove a case. The 109 primary cases (113 in the catalog) in which Semgrep scanned zero files are kept with score zero; Figure 2(c) shows their distribution, again peaking in 2020. Table 5 gives the confusion counts before and after the quarantine for all rules and for the security-category rules alone. 3.7

Reliability and Cost Measurement

Reliability uses deduplicated execution targets, so that a target shared by several cases is not counted as several failures. For CodeQL, the unified ledger has 60,000 case-by-release rows and 59,380 releaseby-target records; these are neither 60,000 vulnerabilities nor 59,380 distinct snapshots. We classify final outcomes as success, timeout, database or extraction failure, build failure, or missing, and we

10

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 6. Qualitative evidence queues (RQ4). Parent-to-fix relations were checked as first-parent comparisons; for CodeQL, the 13 relations cover 26 of the 27 cases.

Selected project-year pairs Unique cases Case × release records Verified parent-to-fix relations Scored findings inspected Before/after file pairs reviewed

CodeQL

Infer

Semgrep

8 27 81 13 39 —

9 34 102 17 620 —

9 42 42 19 of 20 73 67

compare exact target identities (target, commit, role, project, year) across CodeQL releases. An intermediate failed attempt counts as a final failure only if it is explicitly linked to the selected attempt. For cost, we use recorded phase wall time, CPU time, and peak resident set size (RSS). Missing and cached values are never treated as zero, and CPU time is never inferred from wall time. These measurements do not control machine load or cache state, so we use them to describe operating ranges, not to rank tools by speed. 3.8

Qualitative Failure Analysis

For RQ4 we take, for each tool, the primary Study-C cohort and compute every project’s annual F1 averaged over the repetitions. We then rank all early-to-late year differences within each project and keep every pair that ties for the largest decline. The selection therefore follows endpoint differences, and slopes play no role in it. We compare the same endpoints across releases. For CodeQL and Infer, the primary ties come from the 2021 release (four CodeQL ties under R2021 and one Infer tie under v1.1), and the other releases add their own ties as supplementary examples; they are not an exhaustive list of those releases’ ties. Semgrep has one condition. For each selected case we inspect the target identity, the scored snapshot side, the changed-file scope, the vulnerable-to-fixed diff, alert and trace locations, diagnostics, and the definitions of relevant rules or queries. A positive counts as mechanism-supported only if the labelled repair, the scanned source, the rule semantics, and the finding or trace location all agree; otherwise it is label-level, and if the evidence cannot settle the relation it is unresolved. For zero-finding cases we separate three explanations that can overlap: the repair lies outside the scanned files (a scope limitation), the scanned repair file carries a diagnostic (an analysis limitation), or the scanned repair file has no diagnostic and still no finding, which can motivate a rule–target hypothesis. When classifying false negatives, scope mismatch takes precedence. We also apply deterministic location ablations that keep only already-scored alerts in a verified repair file, or on a changed repair line, without changing labels, findings, or repetitions. Table 6 lists the evidence in each queue. For CodeQL we also ran bounded local experiments: a compiler-visibility control, a source-slice query test, and a saved-build-wrapper control (§7.3 and §8.1). None of them reruns a historical target or changes a reported score. Reliability mechanisms are graded by evidence: a direct diagnostic, a locally validated mechanism, or a uniquely linked historical cause. We state the grade with each claim, and “unresolved” always means that the specific historical attribution is not isolated, not that no mechanism is known. 3.9

Validity Safeguards

Membership was frozen before conditions were compared, the same Study-B/C samples are reused for every condition, and all 100 repetitions are retained. The two Study-C cohorts are nested and never summed. Cases, targets, findings, project-years, and repetitions are distinct units, and each claim names its unit. An audit of the CodeQL metric, denominator, weighting, deduplication,

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

11

Table 7. Study-A overall results (changed-file scoring). AUROC ranks raw finding counts. “Sens. F1” is F1 on the common intersection of the tool’s releases (CodeQL 19,169 and Infer 19,450 cases) or, for Semgrep, after removing 109 zero-target cases. Condition

Eval.

CodeQL R2021 CodeQL R2024 CodeQL R2026 Infer v1.1 Infer v1.2 Infer v1.3 Semgrep

19,455 19,344 19,744 19,461 19,700 19,581 19,744

TP

FP

FN

686 729 9,033 9,007 .4848 1,265 1,264 8,385 8,430 .5002 2,042 2,105 7,821 7,776 .4924 345 363 9,358 9,395 .4873 1,003 505 8,820 9,372 .6651 830 872 8,931 8,948 .4877 4,589 4,733 5,274 5,148 .4923

2021 release (R2021, v1.1)

Metric value

.0706 .1232 .1311 .2077 .2070 .2915 .0356 .0663 .1021 .1770 .0850 .1448 .4653 .4784

(b) Infer: F1

0.3

0.3

0.2

0.2

0.1

0.1

0.0

0.0 '18 '20 '22 '24 Source-code year

F1 AUROC

2024 release (R2024, v1.2)

(a) CodeQL: F1

'16

TN Prec. Recall

.4979 .5005 .4969 .4992 .5257 .4981 .4951

FPR Sens. F1 .0749 .1304 .2130 .0372 .0511 .0888 .4790

.1249 .2091 .2913 .0663 .1778 .1446 .4801

2026 release (R2026, v1.3)

(c) Semgrep

0.6 0.4 0.2

TPR (recall)

'16

'18 '20 '22 '24 Source-code year

'16

FPR

F1

AUROC

'18 '20 '22 '24 Source-code year

Fig. 3. Study-A annual profiles by source-code year (x-axis). (a, b) F1; colors and markers denote the release year and mean the same in every release plot of this paper. (c) Semgrep F1, TPR, FPR, and count-based AUROC; the dotted line marks 0.5. Lines connect yearly populations with different projects, not trajectories of the same cases.

coverage, and provenance computations passed all 1,728 checks. The Infer pipeline reproduced all ten reference Study-B/C tables of the original v1.2 workflow (four Study-B and six Study-C outputs) and validated all 60,000 case-by-release identities. The CodeQL and Infer values after the label-conflict gate come from a separate recomputation that reproduces every earlier Study-A/B/C value exactly when the gate is switched off. For Semgrep, we recomputed every derived metric from its confusion counts, and the rule-version identifiers embedded in all 73 queue findings match the rule definitions we inspected. All 27 selected R2026 CodeQL result files are available (six of them recovered from a result archive) and match their target path, commit, and normalized findings. Language-model agents assisted with artifact navigation, source and log inspection, table construction, and running the bounded local control experiments, and review agents checked those artifacts. The labels, the frozen results, and every reported metric come from the study artifacts rather than from the agents, and every qualitative claim is tied to inspectable source, diff, rule, query, or log evidence. 4

RQ1: Ecosystem-Scale Effectiveness (Study A)

Table 7 reports the Study-A results for all seven conditions, and Figure 3 shows the annual profiles. To read the table, take the CodeQL R2026 row: on 19,744 evaluated cases, 49.2% of positive predictions fall on vulnerable cases, 20.7% of vulnerable cases are flagged, and 21.3% of fixed cases are flagged as well, so the raw-count AUROC of 0.4969 is no better than chance. The last column repeats F1 on a sensitivity population that removes coverage differences.

12

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 8. Annual Study-A metrics for Infer. Shading marks each column’s maximum (blue, including ties) and minimum (red). Primary cases per year: 1,879–1,992 (v1.1), 1,914–2,000 (v1.2), and 1,891–1,992 (v1.3). Year Prec. Rec. 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025

4.1

v1.1 v1.2 v1.3 F1 FPR AUC Prec. Rec. F1 FPR AUC Prec. Rec. F1 FPR AUC

.5000 .0267 .0507 .0265 .5217 .0247 .0472 .0226 .4865 .0373 .0693 .0392 .4545 .0314 .0587 .0373 .5047 .0574 .1032 .0557 .4830 .0724 .1259 .0772 .4767 .0421 .0773 .0453 .4833 .0292 .0551 .0312 .4375 .0141 .0272 .0181 .5250 .0212 .0408 .0192

.5001 .5010 .4990 .4971 .5008 .4977 .4984 .4990 .4980 .5010

.6524 .1104 .1889 .0582 .6271 .1121 .1902 .0665 .5550 .1138 .1889 .0911 .5605 .1291 .2099 .1002 .6290 .0818 .1448 .0479 .6832 .1114 .1916 .0514 .7541 .1874 .3002 .0601 .8235 .1120 .1972 .0240 .9565 .0220 .0431 .0010 .7679 .0430 .0814 .0130

.5268 .5229 .5113 .5147 .5169 .5305 .5645 .5447 .5105 .5150

.4833 .0898 .1514 .0950 .4468 .0637 .1115 .0787 .4975 .1014 .1685 .1021 .4657 .0993 .1637 .1127 .5000 .1619 .2446 .1597 .4950 .1009 .1677 .1025 .5000 .0656 .1160 .0645 .4910 .0827 .1415 .0857 .4940 .0412 .0760 .0422 .5000 .0491 .0895 .0492

.4973 .4925 .4998 .4934 .5007 .4992 .5007 .4985 .4995 .4999

CodeQL: Higher F1 without Better Separation

Across the CodeQL releases, F1 more than doubles, from 0.1232 to 0.2077 to 0.2915. At first sight this looks like steady tool improvement. The confusion components say otherwise. Recall rises from 0.0706 to 0.2070, but FPR rises in parallel from 0.0749 to 0.2130, precision stays close to 0.5, and AUROC never moves more than 0.0031 away from 0.5. In absolute terms, R2026 flags 4,147 cases against 1,415 for R2021, adding 1,356 true positives and 1,376 false positives. The newer releases flag more cases on both labels; they do not rank vulnerable cases above fixed ones any better. The ordering survives on the 19,169-case common intersection (F1 0.1249, 0.2091, 0.2913), so it does not come from which targets each release happened to complete. That intersection is itself a selected population, though: it keeps only targets on which every release succeeded and may therefore favor easier targets. It checks the ranking but cannot replace the reliability denominators. The annual profiles in Figure 3(a) are not smooth functions of source age. R2021 stays between about 0.16 and 0.20 until 2020, then falls sharply and reaches about 0.02 in 2024. R2024 swings between years and ends near 0.11 in 2025. R2026 is the highest curve in every year but ends below its own 2016 value. Because each year has a different project mix, these shapes motivate the controlled studies of the next two sections rather than answer the aging question. Finding A1. Newer CodeQL releases reach higher F1 mainly by flagging more cases on both labels: recall and FPR rise together and AUROC stays at chance. A higher score does not mean better discrimination.

4.2

Infer: The Middle Release Performs Best

Infer does not improve with release date. v1.1 finds 345 true positives against 9,358 false negatives (recall 0.0356). v1.2 nearly triples the true positives to 1,003 while keeping false positives at 505, which gives the best Infer precision (0.6651), F1 (0.1770), and AUROC (0.5257). v1.3 then loses ground on both sides: 830 true positives and 872 false positives. Its FPR of 0.0888 is 74% higher than that of v1.2, while its recall is lower. Compared with v1.1, v1.3 raises recall and F1 without improving AUROC. The dominant source of low F1 for all releases is the false-negative count: between 8,820 and 9,358 vulnerable cases receive no changed-file finding at all. A zero score may mean that the snapshot has no issue, that the issue lies off the scored paths, or that capture was limited; it is not a diagnosis of which pattern a checker cannot recognize. Likewise, a finding on a fixed case is a false positive with respect to the label even when it points at a different real defect. On the 19,450-case common intersection, the three releases keep the same order. Precision, recall, F1, AUROC, and FPR are 0.4873, 0.0356, 0.0663, 0.4992, and 0.0372 for v1.1; 0.6660, 0.1026, 0.1778, 0.5260, and 0.0512 for v1.2; and 0.4870, 0.0849, 0.1446, 0.4980, and 0.0889 for v1.3. Because the intersection keeps only cases that every release completed, it can favor cases that are easy to build; it is a check on support, not a correction for selection.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

1.0 0.8

(a) Precision v1.1 v1.2 v1.3

0.6 0.4 '16 '18 '20 '22 '24

0.20

(b) Recall

0.20

0.15

0.15

0.10

0.10

0.05

0.05

0.00

0.00 '16 '18 '20 '22 '24

(c) FPR

0.575

13

(d) Count AUROC

0.550 0.525 0.500 '16 '18 '20 '22 '24

'16 '18 '20 '22 '24

Fig. 4. Annual Study-A precision, recall, FPR, and count AUROC for the three Infer releases by source-code year (x-axis). The F1 curves are in Figure 3(b); the dotted line in (d) marks 0.5.

Table 8 and Figure 4 give the annual values behind Figure 3(b). Each release peaks in a different year: v1.1 reaches F1 0.1259 in 2021, v1.2 0.3002 in 2022, and v1.3 0.2446 in 2020. All three are far lower for the 2024 code than at their peaks. The two figures also show that recall and FPR of v1.1 and v1.3 move together from year to year. v1.3 has its highest recall (0.1619) and its highest FPR (0.1597) in the same year, 2020, and its annual AUROC never exceeds 0.5007; v1.1’s annual AUROC lies between 0.4971 and 0.5010. v1.2 is the only release that separates the labels at all: its annual AUROC stays between 0.5105 and 0.5645, the highest single value of any Infer cell (2022). The v1.2 row for 2024 shows how one metric can mislead in a sparse cell. Precision reaches 0.9565 only because almost no fixed case is flagged (FPR 0.0010), while recall is 0.0220. Coverage also differs by year (Figure 2(a)): in 2016, v1.1 has 1,879 primary cases, against 1,949 for v1.2 and 1,948 for v1.3. These differences do not explain the annual pattern away, because project composition and issue behavior also change from year to year. Finding A2. Infer’s 2024 release, not its newest, has the best label-level discrimination, and the newest release broadens fixed-side alerting without keeping the vulnerable-side gain. Release date does not order sustainability; the unit of evaluation is the whole detector condition.

4.3

Semgrep: High F1 from a Proxy That Barely Separates Labels

Semgrep has the highest F1 (0.4784), but its recall (0.4653) is slightly lower than its FPR (0.4790), and its AUROC is 0.4951: a vulnerable case is no more likely to be flagged than a fixed one. Put as row shares, Semgrep flags 46.5% of vulnerable cases and 47.9% of fixed cases. Table 9 breaks the results down by year. The annual F1 is not monotonic: 0.4558 in 2016, 0.4201 in 2021, 0.5234 in 2024, and 0.5083 in 2025. Figure 3(c) shows why the curve rises after 2021. TPR and FPR move up and down together, while AUROC never leaves 0.484–0.511. Two sensitivity checks leave this picture intact. First, removing the 109 zero-target cases leaves 19,635 cases. All of them were negatives (70 false negatives and 39 true negatives), so TP and FP do not change: recall rises to 0.4686, F1 to 0.4801, and AUROC to 0.4958. FPR rises slightly, to 0.4809, because its denominator shrinks. The yearly effect of the removal never exceeds 0.0051 (Table 9), so the result does not come from giving zero scores to empty scopes. Second, restricting the score to rules in the security category, on the same cases and scopes, yields TP=4,515 and FP=4,642 (Table 5), that is, precision 0.4931, recall 0.4578, F1 0.4748, and FPR 0.4698. The score is also concentrated. Table 10 lists the eight rules that emit the most findings and splits each rule’s cases by label. Together they produce 94.2% of the 78,696 qualifying findings. A single generic rule, insecure-use-memset, produces 49,606 of them (63.03%) and fires in 7,769 cases, of which 3,837 are vulnerable and 3,932 are fixed. The rule matches any memset call and does not require sensitive data at the call site. Each of the top eight rules fires on similar numbers of

14

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 9. Annual Study-A results for Semgrep (primary population). “Zero” is the number of zero-target cases in the year; Δ0 is the F1 change when they are removed. Shading marks each metric column’s maximum (blue) and minimum (red). Year Cases TP

FP FN TN Prec. Recall

2016 2017 2018 2019 2020 2021 2022 2023 2024 2025

425 473 466 484 441 361 501 550 547 485

1,951 1,984 1,956 1,951 1,924 1,981 1,999 2,000 1,998 2,000

412 442 436 462 423 359 472 529 548 506

559 550 542 511 538 630 528 471 451 494

555 519 512 494 522 631 498 450 452 515

.4922 .4831 .4834 .4884 .4896 .4986 .4851 .4903 .5005 .5106

.4243 .4456 .4458 .4748 .4402 .3630 .4720 .5290 .5485 .5060

F1 .4558 .4636 .4638 .4815 .4636 .4201 .4785 .5089 .5234 .5083

FPR AUROC Zero .4337 .4768 .4765 .4949 .4579 .3639 .5015 .5500 .5475 .4850

.4967 .4843 .4869 .4932 .4921 .4993 .4916 .4948 .5106 .5041

6 8 19 16 31 5 6 2 5 11

Δ0 .0013 .0012 .0032 .0030 .0051 .0007 .0010 .0002 .0008 .0010

Table 10. The eight Semgrep rules with the most qualifying findings (primary population). “Cases” counts cases with at least one finding of the rule; TP/FP split them into vulnerable and fixed cases. Bars show both counts on one scale ( vulnerable, fixed). Rule (last ID component) insecure-use-memset insecure-use-string-copy-fn insecure-use-strcat-fn incorrect-use-ato-fn unbounded-copy-to-stack-buffer function-use-after-free insecure-use-strtok-fn double-free

Findings

Share

49,606 63.03% 11,467 14.57% 5,181 6.58% 4,293 5.46% 1,184 1.50% 1,002 1.27% 738 0.94% 653 0.83%

Cases

TP

FP

Cases by label

7,769 3,837 3,932 2,454 1,208 1,246 863 440 423 889 442 447 380 201 179 250 125 125 171 95 76 147 73 74

vulnerable and fixed cases, even the two memory-safety rules (function-use-after-free: 125 vs. 125 cases; double-free: 73 vs. 74). A case can appear under several rules, because rules overlap. Finding A3. Semgrep’s F1 of 0.4784 is not a 47.84% vulnerability-detection rate. The finding count barely separates labels (AUROC 0.4951), the annual curve is non-monotonic, and almost two thirds of qualifying findings come from one generic rule that fires equally on vulnerable and fixed code.

5

RQ2: Project-Matched Temporal Change (Study B)

Study B controls one major alternative explanation for Study-A trends (i.e., project and label composition): For every year pair, the sampling procedure selects the same projects and equal vulnerable/fixed counts per project on both sides; subsequent evaluation exclusions may introduce small imbalances. Table 11 reports the pair-equal mean ΔF1 for each gap, where negative values mean lower F1 in the later year, and Figure 5 plots the same means together with the spread across year pairs. For example, the CodeQL R2024 entry at gap 5 (−0.1376) averages five year pairs, each of which was itself averaged over 100 repetitions. A progressively more negative gap profile would indicate a simple monotonic aging pattern; departures from monotonicity instead show that temporal degradation interacts with the particular years and cases being compared. 5.1

CodeQL: Release-Conditional Temporal Loss

R2024 is negative at all nine gaps, from −0.0332 at one year to −0.1618 at nine years. This is the clearest release-conditional temporal-loss pattern in our full matched analysis: once projects and labels are fixed, later-year code receives lower mean F1 under R2024 at every gap. Because the

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

15

Table 11. Study-B mean ΔF1 (later minus earlier) by source-year gap; red cells are negative, blue cells positive. Support columns: Semgrep’s matched projects per pair and evaluated cases per year side. CodeQL Gap Pairs R2021 R2024 R2026

v1.1

Infer v1.2

−.0123 −.0218 −.0267 −.0297 −.0309 −.0159 −.0201 +.0321 +.0510

+.0025 +.0043 +.0270 +.0379 +.0188 +.0075 +.0805 +.1583 +.1912

−.0115 −.0142 −.0165 −.0005 +.0307 +.0153 +.0087 −.0009 −.0517

Mean Δ (later − earlier)

1 2 3 4 5 6 7 8 9

9 8 7 6 5 4 3 2 1

−.0332 −.0495 −.0734 −.1049 −.1376 −.0445 −.0844 −.1221 −.1618

(a) CodeQL

0.1

−0.1

+.0049 +.0178 +.0394 +.0785 +.0674 +.0518 +.1099 +.1680 +.1370

+.0073 +.0124 +.0091 −.0032 +.0057 +.0094 +.0148 +.0382 +.1284

(b) Infer (bands: pair 2.5–97.5%) 0.2

0.0

−0.2

−.0170 −.0293 −.0511 −.0849 −.0954 −.0505 −.0532 −.0144 +.0555

Support (Semgrep) v1.3 Semgrep Projects Cases/side

v1.1 v1.2 v1.3

0.4

32–113 24–89 23–53 27–55 27–34 21–28 19–27 14–20 12

464–1,492 393–1,374 290–754 99–580 92–352 110–284 72–174 49–92 60–62

(c) Semgrep F1 pair min–max ΔF1

Δrecall Δprecision

0.2

0.0 R2021

R2024

R2026

1 2 3 4 5 6 7 8 9 Gap Δt (years)

0.0

−0.1 1 2 3 4 5 6 7 8 9 Gap Δt (years)

1 2 3 4 5 6 7 8 9 Gap Δt (years)

Fig. 5. Study-B mean change by gap (x-axis, years). (b) Bands show the 2.5–97.5% quantiles of the year-pair means (undefined at gap 9, which has one pair). (c) The band shows the minimum and maximum year-pair ΔF1; Semgrep’s Δrecall drives the gap-9 jump. Pair means share projects and years, so the bands are descriptive.

full gap analysis also contains source years predating R2024, we interpret this result as temporal sensitivity; prospective aging is supported specifically by comparisons in which the code follows the frozen detector condition. The magnitude is not monotonic, however. Gap 6 (−0.0445) is smaller than gap 5 (−0.1376), so the data do not support a constant decay rate. R2021 and R2026 are negative at gaps one through seven, after which R2021 turns positive at gaps eight and nine and R2026 at gap nine. Those reversals rest on three year pairs for R2021 (two at gap 8 and one at gap 9) and on one pair for R2026, and gap 9 is the single 2016–2025 comparison. They do not cancel the shorter-gap losses, but they show that one monotone aging law does not fit all releases. R2024 is also the only historical condition that used buildless extraction; our data do not isolate whether that mode contributes to its consistent losses. Variation across gaps also mixes different year pairs and a changing case composition, so the gap means should not be read as a dose–response curve. 5.2

Infer: Opposite Directions across Releases

The matched Infer results invert the Study-A ordering. v1.1 and v1.3 are positive at all nine gaps, whereas v1.2, the best Study-A release, is negative at six. Table 12 adds the spread across year pairs. It is large relative to most means. At gap 5, v1.1 has mean +0.0188 but a pair standard deviation of 0.1032 and pair quantiles from −0.1173 to +0.1335, so year pairs of opposite sign cancel. The lower 2.5% quantile is positive only at gaps 3, 4, 7, and 8 for v1.3 and at gaps 7 and 8 for v1.1; for v1.2 it is never positive. Variation between the 100 repetitions of a single pair is much smaller than variation between pairs. Table 13 gives it for the two pairs we report at repetition level, the adjacent 2016→2017

16

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 12. Infer Study-B results by gap: mean ΔF1 over year pairs, the standard deviation (SD) of the pair means, and their 2.5% and 97.5% quantiles. Gap 9 has a single pair, so its spread is undefined. Blue cells mark a positive lower quantile. Gap

Mean

1 2 3 4 5 6 7 8 9

+.0025 +.0043 +.0270 +.0379 +.0188 +.0075 +.0805 +.1583 +.1912

v1.1 SD 2.5%

97.5%

Mean

− .0417 − .0978 − .0377 − .0844 − .1173 − .0343 +.0407 +.0749 —

+.0480 +.0774 +.1191 +.1038 +.1335 +.0666 +.1416 +.2416 —

− .0115 − .0142 − .0165 − .0005 +.0307 +.0153 +.0087 − .0009 − .0517

.0283 .0589 .0609 .0774 .1032 .0459 .0574 .1240 —

v1.2 SD 2.5%

97.5%

Mean

− .0625 − .0864 − .0772 − .0476 − .0394 − .0200 − .0118 − .0575 —

+.0129 +.0334 +.0280 +.0407 +.0855 +.0580 +.0364 +.0557 —

+.0049 +.0178 +.0394 +.0785 +.0674 +.0518 +.1099 +.1680 +.1370

.0252 .0407 .0410 .0320 .0522 .0376 .0264 .0843 —

v1.3 SD 2.5%

97.5%

− .0265 − .0155 +.0040 +.0026 − .0506 − .0167 +.0536 +.1047 —

+.0413 +.0835 +.0947 +.1532 +.1370 +.1598 +.1777 +.2314 —

.0234 .0347 .0373 .0551 .0786 .0840 .0664 .0943 —

Table 13. Repetition-level spread of Infer ΔF1 for two year pairs (100 repetitions each). Release

Year pair

Mean ΔF1

Repetition SD

2.5%

97.5%

v1.1 v1.1 v1.2 v1.2 v1.3 v1.3

2016→2017 2016→2025 2016→2017 2016→2025 2016→2017 2016→2025

−.0008 +.1912 −.0724 −.0517 −.0108 +.1370

.0016 .0197 .0092 .0462 .0045 .0198

−.0051 +.1756 −.0898 −.1407 −.0183 +.1204

−.0001 +.2185 −.0532 +.0269 −.0010 +.1648

Table 14. Semgrep Study-B results by gap: mean change in precision, recall, and F1, the range of the year-pair mean ΔF1, matched projects per pair, and cases per year side before (selected) and after (evaluated) the gate. Gap Pairs ΔPrec. ΔRecall 1 2 3 4 5 6 7 8 9

9 8 7 6 5 4 3 2 1

+.0060 +.0078 +.0097 +.0115 +.0034 +.0178 +.0248 +.0179 +.0105

+.0084 +.0165 +.0081 −.0196 −.0028 −.0007 −.0044 +.0557 +.2595

ΔF1 Min pair Max pair Projects Selected/side Evaluated/side +.0073 +.0124 +.0091 −.0032 +.0057 +.0094 +.0148 +.0382 +.1284

−.0108 −.0180 −.0241 −.0348 −.0740 −.0243 −.0602 −.0390 +.1284

+.0361 +.0617 +.0537 +.0361 +.0976 +.0438 +.0729 +.1154 +.1284

32–113 24–89 23–53 27–55 27–34 21–28 19–27 14–20 12

482–1,492 404–1,374 298–754 110–580 102–352 114–284 74–174 50–92 62

464–1,492 393–1,374 290–754 99–580 92–352 110–284 72–174 49–92 60–62

pair and the longest 2016→2025 pair. Here the repetition standard deviation is at most 0.0462. The disagreement between Studies A and B therefore comes from which pairs and projects carry the comparison, not from sampling noise. The table also shows one notable decline that the gap means hide: under v1.2, the 2016→2017 pair has mean −0.0724, and its entire 2.5–97.5% repetition range is negative. The same pair is also negative throughout for v1.1 and v1.3, but only by −0.0008 and −0.0108 on average. The apparent growth of v1.1 and v1.3 at gaps seven to nine rests on three, two, and one pairs; we do not read it as compositional analysis improving on newer code. 5.3

Semgrep: Pair Heterogeneity Dominates the Gap Mean

Semgrep is negative only at gap 4 (−0.0032). Table 14 splits each gap mean into its precision and recall components and gives the range of pair means and the support. Precision changes are small and positive at every gap (0.0034 to 0.0248), whereas recall changes are negative at gaps 4 to 7. The selected and evaluated columns show how many cases the label-conflict gate removes after sampling: at gap 4, the smallest side has 110 selected and 99 evaluated cases.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution? −.00 +.01 +.02 −.01 +.01 −.02 +.03 −.04 +.13 (74)

'17

Earlier source year x

(70)

(50)

(55)

(34)

(25)

(21)

(14)

(12)

+.03 +.04 −.01 +.00 −.04 +.04 −.06 +.12 (89)

(74)

(53)

(36)

(30)

(21)

(19)

0.10

(20)

−.01 −.00 −.02 −.03 +.03 +.04 +.07

'18

(113)

(89)

(52)

(40)

(34)

(25)

(27)

0.05

−.01 −.02 −.02 −.00 −.07 −.02

'19

(108)

(62)

(49)

(37)

(27)

(28)

+.01 −.02 +.01 −.01 +.10

'20

(69)

(50)

(43)

(28)

(27)

Mean ΔF1

'16

17

0.00

−.01 +.01 +.03 +.04

'21

(49)

(39)

(32)

(27)

'22

+.04 +.06 +.05

'23

+.03 +.02

(34)

(24) (32)

−0.05

(23) (30)

−0.10

−.01

'24

(33)

'17

'18

'19

'20

'21

'22

'23

Later source year y

'24

'25

Fig. 6. Semgrep Study-B results for all 45 year pairs. Each cell shows the mean ΔF1 over 100 repetitions (two decimals) and, in parentheses, the number of matched projects. Red cells are negative (lower F1 in the later year), blue cells positive. Table 15. The three most negative and three most positive Semgrep year pairs. Early and late F1 are the matched-sample values; the last column is the 2.5–97.5% range over the pair’s 100 repetitions. Year pair

Projects

Selected/year

Early F1

Late F1

Mean ΔF1

Repetition range

2019→2024 2017→2024 2016→2024 2016→2025 2017→2025 2020→2025

27 19 14 12 20 27

106 74 50 62 92 102

.5978 .5656 .5264 .4489 .4388 .4318

.5238 .5054 .4874 .5773 .5542 .5294

−.0740 −.0602 −.0390 +.1284 +.1154 +.0976

−.1126 to −.0251 −.1291 to +.0124 −.1330 to +.0622 +.0068 to +.2395 +.0457 to +.1882 +.0111 to +.1755

The gap means hide large pair-level differences, which Figure 6 shows for all 45 pairs together with the number of matched projects. Support is largest for adjacent years (up to 113 projects for 2018→2019) and smallest for the longest gaps (12 projects for 2016→2025). The three most negative pairs all end in 2024, and seven of the nine pairs that end in 2025 are positive. Table 15 lists the three most negative and three most positive pairs. At gap 5, pair means span −0.0740 to +0.0976 around a mean of +0.0057. For 2017→2024 and 2016→2024, the repetition range even crosses zero, whereas none of the three positive pairs does. The matched-sample F1 values also differ from the population values of Study A: the 2016→2025 pair starts at 0.4489, below the Study-A value for 2016 (0.4558), and ends at 0.5773, above every annual Study-A F1. Support also shrinks quickly with the gap. Gap 1 pairs have 32–113 projects and 464–1,492 evaluated cases per side; gap 9 has 12 projects and 60–62 cases per side. Figure 5(c) and Table 14 show that the gap-9 value is driven by recall (+0.2595) while precision barely moves (+0.0105): the 2025 side receives more positives, but they are not concentrated on vulnerable cases. Since Semgrep’s rules are identical on both sides, we treat the +0.1284 as a property of one 12-project comparison, not as evidence that distance in time improves detection. Matching fixes projects and label counts, but vulnerability type, module, scan scope, and analysis quality still vary within a pair.

18

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 16. Study-C project slopes and pooled cohort metrics. N/Z/P count projects with negative, zero, and positive F1 slopes; “Mean/yr” is the equal-project mean slope. Cohort F1 and AUROC are equal-year means of pooled annual metrics (Semgrep AUROC not computed for cohorts). The cohorts are nested. Primary cohort (≥3 years, 202 projects) N/Z/P N share Mean/yr F1 AUROC

Condition CodeQL R2021 CodeQL R2024 CodeQL R2026 Infer v1.1 Infer v1.2 Infer v1.3 Semgrep

37/139/26 74/93/35 85/67/50 13/178/11 17/172/13 27/152/23 65/51/86

18.3% 36.6% 42.1% 6.4% 8.4% 13.4% 32.2%

−.00170 −.01926 −.01657 −.00267 −.00673 −.00514 +.01426

.1141 .1918 .2760 .0511 .1722 .1231 .4820

Sensitivity cohort (≥5 years, 61 projects) N/Z/P N share Mean/yr F1 AUROC

.4996 15/34/12 .5003 32/13/16 .4981 36/9/16 .4988 4/49/8 .5273 5/50/6 .4983 8/39/14 — 25/9/27

24.6% 52.5% 59.0% 6.6% 8.2% 13.1% 41.0%

+.00039 −.01247 −.01095 +.00538 −.00353 +.00939 +.00286

.1135 .1929 .2801 .0490 .1838 .1106 .4919

.4999 .5012 .4985 .4990 .5286 .4986 —

Table 17. Study-C cohort precision, recall, and FPR (equal-year means of pooled annual values). For CodeQL and for Infer v1.1 and v1.3, recall and FPR are close, as in Study A. Semgrep has no cohort FPR. Primary cohort Precision Recall FPR

Condition CodeQL R2021 CodeQL R2024 CodeQL R2026 Infer v1.1 Infer v1.2 Infer v1.3 Semgrep

.5030 .5073 .4977 .4813 .7196 .4890 .4928

.0677 .1251 .1942 .0279 .1015 .0730 .4753

.0686 .1255 .1982 .0302 .0475 .0765 —

Negative slope

(a) Primary: ≥ 3 years, 202 projects 37 CodeQL R2021 74 CodeQL R2024 85 CodeQL R2026 Infer v1.1 13 Infer v1.2 17 Infer v1.3 27 65 Semgrep

0

20

139

40

.4962 .5121 .4996 .4750 .7238 .4872 .4919

Share of projects (%)

80

Positive slope

15

34 32 36

4 5 8 25

100

.0689 .1240 .2021 .0289 .0541 .0675 —

(b) Sensitivity: ≥ 5 years, 61 projects

86

60

.0687 .1257 .1992 .0269 .1105 .0646 .4961

Zero slope

26 35 50 11 13 23

93 67 178 172 152 51

Sensitivity cohort Precision Recall FPR

0

20

12 16 16 8 6 14

13 9 49 50 39 9

40

27

60

Share of projects (%)

80

100

Fig. 7. Direction of project-level F1 slopes in Study C. Red, gray, and blue mark negative, zero, and positive slopes; numbers inside bars are project counts. Finding B. Exact project-by-label matching reveals a consistent later-year loss for CodeQL R2024, mostly negative short-gap changes for R2021 and R2026, positive changes at every gap for Infer v1.1 and v1.3, small mixed changes for Infer v1.2, and mostly small changes for Semgrep outside the sparse longest gaps. Sparse long gaps and large pair-to-pair differences make a tool-wide “F1 lost per year of code age” unsupportable.

6

RQ3: Within-Project Change (Study C)

Study C asks a narrower question than Study B: as the same project evolves, does its F1 change? Table 16 reports the distribution of project-level F1 slopes and the pooled cohort metrics, and Figure 7 shows the slope directions as shares. To read the first row: under CodeQL R2021, 37 of the 202 primary projects have a negative slope, 139 a zero slope, and 26 a positive slope, and the mean slope is −0.00170 F1 per year. Table 17 adds the cohort precision, recall, and FPR. Slope signs are not significance tests, and the 61 five-year projects also belong to the primary cohort.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

19

Table 18. Decomposition of zero Infer F1 slopes. “All-zero”: F1 is zero in every usable year; “Const. >0”: the same nonzero F1 in every year; “Nonconst. ≈0”: F1 varies, but the slope stays within the zero tolerance.

6.1

Release

Cohort

v1.1 v1.1 v1.2 v1.2 v1.3 v1.3

Primary Sensitivity Primary Sensitivity Primary Sensitivity

Zero slopes

All-zero

Const. >0

Nonconst. ≈0

178 49 172 50 152 39

176 48 167 49 144 38

2 1 3 0 8 1

0 0 2 1 0 0

CodeQL: Higher Level, More Declining Projects

The number of projects with negative descriptive F1 slopes (i.e., declining projects) doubles from R2021 to R2024 and grows again in R2026 (37, 74, 85), and the mean slope is far more negative under the two later releases (−0.00170, −0.01926, −0.01657). At the same time, pooled cohort F1 rises from 0.1141 to 0.2760 while AUROC stays within 0.0019 of 0.5. Table 17 shows the reason for the rising level: as in Study A, recall and FPR grow together and stay almost equal in every release (for example 0.1942 and 0.1982 for R2026). Level and trajectory are different properties. A release can flag more cases overall and still lose its signal faster as individual projects evolve. The nested five-year cohort keeps the ordering (15, 32, and 36 of 61 projects decline), so the difference between R2021 and the later releases does not come from short trajectories alone. The cohort metrics and the mean project slope use different weights and should not be read as two estimates of the same quantity. 6.2

Infer: Flat because Empty

Most Infer slopes are zero: 178, 172, and 152 of 202 primary projects. Table 18 decomposes these flat trajectories and shows that they rarely represent stable detection. For v1.1, 176 of the 178 flat projects have F1 equal to zero in every usable year, and only two have a constant nonzero F1; the numbers are 167 of 172 for v1.2 and 144 of 152 for v1.3. A flat slope thus usually means that Infer never yields a label-level signal for the project. A zero F1 does not mean that Infer reported nothing: the project may still receive findings on fixed cases or outside the scored files. Every primary and sensitivity project has a fitted F1 slope under all three releases. AUROC slopes are less complete, because AUROC is undefined when one label is missing from a project-year after exclusions: three primary projects and one sensitivity project have no AUROC trend under v1.1 and v1.3, while v1.2 has all of them. Projects with a nonzero F1 slope use between three and nine usable years (five to nine in the sensitivity cohort), and their endpoints cover all of 2016–2025. Among nonzero trajectories, both directions occur under every release. The primary mean slopes are mildly negative, while the sensitivity cohort turns v1.1 and v1.3 positive (+0.00538 and +0.00939). That sign change reflects which projects supply long histories rather than an improvement, because the restricted cohort is selected on the number of usable years. Figure 8(a) shows the pooled primary-cohort F1 by year: the releases cross repeatedly, v1.2 peaks at about 0.31 in 2022, and all three fall to between about 0.01 and 0.07 by 2025. The cohort precision of v1.2 (0.7196) is far above that of the other releases, again because v1.2 flags few fixed cases (Table 17). 6.3

Semgrep: A Rising Curve That Projects Do Not Share

Table 19 gives the Semgrep cohort values for every year, and Figure 9 plots their precision, recall, and support. The primary cohort falls from F1=0.4826 in 2019 to 0.4155 in 2021 and then rises to 0.5321 in 2024; the five-year cohort follows the same shape (0.5118, 0.4007, 0.5335). Two facts temper this rise. First, it is carried by recall, which dips to 0.3565 in 2021 and peaks at 0.5683 in

20

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

(a) Infer: F1, primary cohort v1.1

0.3

v1.2

(b) Semgrep: F1 and recall F1 ( ≥ 3 y) F1 ( ≥ 5 y)

v1.3

Recall ( ≥ 3 y) Recall ( ≥ 5 y)

150

0.6

0.2

≥5 y

≥3 y

100

0.5

0.1

(c) Semgrep: projects

50

0.4

0.0

0 '16

'18

'20

'22

'24

'16

'18

'20

'22

'24

'16

'18

'20

'22

'24

Fig. 8. Study-C annual cohort profiles by source-code year (x-axis), pooled over projects and averaged over 100 repetitions. (b) Solid lines: primary cohort; dashed lines with open markers: nested sensitivity cohort. (c) The number of contributing Semgrep projects shrinks sharply after 2020. Table 19. Semgrep Study-C annual values for both cohorts: contributing projects, selected cases per repetition (balanced before the label-conflict gate), the range of evaluated cases over repetitions, and pooled precision, recall, and F1 averaged over 100 repetitions. Year Proj.

Primary cohort (≥3 years) Sel. Eval. Prec. Recall

2016 2017 2018 2019 2020 2021 2022 2023 2024 2025

86 97 130 127 121 92 66 59 45 46

1,454 1,442 1,460 1,478 1,138 1,682 1,688 1,774 1,830 1,652

Total

869 15,598

1,415–1,425 1,432–1,433 1,430–1,433 1,447–1,452 1,094–1,096 1,672–1,673 1,687–1,688 1,774 1,828 1,652

.5006 .4796 .4828 .4876 .4954 .4979 .4849 .4866 .5002 .5124

.4759 .4666 .4687 .4826 .4566 .4155 .4848 .5084 .5321 .5289

34 41 53 53 52 43 37 35 25 23

Sensitivity cohort (≥5 years) Sel. Eval. Prec. Recall 980 972–974 .4976 1,132 1,124 .4833 1,004 980–982 .4828 1,100 1,075–1,078 .4896 790 752–753 .4930 1,508 1,498–1,499 .4886 1,552 1,551–1,552 .4848 1,706 1,706 .4859 1,772 1,770 .5007 1,582 1,582 .5127

.5165 .4753 .5179 .5361 .4336 .3396 .4879 .5359 .5710 .5468

F1 .5068 .4793 .4998 .5118 .4614 .4007 .4863 .5097 .5335 .5292

396 13,126

(a) Precision and recall Precision ( ≥ 3 y) Precision ( ≥ 5 y)

.4536 .4543 .4553 .4778 .4235 .3565 .4848 .5323 .5683 .5466

F1 Proj.

(b) Contributing projects (c) Selected cases per repetition

Recall ( ≥ 3 150 y) Recall ( ≥ 5 y)

≥5 y

≥3 y

2000

≥5 y

≥3 y

100

0.6 0.5 0.4

1000

50 0 '16

'18

'20

'22

'24

0 '16

'18

'20

'22

'24

'16

'18

'20

'22

'24

Fig. 9. Semgrep Study-C cohorts by source-code year (x-axis). (a) Pooled precision and recall; solid lines: primary cohort, dashed lines with open markers: sensitivity cohort. (b) Projects that contribute to each year. (c) Selected cases per repetition. Fewer projects carry more cases in later years.

2024, while precision stays within 0.479–0.513. Second, the number of contributing projects falls from 130 in 2018 to 45 in 2024, while the number of selected cases per repetition rises from 1,460 to 1,830. In 2024, 1,830 cases come from 45 projects (about 41 per project), against 1,454 cases from 86 projects in 2016 (about 17): later years reflect fewer, larger projects. The annual values pool cases across projects; they are not equal-project means and do not imply that most projects followed the same path over the decade. At the project level, 86 primary projects (42.6%) have positive slopes, 65 (32.2%) negative ones, and 51 (25.2%) flat ones, with a mean of +0.0143 F1 per year. A higher pooled F1 in later years therefore does not mean that every project improved. In the five-year cohort the split is nearly

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

21

Fig. 10. Distribution of Semgrep project F1 slopes (F1 per source year) in the primary and sensitivity cohorts; the dashed line marks zero. Slopes within ±10−10 count as flat. Table 20. The nine tied largest Semgrep Study-C declines (F1 1.0→0.0) with a representative vulnerable case from each endpoint. The late cell gives TP/FP/FN/TN per repetition (every early cell is 1/0/0/1). Diagnostics: P = partial parsing, T = rule timeout, S = syntax error, — = none. Project

Years

Late cell Early representative (TP): CVE, findings, diag., file

Late representative (FN): CVE, diag., file

MuJS FreeRDP libreswan MilkyTracker MilkyTracker OpenBSD Varnish Varnish libxml2

2018→2022 2021→2025 2019→2020 2019→2020 2019→2022 2019→2024 2017→2019 2017→2020 2019→2020

0/1/2/1 0/0/1/1 0/0/1/1 0/0/1/1 0/0/1/1 0/0/1/1 0/0/2/2 0/0/1/1 0/1/4/3

CVE-2022-44789; —; jsobject.c CVE-2025-68118; P; certificate_data.c CVE-2020-1763; P; ikev1.c CVE-2020-15569; —; ChannelMixer.cpp CVE-2022-34927; T; SectionSamples.cpp CVE-2025-30334; S; if_wg.c CVE-2019-20637; —; vdef.h CVE-2021-36740; P; cache_http2.h PRIMEVUL-2020-0e1a. . . ; S; parser.c

CVE-2018-6191; 1; P, T; jsparse.c CVE-2021-37594; 2; P; timezone.c CVE-2019-12312; 9; P, T; kernel.c CVE-2019-14464; 2; —; LoaderXM.cpp CVE-2019-14464; 2; —; LoaderXM.cpp CVE-2019-8460; 21; P, S, T; drm_linux.c CVE-2017-8807; 6; P; cache_acceptor.c CVE-2017-8807; 6; P; cache_acceptor.c CVE-2019-19956; 25; P, T; relaxng.c

even (27 versus 25) and the mean drops to +0.0029. Figure 10 shows the full slope distributions. Most slopes lie within ±0.1 F1 per year. The primary cohort has tails beyond ±0.3, whereas no project in the five-year cohort exceeds ±0.2; extreme slopes thus come from short histories. Nine project-year pairs tie for the largest Semgrep decline, from F1=1.0 to 0.0 (Table 20). In all of them, every early cell has exactly one true positive and one true negative per repetition, so a single changed prediction erases the whole F1, and in these selected cells the counts do not change across the 100 repetitions. The late cells differ. MuJS 2022 has two false negatives and one false positive, libxml2 2020 has four false negatives and one false positive, Varnish 2019 has two false negatives, and the others have one false negative and no false positive. The same F1 drop therefore aggregates different error patterns. The representative cases in the table are vulnerable cases from the early and late cells; they are not the same vulnerability seen twice, and the source year of a snapshot can differ from the year in its CVE identifier (the 2020 Varnish case carries CVE-2021-36740). Because MilkyTracker’s and Varnish’s pairs share their start year, the nine rows are not independent. After removing shared cells, these pairs give the 42-case queue of §7, with 7 TP, 2 FP, 14 FN, and 19 TN. Finding C. Within-project evidence separates detection level, trajectory, and the presence of any signal. Later CodeQL releases have higher F1 but more declining projects; most flat Infer projects never produce a positive F1; and Semgrep’s positive mean slope nearly disappears when longer histories are required. The largest drops start from early cells with only two cases.

7

RQ4: Failure Mechanisms behind the Largest Drops

Studies A–C show where scores change; RQ4 asks what the recorded evidence says about why. Table 21 summarizes the mechanisms we found, each with concrete cases and the limit of what the

22

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 21. Mechanisms observed in the selected drops (RQ4) and execution records (RQ5). “Observed” does not imply population prevalence. Mechanism

Concrete evidence (tool)

Scope mismatch

HAProxy 2019 alerts in proto_htx.c, fix in mux_h2.c (CodeQL); libiec61850 alerts in control.c, fix in cotp.c (Infer); 11 of 20 vulnerable/fixed groups scored on disjoint files (Semgrep) Incidental 17 memset findings in libxml2 relaxng.c; two memset findings in positive MilkyTracker XModule.h; atoi and sizeof-pointer findings (Semgrep); DEAD_STORE warnings (Infer) Fixed-side HAProxy 2023 overflow check present before and after the fix (CodeQL warning R2024); 10 file-level hits on Slurm’s fixed 2020 snapshot (Infer v1.2); six sizeof-pointer findings on the MuJS fixed side (Semgrep) Rule/query Nagios NULL entry guard vs. nullness predicates; BlueZ bonding check vs. authentication queries (CodeQL); Varnish CRLF-length invariant vs. API mismatch patterns (Semgrep) Trace mismatch libxml2 use-after-free traces start at the C assignment delete = cur; (Semgrep) Parse/extract Partial parsing and file-wide syntax errors (Semgrep); missing headers, limits incomplete types (CodeQL); empty or partial capture (Infer) Rule-specific Two string-mismatch rules time out on MilkyTracker timeout SectionSamples.cpp, while the repair is in LoaderXM.cpp (Semgrep) Build failure Query cost Repeated snapshot Small cells

What the evidence does not show That off-repair alerts are unrelated to the CVE; data flow can cross files That the warnings are false; only that they do not reach the repair That the warning is wrong for every defect That no configured rule could ever match The parser’s internal cause Which construct was lost

The cause of a miss in another file or on the other label No compiler seen by a traced build; buildless discovery that ignores GCC The cause of every (CodeQL); unmaterialized pinned submodules (Infer) historical failure row Two range-analysis queries consume a 12-hour budget on BlueZ (CodeQL) Universal nontermination or OOM Two libxml2 identifiers select the same commit pair and parser.c change Two independent (Semgrep) confirmations Every selected early cell has one TP and one TN (all tools) Population-wide deterioration

evidence shows. The selected cases are extremes, so the table describes which mechanisms exist, not how often they occur. The 42 Semgrep cases and the small, shared cells of the other queues cannot estimate the prevalence of any cause in the full population. 7.1

The Changed-File Proxy Can Manufacture a Drop

CodeQL. Table 22 lists the selected CodeQL and Infer drops. Each release column gives F1 at the earlier and later endpoint, and the last two columns repeat the selecting release’s F1 under the two location ablations. In the first row, for example, CodeQL R2021 falls from 1 to 0 on HAProxy 2019→2022, but both endpoints are already 0 when only alerts in the repaired file count. Among the 39 scored CodeQL findings in the selected cases, 22 lie outside the verified repair file, 11 lie in the repair file but outside the changed hunk, and only six touch a changed line. These counts come from 27 extreme cases and do not estimate a population false-positive rate. The three HAProxy ties (§2.3) and BlueZ 2016→2020 all disappear once alerts must lie in the repaired file: in each, the earlier positive alert lies outside the paired repair file. For the HAProxy pairs, the later releases did not improve the signal; the earlier signal simply disappeared. Nagios is the exception. Its 2016 signal survives even the changed-line restriction under all three releases, and its 2018 NULL-dereference repair is genuinely missed (§7.3). For Unbound, a 2024 case lacks a paired repair, so we claim no ablated value for its later endpoint.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

23

Table 22. Selected maximum drops for CodeQL and Infer: F1 at the earlier→later endpoint for each release, and the selecting release’s F1 when only scored alerts in a verified repair file or on a changed repair line are kept. Red marks a drop; n/r: no ablation was computed for this supplementary pair; n/a: no paired repair at the later endpoint; “no drop”: both endpoints are equal under the ablated score. Tool

Project, years

Selected by 2021 release 2024 release 2026 release Repair file Repair line

CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL

HAProxy 2019→2022 HAProxy 2019→2023 HAProxy 2019→2024 Nagios 2016→2018 Nagios 2016→2017 BlueZ 2016→2020 BlueZ 2016→2023 Unbound 2020→2024

R2021 R2021 R2021 R2021 R2024 R2024 R2024 R2024

1→0 1→0 1→0 1→0 1→.67 0→0 0→0 0→0

0→0 0→.67 0→0 1→0 1→0 1→0 1→0 1→0

0→0 0→0 0→0 1→0 1→0 1→0 1→0 1→0

0→0 0→0 0→0 1→0 n/r 0→0 n/r 0→n/a

0→0 0→0 0→0 1→0 n/r 0→0 n/r n/a

Infer Infer Infer Infer Infer Infer Infer Infer Infer

libiec61850 2020→2023 firejail 2019→2022 acrn-hypervisor 2020→2021 Slurm 2016→2020 Slurm 2017→2020 mosquitto 2018→2021 mosquitto 2018→2024 folly 2018→2019 folly 2018→2021

all v1.2 v1.2 v1.2 v1.2 v1.3 v1.3 v1.3 v1.3

1→0 0→0 0→0 0→0 0→0 0→0 0→0 0→0 0→0

1→0 1→0 1→0 1→0 1→0 0→0 0→0 0→0 0→0

1→0 0→0 0→0 0→0 0→0 1→0 1→0 1→0 1→0

0→0 1→0 1→0 1→0 1→0 no drop no drop no drop no drop

0→0 0→0 0→0 0→0 0→0 no drop no drop no drop no drop

Table 23. Confusion counts (TP, FP, FN, TN) per repetition for the CodeQL cells behind Table 22. FP means disagreement with the fixed label, not a manual judgment. Project and year HAProxy 2019 HAProxy 2023 Nagios 2016 Nagios 2018 BlueZ 2016 BlueZ 2020 Unbound 2020

Cases

R2021

R2024

R2026

2 2 2 2 2 4 2

(1, 0, 0, 1) (0, 0, 1, 1) (1, 0, 0, 1) (0, 0, 1, 1) (0, 0, 1, 1) (0, 0, 2, 2) (0, 0, 1, 1)

(0, 0, 1, 1) (1, 1, 0, 0) (1, 0, 0, 1) (0, 0, 1, 1) (1, 0, 0, 1) (0, 0, 2, 2) (1, 0, 0, 1)

(0, 0, 1, 1) (0, 1, 1, 0) (1, 0, 0, 1) (0, 0, 1, 1) (1, 0, 0, 1) (0, 0, 2, 2) (1, 0, 0, 1)

Table 23 shows the confusion counts behind the CodeQL endpoints. They are the same in all 100 repetitions, so each count holds per repetition and is not multiplied by 100. Similar F1 changes hide distinct patterns. Under R2024, HAProxy 2023 has one true and one false positive (F1=2/3); under R2026 it loses the true positive and keeps the false positive. Nagios goes from one TP and one TN in 2016 to one FN and one TN in 2018 under every release. BlueZ 2020 is the only selected cell with four cases per repetition. We also checked whether the final security filter could have removed an existing alert. Of the 81 case-by-release records, 68 have a zero score and no changed-file finding of any kind before that filter, so the filter did not discard an alert in a scored file. Infer. Infer exposes a complementary failure. Of the 620 scored findings in its selected cases, 610 lie in a verified repair file, so file-level filtering barely changes the score; yet only two lie on a changed repair line. Of the 27 selected pair–release combinations, 11 show a drop under the original score, four still show it under the repair-file restriction (firejail, acrn-hypervisor, and both Slurm pairs, all under v1.2), and none survives the changed-line restriction. File-level agreement is therefore necessary but not sufficient: an alert elsewhere in a large repaired file can create the early true positive or the later false positive that drives a drop. The scored findings are also not one kind of security finding. They include dead stores, buffer overruns, integer overflows, null dereferences, and large allocations. Table 24 gives the raw scores at every endpoint, and Table 25 shows where the scoring findings lie.

24

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 24. Infer changed-file scores at the selected endpoints, as vulnerable/fixed case scores per release. “(e)” marks an empty_capture observation: the run produced a legal report, but capture recorded no source files. Project

Year

v1.1

v1.2

v1.3

What the endpoint shows

libiec61850 libiec61850 firejail firejail Slurm Slurm Slurm

2020 2023 2019 2022 2016 2017 2020

2/0 0/0 0/0 (e) 0/0 (e) 0/0 (e) 0/0 (e) 0/0 (e)

2/0 0/0 3/0 0/0 298/0 298/0 0/10

2/0 0/0 0/0 (e) 0/0 (e) 0/0 (e) 0/0 (e) 0/0 (e)

nonempty positive vulnerable endpoint in all releases zero score in all releases; captures are nonempty only v1.2 captured and scored the snapshot zero score; v1.1 and v1.3 captured nothing only v1.2 captured and scored the snapshot same vulnerable code as 2016 (CVE-2016-10030) v1.2 keeps ten file-level hits on the fixed case

Table 25. Where the scoring Infer findings lie, and what the verified repair changed. Endpoint

Primary finding locations

Score and issue types

Verified repair change

libiec61850 2020, vulnerable

control.c:1223 and 1382, in Control_readAccessControlObject and Control_writeAccessControlObject fs_lib.c:265, preproc.c:156, sandbox.c:947

2; DEAD_STORE

cotp.c:723 adds a packet-size guard before the error exit

3; two INTEGER_ OVERFLOW_L2, one INFERBO_ALLOC_MAY_ BE_BIG 298 of 11,875 findings; buffer overrun, integer overflow 10 of 9,236 findings; null dereference, buffer overrun, dead store

reorganizes seccomp paths, cleanup, and read-only protection across four files

firejail 2019, vulnerable Slurm 2016, vulnerable

req.c, e.g., lines 294, 315, 346, 466

Slurm 2020, fixed

x11_util.c, ten lines between 180 and 281

req.c adds _open_as_other and replaces a direct open x11_util.c creates a protected temporary xauth file

libiec61850 shows the mechanism in detail. All three Infer releases report the same two DEAD_STORE findings in the 2020 vulnerable snapshot, about an unused write to nextVarName in src/iec61850/server/mms_mapping/control.c. The paired CVE-2020-15158 repair adds the guard if (self->packetSize <= bufPos) goto exit_error in src/mms/iso_cotp/cotp.c, and the recorded vulnerable commit is the parent of the fixed commit. The two scores nevertheless refer to different commit comparisons. The vulnerable case’s changed-file list describes that snapshot’s own parent-to-commit change in control.c, whereas the fixed case’s list describes the repair in cotp.c. The case score is two in every release, although the total number of findings in the three reports differs a lot. Under the repair-file restriction, the 2020 endpoint becomes zero for every release, and the “drop” vanishes. The case shows that the original contrast depends on the proxy. It does not show a checker regression, and we did not judge whether the two dead stores are real defects in the code. Slurm shows the opposite situation. The 2016 and 2017 vulnerable endpoints hold the same vulnerable code for CVE-2016-10030, whose repair adds _open_as_other in src/slurmd/slurmd/req.c, opens the file after changing credentials in a child process, and replaces the direct open in _prolog_error. Under v1.2, the vulnerable report holds 11,875 findings, 298 of them buffer-overrun and integer-overflow warnings in req.c. The fixed 2020 endpoint (CVE-2020-27746) holds 9,236 findings, ten of them in x11_util.c, where the repair creates a mode-protected temporary xauth file, unlinks it, and frees the argument array. That 2020 cell has one false positive and one false negative per repetition. At file level the drop persists. At line level the vulnerable score becomes zero and the fixed score two, so both endpoints have F1=0 and the comparison carries no information. The firejail 2019 case (CVE-2019-12589) is similar: its report holds 35 findings, three of which hit the scored files, and none of the three lies on a changed line. The repair removes standalone seccomp paths, introduces a seccomp directory, creates it, and makes it read-only. For acrn-hypervisor, a

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

25

single vulnerable-side hit carries the drop. In the later acrn-hypervisor (2021) and mosquitto (2024) cells, each label has two cases per repetition. Semgrep. Semgrep’s queue shows the same problem at the level of benchmark construction. Its 42 cases form 20 vulnerable/fixed groups that share project, year, and identifier, plus two unmatched FreeRDP records. In 19 of the 20 groups, the vulnerable target is the parent of the fixing commit, but only nine groups score the same changed files on both sides; eleven use disjoint files, ten of them with adjacent commits. A vulnerable case can be scored on the files changed into that snapshot, while the fixed case is scored on the files changed by the fix. In MuJS CVE-2022-30975, for example, the vulnerable commit is the parent of the fixing commit, but the scored vulnerable file pp.c already contains the if (f) fclose(f) guard, because the recorded diff added it. The fix itself adds if (prog) checks before AST-pointer dereferences in jsP_dumpsyntax and jsP_dumplist in jsdump.c. The false negative therefore cannot test whether Semgrep would detect the missing guard, and the fixed-side false positive comes from six sizeof-pointer-type findings unrelated to the repair. A different file set does not by itself mean broken commit lineage. It means that a valid label can be paired with a scored file that does not contain the repaired operation. 7.2

A Finding Can Be Real yet Irrelevant to the Label

Semgrep. All nine positive Semgrep cases in the selected queue (seven TP and two FP) are labellevel: none of them reaches the labelled repair mechanism. Table 26 summarizes the queue by project. Most early positives come from generic memset, string-copy, or strcat rules, and others from atoi, stat-before-action ordering, or sizeof-pointer rules, in code that the fix does not touch. • Varnish CVE-2017-8807. Six generic memset findings appear in a transport-identification change. The fix adds if (l > ll) l = ll before a memcpy in cache_fetch.c, which lies outside the scope of the vulnerable case. • MilkyTracker CVE-2019-14464. The case scans LoaderXM.cpp and XModule.h after a change that adds instrument and sample-count checks; the two findings are memset calls in the header. The fixing commit adds S3M instrument and pattern-count checks in LoaderS3M.cpp. The commits are adjacent; the file sets do not overlap. • OpenBSD CVE-2019-8460. Twenty-one memset and sizeof-pointer findings come from a broad file-offset locking change. The fixing record, whose parent is the vulnerable target, adds a limit of 128 SACK holes in two other files. The locking diff must not be read as the CVE repair. • MuJS CVE-2018-6191. One strcat finding at line 42 of jsparse.c comes from a change to recursion-depth bookkeeping. The fixing record clamps an exponent in jsdtoa.c, a file absent from the vulnerable scope. • FreeRDP CVE-2021-37594. Two generic copy and memory findings come from a timezone.c change that replaces getenv. The fixed record selected for the same year belongs to a different CVE (CVE-2021-37595, bounds checks in wf_cliprdr.c), so the two are independent cases rather than a before and after view of one repair. • libreswan CVE-2019-12312. Nine findings (string and memory functions, atoi, and stat-beforeaction ordering) come from a hardware-offload change across three files. The fixing record changes ikev2_send.c, and the two commits are not in a direct parent relation; this is the one group of 20 that fails that check. • libxml2 CVE-2016-3709. The second false positive is a memset finding on the fixed side, unrelated to the URI and HTML escaping change in HTMLtree.c. The libxml2 CVE-2019-19956 case is the most telling. The vulnerable record scores relaxng.c after a change that adds NULL checks; the selected fixing commit is a memory-leak repair in parser.c. Its 25 findings comprise 17 generic memset warnings and eight local-variable-new-delete

26

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 26. Semgrep qualitative queue (42 cases) by project. V/F: vulnerable/fixed cases; outcomes: TP/FP/FN/TN. Project

Years

V/F Outcomes Principal evidence

MuJS

’18, ’22

3/3

1/1/2/2

FreeRDP libreswan MilkyTracker OpenBSD Varnish libxml2

’21, ’25 ’19, ’20 ’19, ’20, ’22 ’19, ’24 ’17, ’19, ’20 ’19, ’20

2/2 2/2 3/3 2/2 4/4 5/5

1/0/1/2 1/0/1/2 1/0/2/3 1/0/1/2 1/0/3/4 1/1/4/4

Incidental strcat; already-guarded scanned pp.c; separate jsdump.c repair; property-semantics FN Different 2021 identifiers; certificate-length fix with partial parsing Generic findings; no 2019 parent link; state access near a macro diagnostic Disjoint scope; lifetime defect outside scope; string-rule timeouts Positives away from the SACK repair; mbuf layout fix with syntax errors Disjoint repairs; macro API observation; aligned HTTP length invariants UAF traces from C assignments; syntax errors; escaping FP in another file

(a) MilkyTracker, before fix

(b) Varnish, before fix

(c) libxml2: trace sources

205 if (mixer) 206 delete mixer; ... 208 if (player) 210 if (mixer->isActive() && !mixer->isDeviceRemoved(player)) 211 mixer->removeDevice(player); 212 delete player;

131 if (!vct_iscrlf(r, htc->rxbuf_e)) { ... 136 q = r; 137 assert(r < htc->rxbuf_e); 138 r = vct_skipcrlf(r, htc->rxbuf_e); 139 if (r >= htc->rxbuf_e) 140 break;

7037 xmlNodePtr cur, delete; ... 7062 delete = cur; (3 traces) ... 7418 delete = cur; (5 traces)

after fix

after fix

rule source patterns

206 if (player) 208 if (mixer && mixer->isActive() && !mixer->isDeviceRemoved(player)) 209 mixer->removeDevice(player); 210 delete player; ... 213 if (mixer) 214 delete mixer;

131 i = vct_iscrlf(r, htc->rxbuf_e); 132 if (i == 0) { ... 137 q = r; 138 r += i; 139 assert(r <= htc->rxbuf_e); 140 if (r == htc->rxbuf_e) 141 break;

delete $LOCAL_VAR delete[] $LOCAL_VAR

Fig. 11. Source-level evidence from the Semgrep queue (red: pre-fix; green: post-fix; gray-blue: trace or rule). (a) A real use-after-free in PlayerGeneric.cpp that the vulnerable case never scans. (b) An aligned no-finding case: the fix keeps the CRLF length and accepts the exact buffer end. (c) Traces that start from a C assignment, matched by C++ delete patterns. Indentation simplified; “...” marks omitted lines.

findings, the latter reporting a use after a C++ delete. Figure 11(c) shows what the traces actually start from: xmlRelaxNGCleanupTree declares xmlNodePtr cur, delete;, and every trace names an assignment delete = cur; (line 7418 for five traces, 7062 for three) as the release, whereas the rule’s sources are delete $LOCAL_VAR and delete[] $LOCAL_VAR. The trace sinks are later accesses to cur. The real cleanup happens elsewhere, through xmlUnlinkNode(delete) and xmlFreeNode(delete). The use-after-free reading is thus not supported at the nominated locations, which is consistent with C code being matched by a C++ pattern. The case still counts as a true positive, because any qualifying finding suffices. The same file also records two rule timeouts and partial parsing at line 667. Table 27 sets the inspected rules against what they actually test. It explains why the positives above say little about the repairs. Most rules check for an API call or a local pattern. The reviewed patches instead change how pointer position, length, allocation, and state relate. CodeQL and Infer. The same gap between finding and label occurs for the other tools. Infer’s libiec61850 and firejail positives were discussed above. For CodeQL, changes in the query pack alter which proxy alerts exist. The vulnerable BlueZ CVE-2016-7837 case scores 0, 4, and 4 across releases. Under R2024, three of the four alerts come from cpp/missing-check-scanf at lines 3052, 3058, and 3070 of src/device.c, and one from cpp/overrunning-write at line 403. The scanf query follows an uninitialized scanf-like output to an unguarded use within the same function, and it is absent

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

27

Table 27. What the inspected Semgrep rules test, and what that means for the selected cases. Rule or rule class

What the rule tests

Consequence for the selected cases

insecure-use-memset

Presence of a memset call; no sensitive data required Calls to strcat/strncat or strcpy/strncpy

Explains many incidental TP and FP signals; in xmllint.c the repair itself adds a memset Do not test exponent bounds, protocol framing, or property-definition semantics

Selected local sizeof/pointer expressions Enumerated library return values used in restricted dereference or call patterns

MuJS fixed-side findings do not show the missing AST-pointer guards Not a general checker for every nullable project object, but the configuration does contain null-use checks The eight libxml2 traces nominate assignments to a C identifier, not a release Their MilkyTracker timeouts say nothing about the loader bounds repair Absent from the inspected generic API patterns; other rules remain untested

insecure-use-strcat-fn, insecure-usestring-copy-fn sizeof-pointer-type null-library-function

A deletion source followed by use of a local pointer Taint between narrow and wide string APIs Protocol and layout invariants Relations among pointer, length, allocation, and state in the patches local-variablenew-delete String-mismatch rules

from the saved R2021 query list. The fix, however, bounds a loop in tools/csr.c. Rule selection thus contributes to the score change, but extraction differences and incomplete invocation records prevent assigning the whole change to it. Unbound CVE-2020-28935 scores 0, 8, and 4. The eight R2024 alerts are four cpp/use-after-free results in the generated util/configparser.c and four cpp/path-injection results in util/configlexer.c, while the fix hardens pidfile handling in daemon/unbound.c. The 2024 Unbound cell also contains different vulnerabilities from the 2020 cell, so the contrast mixes alert availability with defect composition; it is not one CVE becoming harder to detect as time passes. Finding Q1. Across all three tools, the selected label-level positives often come from alerts whose location or semantics do not reach the labelled repair. File-level agreement is especially misleading for Infer, where 610 of 620 selected findings lie in a repair file but only 2 touch a changed line. Temporal F1 can therefore change because incidental alerts come and go, even when coverage of repair mechanisms does not.

7.3

No Finding Has Several Possible Explanations

A zero score can mean that the relevant file was not scanned, that the parser or extractor did not represent a needed construct, that no rule or query encodes the defect, or that the analysis encodes it but reported no path. The selected evidence separates these layers where possible. Semgrep. The Semgrep queue has 33 no-finding cases: 14 false negatives and 19 true negatives. Of the 14 false negatives, five have changed-file scopes disjoint from their fixing record (two of these also carry diagnostics), seven have aligned scopes with diagnostics, and two have aligned scopes without diagnostics; nine of the 14 thus carry at least one diagnostic. A true negative here only means that no finding appeared on a fixed case; it does not show full analysis of the file. Scope. MilkyTracker CVE-2020-15569 belongs to the disjoint group. Figure 11(a) shows a genuine lifetime defect: the destructor deletes mixer at line 206 and then calls mixer->isActive() and mixer->isDeviceRemoved(player) at line 210; the fix adds a null check and moves the deletion after the last use. The vulnerable case, however, scores ChannelMixer.cpp after its parent’s change to mixerNumActiveChannels, with no finding and no diagnostic. The false negative cannot tell whether Semgrep’s lifetime rules would catch the destructor. MilkyTracker CVE-2022-34927 is also disjoint: it scans SectionSamples.cpp, where two narrow/wide string-mismatch rules time out, while the repair checks instr[y].size before computing instr[y].size - 4 in LoaderXM.cpp.

28

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

The timeouts show that those rules did not finish on that file, but they cannot explain a miss in another file. Varnish CVE-2019-20637 shows a third variant. The case scores include/vdef.h, whose bstrcpy macro (lines 60–64) contains an assertion followed by strcpy(dst, src); no finding and no diagnostic appear, although the string-copy rule targets exactly that call. The adjacent fixing commit resets req->err_code and req->err_reason in cache_req_fsm.c, so the macro is not the repair. Why the copy rule stays silent inside the macro is an open question; macro handling is one hypothesis. Aligned, no diagnostics. The two aligned, diagnostic-free cases are the cleanest candidates for a rule–target mismatch. In Varnish CVE-2019-15892 (Figure 11(b)), both snapshots scan cache_http1_proto.c. The fix keeps the length returned by vct_iscrlf, advances the pointer by exactly that length, changes the assertion to allow equality with the buffer end, and stops there. The inspected API-presence patterns do not express this pointer–length invariant. MuJS CVE-2022-44789 replaces js_setproperty calls with js_defproperty calls in jsobject.c, for example js_setproperty(J, -2, "value") with js_defproperty(J, -2, "value", 0), a JavaScript property-semantics distinction that no inspected rule models. Both sides of both cases have zero findings. They remain candidate explanations; they do not prove that no rule in the configuration could ever detect these defects. Aligned, with diagnostics. Here the diagnostics weaken the negative result without naming the missed construct. • OpenBSD CVE-2025-30334 changes mbuf allocation in if_wg.c (allocate len + max_hdr, align the payload, set lengths explicitly); both sides report syntax errors from line 1. • libreswan CVE-2020-1763 changes an error-reporting path around line 2132 of ikev1.c. The vulnerable side reports partial parsing at a token-pasting macro near line 276. Only the fixed side records string-mismatch rule timeouts, so the timeouts cannot explain the vulnerable-side miss. • FreeRDP CVE-2025-68118 tightens a hostname-length check in certificate_data.c: the patch passes the buffer capacity minus one, rejects a length that reaches the capacity, and bounds a strdup. Both sides report partial parsing near a formatted string at line 87. The inspected copy patterns do not test such producer–capacity relations, and whether rule coverage or parsing caused the miss remains unresolved. • Varnish CVE-2021-36740 parses Content-Length in the HTTP/2 code, counts body bytes, and rejects mismatches; both sides report partial parsing around generated headers and enumerations. • libxml2 CVE-2020-24977 zeroes four bytes at the end of the input buffer in xmllint.c before encoding entities. The repair itself adds a memset call, which the generic memset rule should match, yet no finding is reported; the file carries file-spanning syntax errors. The miss can therefore not be blamed on the absence of an unsafe-API pattern. • libxml2 PRIMEVUL-2020-0e1a49c89076 and CVE-2020-7595 select the same target and fixing commit and the same end-of-input change in parser.c, under two identifiers. Their four records are one observation, not two confirmations. CodeQL. Nagios 2018 contains a conditional NULL path. The query handler qh_input initializes query to NULL and assigns it only if the request contains a space, so requests such as #core, #help, or #echo pass NULL to handlers whose old versions dereference it. The fix adds NULL guards at the three handler entries. This is a static path in the source; we did not reproduce an exploit. Neither saved suite selects MissingNullTest or InconsistentNullnessTesting, and the selected cpp/redundant-null-check-simple query requires a dereference followed by a NULL test, a different pattern. To test whether adding a nullness query would help, we copied the exact old and new bodies of qh_echo, qh_help, and qh_core into an isolated R2024 fixture with a synthetic dispatcher that passes NULL and with positive controls. All four input files matched the archived source bytes,

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

29

and extraction succeeded for all three translation units. MissingNullTest reported its local-null control and cpp/redundant-null-check-simple its dereference-before-check control, but neither reported any of the six original bodies. The positive controls of InconsistentNullnessTesting did not report, so we draw no conclusion about that query. A diagnostic query returned 31 nullness/use tuples in the six bodies, including 18 dereferences of buf across all of them, and none satisfied the maybeNull predicate. The library assumes pointers valid by default and follows local control flow, so the caller’s NULL argument does not reach the callback parameters. Adding the query alone would not catch this pattern. Extraction may also play a part. For the 2018 handler file base/query-handler.c, the R2024 results record 38 extraction warnings, including a missing include/config.h. One falls inside qh_echo (lines 25–42), five inside qh_help (370–402), none inside qh_core (404–470), and 32 elsewhere. The diagnostics also report an incomplete struct query_handler at the dispatch line (227) and at the callback storage (328), so extraction problems touch the NULL path itself, although we cannot show that IR edges were lost. The recovered R2026 run reports 101 partial and 36 successful extraction commands; its score stays zero, and its 79 finding signatures match the frozen record exactly. For BlueZ CVE-2020-0556, the fix adds a bonding check in hog_accept: the fixed code calls device_is_bonded and returns -ECONNREFUSED for an unbonded device before bt_hog_attach. Both roles have zero changed-file hits in every release, despite unrelated findings elsewhere. We inspected two authentication queries. The first, cpp/user-controlled-bypass, requires tainted data to reach a condition that contains a hard-coded address; the second, cpp/tainted-permissions-check, requires it to reach a modeled privilege operation. A missing bonding policy provides neither. Under R2024 the scored file hog.c also carries eleven extraction problem notifications per role. BlueZ CVE-2020-27153 shows a scoring-file mapping limit. The vulnerable case is scored on mesh/cfgmod-server.c, while the verified repair changes src/shared/att.c. No release reports an alert on att.c for either role, so the mapping did not hide an existing alert. The upstream fix addresses an invalid read in enable_ccc_callback after the client has been torn down during an earlier disconnect callback: bt_gatt_client_free frees a characteristic through notify_chrc_free, and the callback later dereferences it. The saved R2021 query list excludes the use-after-free and double-free queries, and R2024 includes both, yet R2024 reports nothing on att.c. Whether the free reaches the later use through the queue callbacks and client-owned fields is exactly what the query would have to model, and we did not isolate the failing predicate. Across the paired CodeQL repairs, extraction warnings on the repair file occur in 8 of 26 R2021 and 20 of 26 R2024 records, and all 27 selected R2026 result files report partial extraction. Either the presence or the absence of a warning on a file is weak evidence, because a file can be partly extracted without a warning on it. Table 28 summarizes what the evidence supports for each CodeQL no-signal case and what would settle the remaining question. Infer. Infer’s no-finding endpoints include valid empty reports. For firejail and Slurm, the v1.1 and v1.3 endpoints are empty_capture observations with score zero (Table 24), so their “0→0” rows in Table 22 reflect missing capture rather than evaluated code; only v1.2 captured those snapshots. Keeping empty and partial captures in the evaluation avoids a stricter completion rule for newer releases, but it lets capture coverage enter measured effectiveness. 8

RQ5: Operational Sustainability

RQ5 looks at the runs themselves: whether they finish, what a “success” contains, and what they cost. The three tools record different things, so we report each one in its own terms before comparing.

30

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 28. CodeQL cases without a scoring signal: what the evidence supports and what remains open. Case

Supported explanation

Open question; deciding check

Nagios 2018 NULL repair

Positive controls pass; the repaired parameter accesses fail maybeNull; the target file has missing-header diagnostics Client-data lifetime path in the source; R2021 excludes the memory queries The repair adds an authorization policy that the two inspected queries do not encode Recovered results are identity-validated

Whether any nullness analysis models the caller-to-handler NULL path; query inclusion alone is insufficient in the fixture Why R2024’s use-after-free query stays silent; trace free(chrc) to the later use Coverage by the full suite; compare an explicit bonding query on the same database Source repair versus query or IR contribution is not isolated Query selection versus extraction or semantic change; needs compatible databases

BlueZ 2020 callback repair BlueZ 2020 bonding repair HAProxy 2023 under R2026 BlueZ 2016, Unbound Score and query-list differences are observed 2020

Table 29. CodeQL reliability. (a) Final outcomes on deduplicated targets. (b) Status transitions of identical targets (target, commit, role, project, year) between releases. S: success; F: failure. (b) Same-target transitions (a) Final outcomes Targets Success Timeout DB fail R2021 R2024 R2026

8.1

19,780 19,780 19,820

19,494 19,390 19,820

104 290 0

Rate

182 1.446% 100 1.972% 0 0.000%

Earlier → later

S→F F→S F→F

S→S

R2021 → R2024 final R2021 → R2026 checkpoint R2024 → R2026 checkpoint R2021 → R2026 final R2024 → R2026 final

281 278 277 0 0

19,213 19,216 19,113 19,494 19,390

177 269 372 286 390

109 17 18 0 0

CodeQL: Status Moves in Both Directions

Table 29(a) reports final outcomes on deduplicated targets, and Table 29(b) follows the same target identities across releases. The two historical releases end with 676 final failures (394 timeouts and 282 database failures), which cover 567 distinct targets, because 109 targets fail under both. These are target counts. Several cases can share a target, so the 290 R2024 timeout targets correspond to the 332 timeout cases of Table 4. Of 19,780 exact historical targets, 281 succeed under R2021 but fail under R2024, while 177 move the other way. A newer release is therefore not uniformly more executable. The 281 regressions consist of 277 timeouts and four database failures. The four database failures are the vulnerable and fixed targets of KeepKey CVE-2022-30330 and pdns CVE-2018-1046; their R2024 database step ran for only 1.8–2.4 seconds, and the same targets succeed under R2021 and R2026. No raw attempt or error output survives for them, so their cause stays unknown. R2026 reaches final success on all 19,820 targets, but its recovered July checkpoint contains 295 failures, so “zero final failures” describes delivered coverage, not first-pass robustness. A second view compares different snapshots of the same project under one release. We found 161 pairs in which an older snapshot succeeds and a newer one fails, spread over 41 release-by-project combinations. In GPAC under R2021, for example, a fixed 2018 target succeeds and a distinct fixed 2019 target ends in a database failure; in QEMU, a fixed 2016 target succeeds and a fixed 2020 target fails during database creation after 4.6 seconds, before analysis starts. Neither record keeps a command, an error message, or resource data, so these pairs show where the failure happened but cannot say why. They concern different targets and must not be added to the exact-target counts. Build and discovery mechanisms. The historical archive keeps mostly coarse final statuses. It lacks the selected attempt, the full command, error output, exit code, disk, and machine load for each final record. A recovered raw ledger of 6,138 attempts contains 56 rows with direct diagnostics, and they point to two mechanisms.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

31

Table 30. Candidate causes of CodeQL execution failures and the strength of the evidence for each. Candidate cause

Observed fact

Build system and extraction

No-source errors; tracer logs show no compiler call; empty object directories

Missing evidence

Status

Archived workspace, build Raw mechanism output, clean-build control supported; final attribution open Compiler discovery 50 compiler-not-found rows in buildless Selected attempt; historical Raw mechanism mode on targets with C++ files PATH and compiler supported; final cause versions unconfirmed Dependencies and No direct dependency or mirror failure in Dependency, submodule, Hypothesis only mirrors the audited records and checkout fields Extractor or query No final diagnostic names this class Error output; same-target Not supported as a final support control cause Timeout, memory, 394 timeouts clustered at two phase limits Final RSS, disk, load, Timeout observed; disk, load process tree cause open Runner path, cache, Raw commands show autobuild and a PATH Final environment and Raw level only temp error cache state

The first concerns traced builds. Six R2021 attempts end with return code 32 and the message that no source code was seen and extracted. Their build-tracer logs list only shell utilities (sh, mkdir, find, grep) and no compiler call, and their object directories are empty. The saved build wrapper runs its compiler fallback only when a recursive search finds no object file or compile-command file. Both affected projects track prebuilt objects in their repositories: the two Sandboxie commits contain 20 tracked .obj files and the two js-compute-runtime commits contain three tracked .o files, and neither has a build file that the wrapper’s main branches recognize. In a local control with one compilable C file, adding one tracked object file from each project and then removing it changed the number of compiler invocations from 1 to 0 to 1, while the wrapper exited successfully every time. The wrapper thus mistakes the presence of a build artifact for evidence of fresh compilation. The second concerns buildless extraction. Fifty R2024 attempts report that no suitable C++ compiler was found, although the targets contain C++ files (322 in each of four audited targets). CLI 2.20.0 looks only for cl, clang, and clang++, and never for gcc or g++. In a controlled experiment on one synthetic C file, a GCC-only environment failed (exit 2), adding Clang succeeded and extracted the file, and removing Clang failed again. The R2026 extractor, by contrast, finds gcc and g++ after looking for Clang. These local experiments establish the mechanisms; they do not recover the historical environment of every failure. In particular, the 50 attempts were not final failures: the same targets end in 46 final successes and 4 timeouts, so they may be earlier attempts that were later replaced. Linking raw attempts to final records. All 56 raw diagnostics can be tied to a target, but not to the attempt that produced the final record. The six no-source rows agree with final database failures; of the 50 compiler-not-found rows, 4 agree with final timeouts and 46 with final successes. Across three attempt sources, 34 attempts overlap only 30 of the 676 final-failure keys, and 24 of those attempts succeeded although the final index records a failure. When successful and failed attempts coexist on the same inputs, an identity match cannot tell which execution produced the final row. We therefore treat the mechanisms in Table 30 as supported only at the level stated there. Timeouts and query cost. The 394 historical timeouts split into 342 analysis-stage cutoffs near 1,800 seconds and 52 phase cutoffs near 7,200 seconds; 47 successful analyses ran for at least 1,700 seconds, so the cap also censors slow successes. The cutoffs tell us at which stage a run stopped but not why it was slow, and without memory, disk, and load records we cannot upgrade a timeout to an out-of-memory or contention claim. BlueZ CVE-2020-0556 shows what can lie behind a timeout. Analysis took 83.4 and 88.3 seconds (vulnerable and fixed) under R2021 and 176.2 and

32

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 31. The 295 R2026 checkpoint failures: what failed, and which recovery run delivered the final success. (a) Checkpoint failure

(b) Recovery that delivered success

Failure type

Recovery run

Database exit 2 (buildless, changed-file scope) Analysis timeout Database input unavailable Analysis exit 2 Analysis killed (memory not confirmed) Database timeout

Count 219 48 15 11 1 1

Header-driver recovery First standalone-source tranche Continuation of that tranche BlueZ reduced-suite retry Scoped retry (one target)

Count 217 43 23 11 1

Fresh database and analysis / cached reuse 285 / 10

185.2 seconds under R2024. Under R2026 both targets timed out in the checkpoint run, and a later retry on the existing databases timed out again after 43,200 seconds, with the watchdog reporting no stall. The evaluator logs show that 96 of the 98 queries in the suite already had cached results. Only cpp/overrun-write and cpp/invalid-pointer-deref were still evaluating the recursive modulus analysis of the range-analysis library, which tracks facts of the form “expression equals a base plus an offset modulo a modulus”. Near the end, the fixed run was at iteration 18,850 and the vulnerable run at 18,972, still producing new facts. Removing one of the two queries left the other in the same recursion; removing both produced valid results from the same databases. The selected recovery used fresh buildless databases and a reduced query set. It finished in 61.97 and 61.27 seconds, with 152 successful and 312 partial extraction commands out of 464 in each role. Faster completion here came with a smaller query set and partial extraction, so the recovered result is not equivalent coverage. Recovery hides fragility. Table 31 breaks down the 295 R2026 checkpoint failures and the recovery runs that later turned them into successes on the same commit and role. All 219 database exit-2 failures used changed-file-scoped buildless extraction; a local preflight routed 214 of them to a header-driver recovery and five to a standalone-source recovery. The 15 unavailable inputs were commits missing from the checkpoint mirror, all of which were present in the later mirror. Twelve recovery runs failed on the way (11 analysis timeouts and one database failure) before a later run succeeded. Success can also coexist with incomplete extraction. In 45 local validation attempts, partial-extraction diagnostics appeared in 8 of 23 R2021 and 22 of 22 R2024 runs, and one R2024 htmldoc log shows 81 successful and 27 failed compilation commands out of 108. Membership history. The target set changed once during the study: 40 target identities were removed and 40 added. Sixteen results supplied for removed identities were quarantined (11 had succeeded and 5 had failed), and 89 Chromium targets missing from one delivery remained in the dataset; together with the 40 new identities they accounted for 129 results that had to be produced later. No commit or identity mismatch remains among the current records. Because 11 of the retired results had succeeded, the removal cannot be read as CodeQL failing on those targets. 8.2

Infer: Build Availability and Empty Success

Infer v1.1 ends with 232 build failures and 8 analysis timeouts; v1.3 ends with 108 build failures and 11 capture timeouts. Table 32 gives the units. The 359 release-by-case failure records cover 251 unique cases, 108 of which fail under both releases, and they are documented by 1,318 attempt records. The dominant diagnosis is unmaterialized pinned submodules (216 v1.1 and 92 v1.3 rows). It is a property of the project, its dependencies, and the environment rather than of the checkers. The remaining build failures are 15 rows per release whose capture database was still absent after continuation and one row per release with a missing dependency or build strategy. These labels

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

33

Table 32. Infer final failures and their evidence. Each failure has one direct failure packet; attempt-history records include earlier attempts on the same cases. Release v1.1 v1.3 Union

Failure records

Build failed

Timeout (stage)

Attempt records

Unique cases

240 119 359

232 108 340

8 (analyze) 11 (capture) 19

962 356 1,318

240 119 251

Table 33. Infer original runner status vs. final selected status (case-by-release rows). v1.2 is omitted because it uses an older ledger. Original status

Final

build_failed build_failed empty_capture empty_capture success partial capture timeout timeout

build_failed success build_failed success success success success timeout

v1.1

v1.3

217 1,389 15 5,347 125 12,887 12 8

93 1,513 15 4,120 786 13,444 18 11

name the selected diagnosis; they do not prove a unique root cause. Figure 2(a) shows how the failures spread over the years. Table 33 shows that a final “success” summarizes very different histories. Only 125 v1.1 and 786 v1.3 cases succeeded in their original run without a partial-capture flag. Most final successes come from partial capture (12,887 and 13,444) or from recovery after a build failure (1,389 and 1,513) or an empty capture (5,347 and 4,120). In 5,716 v1.1 and 4,460 v1.3 cases, the selected result comes from an analysis that was resumed from preserved capture data. A recovered empty capture yields a legal report, possibly empty, and therefore a score, but it does not show that the source tree was analyzed. Infer thus has three different notions of coverage: a report that can be scored, a capture that recorded some source files, and a capture that recorded the case’s changed files. None of them shows that the vulnerable code was analyzed. Two same-project contrasts illustrate the point. They are drawn from 119 older-success/laterfailure contrasts across 18 projects, which are diagnostic observations rather than controlled pairs. Under v1.1, a QEMU 2021 vulnerable case ends as a success with score zero, but its original attempt recorded zero source files and zero procedures, and its log says there was nothing to compile or analyze. The 2022 case fails at setup after about 1.3 seconds: its record lists 23 pinned submodule links, and the recovery history notes missing local objects and disabled remote URLs. Under v1.3, a Linux 2019 fixed case completes with partial capture (16 source files and 253 procedures), but capture of the changed floppy.c was not established; its setup logs also report missing libelf and OpenSSL headers. The 2020 case hits the 7,200-second capture deadline after 7,200.2 seconds and is killed (exit code −15). That is a deadline; it does not show an infinite loop or a checker defect. Both pairs concern whether Infer can observe the code, not whether its checkers aged. The v1.2 archive reports no final failures, but its 49 legacy records (38 timeouts, 10 workspace failures, one build failure) follow a different ledger schema, and only 12 map to current case identities, so we do not rank v1.2 reliability against the other releases.

34

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 34. Semgrep diagnostics. Left: prevalence in the catalog and the primary population (categories overlap). Right: mutually exclusive strata of the primary population; zero-target cases are assigned first, and “parsing” covers partial parsing and both kinds of syntax error. The strata sum to the Study-A confusion counts. Diagnostic Any diagnostic Partial parsing Syntax error Other syntax error Rule timeout Zero targets None, >0 targets

Catalog Primary 15,765 13,274 2,641 655 3,792 113 4,122

Stratum

15,590 13,150 2,603 647 3,738 109 4,045

n Share

TP

FP

FN

TN Prec.

No diagnostics 4,045 20.5% 640 658 1,371 1,376 Parsing only 11,852 60.0% 2,527 2,644 3,376 3,305 Timeout only 366 1.9% 102 96 86 82 Parsing + timeout 3,372 17.1% 1,320 1,335 371 346 Zero targets 109 0.6% 0 0 70 39

F1 FPR AUC

.493 .387 .489 .456 .515 .529 .497 .608 0 0

.324 .444 .539 .794 0

.498 .493 .506 .504 .500

Primary cases (%)

100 80 60

Any diagnostic Parsing

40

Rule timeout Zero scanned files

20 0 '16

'18

'20 '22 Source-code year

'24

Fig. 12. Share of primary Semgrep cases with each diagnostic type per source-code year (x-axis; types overlap). Diagnostics are present in most cases in every year, under a successful run status.

8.3

Semgrep: Success with Degraded Coverage

All 20,000 Semgrep runs report success, including the 113 catalog cases (109 primary) that scanned zero files. A status-based measure would show perfect executability, and there is no old-success/newfailure transition to analyze. A successful status, however, only means that the command returned a result record. It does not show complete parsing, complete rule evaluation, or complete coverage of the target. Table 34 gives the diagnostics. Of the 19,744 primary cases, 15,590 (78.96%) carry at least one diagnostic: 13,150 record partial parsing, 2,603 syntax errors, 647 other syntax errors, and 3,738 rule timeouts. A further 109 cases scan zero files, and 4,045 have neither a diagnostic nor an empty scope. Figure 12 shows that diagnostics are common in every year: the share of cases with any diagnostic ranges from about 71% (2021) to more than 85% (2024), almost entirely driven by parsing problems, while the share with rule timeouts stays near 20% until 2023 and drops below 10% in 2024 and 2025. The strata resist a simple correction. Cases with both parsing and timeout diagnostics have the highest F1 (0.6075) and the highest FPR (0.7942), while cases without diagnostics have F1 0.3868; AUROC is close to 0.5 in all strata. Precision tells the same story from another angle: it lies between 0.489 and 0.515 in every non-empty stratum. Diagnostics change how much a case is flagged, while the share of flags on vulnerable cases stays the same. Diagnosed cases differ in files, projects, and finding density, so the higher F1 of some strata is an association with a different case mix, not evidence that diagnostics improve detection. Excluding them would keep only one fifth of the corpus, a different population, and would not reveal a clean estimate. The AUROC of 0.5 for the zero-target stratum only reflects identical zero scores on both labels. The recorded diagnostics name specific constructs: token-pasting macros in libreswan, formatted strings in FreeRDP, generated headers and enumerations in Varnish, file-spanning syntax errors in OpenBSD and libxml2, and rule timeouts in MilkyTracker. Within a project, scan quality can change between years. MilkyTracker’s two 2019 and two 2020 cases have no diagnostic, while one

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

35

Table 35. Recorded cost of successful runs (seconds unless noted). “Tail” is the 95th percentile, except for Infer (90th). Cached CodeQL phases and missing fields are excluded, not counted as zero. Infer rows use all selected attempts after the identity gate, including resumed analyses and label-conflict cases, since cost does not depend on the label; Semgrep rows use its primary population. Condition

Phase / measure

CodeQL R2021 CodeQL R2024 CodeQL R2026 CodeQL R2026 CodeQL R2026 Infer v1.1 Infer v1.3 Semgrep Semgrep Semgrep

database / analysis wall 19,213 4.6 / 17.3 172.64 / 540.58 — database / analysis wall 19,213 5.3 / 22.3 178.68 / 277.00 — database / analysis wall 11,789 / 11,826 7.80 / 73.26 114.16 / 3,359.36 — database / analysis CPU 7,858 / 7,866 16.07 / 150.69 580.29 / 1,310.78 — database / analysis RSS (MiB) 7,858 / 7,866 644 / 3,869 3,966 / 11,835 — / 38,580 capture / analyze wall 19,713 8.85 / 0.36 65.20 / 1.97 11,334 / 21,362 capture / analyze wall 19,833 19.67 / 0.52 109.47 / 3.88 7,359 / 33,466 scan wall 19,314 62.46 394.83 3,504.56 scan CPU (user + system) 19,314 5.10 168.53 9,298.75 peak RSS (MiB) 19,314 183.30 230.30 820.90

n

Median

Tail

Max Note matched targets ratio 1.227 / 1.173 invoked phases only 66.7% / 66.5% of invoked per phase selected attempts selected attempts 430 missing (cached) per scan per scan

of its two 2022 cases reports two string-rule timeouts; Varnish’s four 2019 cases have no diagnostic, while both 2020 cases report partial parsing around generated HTTP/2 constructs. Early FreeRDP, libreswan, OpenBSD, and libxml2 cases already carry diagnostics, though, so their later no-finding results are not a uniform move from complete analysis to failure. Naming the exact construct or rule that was lost would need diagnostic detail that the recorded scans do not keep. 8.4

Cost on Successful Runs

Table 35 collects the available cost measurements. The tools record different fields under different policies, so the table is an envelope per condition, not a ranking. CodeQL. On the 19,213 targets that succeeded under both historical releases, R2024 analysis is slower than R2021 on 12,582 targets (65.5%), faster on 6,531, and equal on 100; the median per-target ratio is 1.173. Yet the R2024 95th percentile is lower (277.00 vs. 540.58 seconds). Typical and tail behavior thus move in opposite directions. Each of these targets ran on the same machine in both releases (13,637 on one host and 5,576 on another), and the ratio is computed per target before taking the median. The matched set excludes the 458 targets that changed status, so it is biased toward targets that both releases could finish. R2026 has a median analysis time of 73.26 seconds but a 95th percentile of 3,359.36 seconds and a single-phase RSS maximum of 38,580 MiB, which is the peak of one phase rather than of the whole process tree. Of its phase slots, 8,031 database and 7,994 analysis slots were served from cache (about 40%), and their current cost is unknown. CPU and memory were recorded for only two thirds of the invoked phases. We do not rank R2026 against the historical timings, because its recovery policies, query coverage, and cache use differ. Infer. Infer’s medians are small, but recovery budgets produce long maxima. Table 36 separates all selected attempts from the original attempts that needed no resumed analysis; the two populations are not paired, so their difference is not a speedup estimate. The two longest analyses ran under the 43,200-second recovery budget: an ACPICA 2017 fixed case took 21,362 seconds under v1.1 and a jsish 2020 fixed case 33,466 seconds under v1.3, both with partial capture and a clean exit. The longest captures belong to one PyTorch 2024 fixed case under a 21,600-second recovery budget. Under v1.1, 587 seconds of CMake build and 10,747 seconds of Infer capture produced an empty capture that a resumed analysis later turned into a success with score 5; under v1.3, 586 seconds of CMake and 6,773 seconds of capture covered the changed path directly. Rare long analyses are the largest maxima, so the data do not support a blanket claim that capture dominates the tail. v1.2 lacks comparable timing, and no Infer release records CPU time or peak memory in its final ledger. Semgrep. Semgrep’s resource fields cover 19,314 primary cases (97.82%). The 430 missing cases all fall in 2016–2020, and every one of them carries a recorded cache-return message. Wall time is

36

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Table 36. Infer wall time of selected attempts (seconds). “All selected” uses every selected attempt, with resumed analyses timed from the continuation; “original only” excludes resumed rows. Release, population

Phase

Attempts

Median

90th pct.

Maximum

v1.1, all selected v1.1, original only v1.1, all selected v1.1, original only v1.3, all selected v1.3, original only v1.3, all selected v1.3, original only

capture capture analyze analyze capture capture analyze analyze

19,713 14,009 19,713 14,009 19,833 15,383 19,833 15,383

8.85 22.32 0.36 0.27 19.67 41.44 0.52 0.35

65.20 66.24 1.97 0.88 109.47 121.48 3.88 2.98

11,334.17 2,918.44 21,361.92 21,361.92 7,359.40 7,359.40 33,466.22 33,466.22

Fig. 13. Empirical cumulative distributions of Semgrep wall time, CPU time, and peak RSS (19,314 primary cases; logarithmic x-axes for time). The dotted line marks the 95th percentile. (a) Median wall time (s)

(b) Median CPU time (s)

100

(c) Median peak RSS (MiB)

190

5.0 50

180

2.5

0

170

0.0 '16

'18

'20

'22

'24

'16

'18

'20

'22

'24

'16

'18

'20

'22

'24

Fig. 14. Yearly medians of Semgrep wall time, CPU time, and peak RSS by source-code year (x-axis). Cases per year with resource fields range from 1,794 (2020) to 2,000.

measured around the scan command; CPU time (user plus system) and peak RSS come from GNU time and describe a single scan without summing parallel scans. Figure 13 shows heavy right tails for wall and CPU time and a narrow memory band. The 99th percentiles are 625.99 seconds of wall time, 572.04 CPU seconds, and 310.80 MiB. The maximum CPU time (9,298.75 seconds) exceeds the maximum wall time (3,504.56 seconds). Figure 14 shows the yearly medians. Median wall time peaks at 100.3 seconds for the 2017 code and falls to about 30 seconds for 2024 and 2025, while median CPU time (4.59–5.55 seconds) and memory (181.6–183.9 MiB) hardly move. The drop in wall time is therefore more plausibly tied to runner conditions and caching, which we cannot separate, than to less analysis work, and we do not read it as a speedup on newer code. The resource records also do not exist per Study-B or Study-C draw, so we make no matched efficiency claim. Finding Q2. Final success is necessary but not sufficient evidence of reliability. CodeQL targets change status in both directions, and recovery can change the query set and extraction scope; Infer recovers thousands of empty or failed captures into scored successes; and Semgrep reports success for every run although 78.96% of cases carry parsing or timeout diagnostics. Costs are heavy-tailed, and the recorded fields differ too much across tools to support a cross-tool ranking.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

9 9.1

37

Discussion What SAST Sustainability Means in Practice

Temporal sustainability is a relation among a detector condition, a code population, a measurement scope, and an execution environment; it is not a number attached to a tool name. Three distinctions organize our evidence. First, level and trajectory differ. Later CodeQL releases score higher but lose ground within projects faster than R2021, and Infer v1.2 has the highest level while v1.1 and v1.3 have the more positive matched-gap profiles. Second, the three designs answer different questions. Calendar-year populations, matched pairs, and within-project slopes disagree in several places (for example, Infer’s ordering flips between Studies A and B), and the disagreements we examined trace back to which projects and cases carry each comparison. No single curve substitutes for the three. Third, label agreement, mechanism evidence, and operational completion are separate outcomes. A changed-file warning may be unrelated to the repair, and a final success may follow retries, query exclusion, or an empty capture. Treating every degraded run as a negative prediction would mix reliability into effectiveness; ignoring degraded runs would overstate coverage. One pattern runs through almost every condition. Across many of our release and year comparisons, higher case-label F1 accompanies increases in both recall and FPR, while count-based AUROC remains near 0.5. Infer v1.2 is the exception, because it flags few fixed cases. A reader who looks at F1 alone, or at recall alone, would conclude that newer tools or newer code are handled better. Read together with FPR and AUROC, the same numbers say that the tools flag more of everything. 9.2

Implications

For tool builders. Release regression suites should replay project histories, not only fixed benchmarks, and report repair-aligned detections, front-end coverage (parse, extraction, and capture rates), query-level runtime tails, and same-target status transitions. Recall should always be paired with a fixed-snapshot control, so that broader alerting is not mistaken for better discrimination; our CodeQL and Infer results show how easily the two diverge. Dashboards should distinguish projects with stable nonzero performance from projects that never produce a signal. Recovery runs should report which queries or files they dropped; for BlueZ, per-query time budgets with explicit reporting of skipped queries would have made a silent timeout visible. Build wrappers should not take the presence of an object file as proof of compilation, and buildless extractors should report which compilers they searched for. For practitioners. Before upgrading the analyzer in a CI gate, run the candidate release on fixed commits from the organization’s own history and compare repair-relevant warnings, analyzed scope, diagnostics, status, and runtime with the current release. Default configurations are common in practice [8], but our results show that the configuration, extraction mode, and recovery policy can change outcomes as much as the engine version. A run that ends in “success” should be checked for parse and capture diagnostics before its silence is trusted. SAST should remain one control among tests, fuzzing, review, and runtime defenses; historical scores do not show zero-day coverage. For benchmark designers and researchers. A temporal benchmark should store the vulnerable parent, the fix, verified repair hunks, the source year, a canonical project identity, a build recipe, the detector condition, and attempt-level provenance. Scoring should use comparable semantic scopes on both labels; our Semgrep queue shows that adjacent commits can still be scored on disjoint files, and that two identifiers can point at one snapshot. Broad proxies should be calibrated with blinded mechanism review, and the inference unit should be explicit: 100 resamples are not 100 detections, nested cohorts are not additive, and a single year pair is one comparison.

38

9.3

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Why This Matters for Zero-Day Discovery

SAST is most valuable before a defect has a name: a general taint query, abstract domain, or bounds rule can expose an instance that later becomes a CVE. Our results identify three ways in which this transfer to new code can fail. The defect idiom can fall outside the rule or query predicate (Nagios, Varnish); the relevant code can never reach the analysis because of build, capture, parsing, or scoping problems (MilkyTracker, QEMU, OpenBSD); or unrelated warnings can mask the absence of mechanism coverage (libxml2, libiec61850). We do not equate later-year CVEs with zero-days. What we measure is a necessary condition for prospective discovery: rule semantics that remain applicable, together with observable analysis coverage, as code evolves. 9.4

What the Design Leaves Open

Several analyses would sharpen these results, and we did not run them. Study B pools cases within each pair, so large projects weigh more; a project-level variant that computes ΔF1 per project and then averages projects would show whether a few projects drive a pair. Study C reports mean slopes and their signs; median slopes with uncertainty per project, and a pooled model with a random effect per project (for example, a logistic model of per-case correctness on source year), would turn the descriptive slopes into an estimate. Our year-pair ranges describe resampling variation; they are not confidence intervals. A hierarchical model with project, CWE, scope, and condition effects is the natural way to obtain intervals for the population of projects. Finally, reliability and cost were measured on the stored Study-A runs only. Following failure rates and run time along the Study-C trajectories would need resource records per resampled case, which the archives do not keep. Each extension can reuse the frozen memberships and results in our package. 9.5

Threats to Validity

Construct validity. Our score is a proxy: an in-scope alert counts as a detection without CWE agreement or repair overlap, and F1 on a balanced corpus is not the precision a developer would see. We therefore report AUROC and FPR beside F1 and calibrate the selected extremes with repair-file and repair-line ablations; the ablations cover the selected cases, not all 20,000. Count-based AUROC is a weak notion of ranking, but it treats all tools alike. Labels, source years, and CVE-to-repair links come from the corpus. We exclude the 256 case identities that carry both labels from labelbased evaluation for all tools and checked parent-to-fix relations for the inspected cases, but label noise known from such datasets [22] may remain, and commit adjacency does not validate every CVE-to-repair binding. The CodeQL score depends on a security-relevance filter whose effect we checked only for the selected cases, where it removed no alert in a scored file. Internal validity. A detector condition bundles engine, rules, extraction mode, recovery, and cache state, so release differences are not pure version effects; CodeQL R2024, for example, is also the only buildless historical condition. Apart from the shared label-conflict gate, each tool has its own exclusion gate. The common-intersection results preserve every ordering we report, but they keep only cases that every release completed and may favor easy targets. Infer keeps the highest-scoring completed candidate within each reuse group. This rule can raise both true and false positives, its effect can differ between releases with different numbers of candidates, and the common intersection does not control it; we have not quantified it. Matching fixes projects and label counts but not CWE, module, code size, or analysis quality, and in some pairs one project supplies most of the matched cases. Cost figures describe operating ranges, not tool speed, because machine load and caching were not controlled. Conclusion validity. Year pairs share projects, repetitions resample a fixed pool, and the Study-C cohorts are nested, so significance tests that assume independence would overstate certainty. We

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

39

report spread and support instead, rely on sign patterns that recur across many pairs or survive the sensitivity cohort, and draw no conclusion from gap 9, which rests on one pair. Repetition-level spread is available only for selected pairs (two Infer pairs and six Semgrep pairs). External validity. We study open-source C/C++ code, three analyzers, and default or locally curated rules on changed files. Other languages, commercial rules, proprietary code, and wholerepository scans may behave differently. The source years cover 2016–2025. The complete yearpair analysis intentionally includes code that predates some detector conditions and therefore characterizes temporal sensitivity across code vintages, rather than necessarily prospective aging. Only comparisons between code populations following the frozen detector’s release can provide direct evidence of prospective aging. Qualitative and operational evidence. The RQ4 queues are selected extremes; they show that a mechanism exists, not how often. Missing historical intermediate representations, environments, and links to the selected attempt prevent a unique causal attribution for many historical failures, so we grade each claim by its evidence. CPU, memory, retries, cache state, queueing, and machine load are recorded incompletely. Language-model agents assisted source and log inspection, and the cross-check was not blind. We therefore tied every claim to source, diff, rule, query, or log evidence that others can inspect. 10

Related Work

SAST effectiveness on real vulnerabilities. Cross-sectional studies agree that SAST misses many real vulnerabilities. Li et al. report that seven Java SAST tools together leave 70.9% of realworld vulnerabilities undetected [45]; Lipp et al. find that C analyzers miss 47–80% of 192 real vulnerabilities [47]; Habib and Pradel find that three Java bug detectors catch only a small fraction of 594 real bugs [34]; and Bennett et al. measure individual detection rates of 11.2–26.5% before extending Semgrep’s rules [9]. A synthetic Java benchmark shows high precision but low recall [29], rule-based and learning-based approaches have been compared directly [23], and NIST’s SATE exercises evaluate tools on injected and collected bugs [25]. Charoenwet et al. study SAST warnings in secure code review [19]. All of these studies evaluate one point in time. We show that a reasonable score can coexist with a finding count that barely separates vulnerable from fixed code, and that a fixed detector’s score moves as the code ages. Static analysis in practice. Industrial reports describe what keeps analyzers useful at scale [10, 28, 53], and developer studies document distrust of false positives, workflow friction, and contextdependent use [7, 21, 37, 39, 58]. Practitioners fear false negatives most and rarely benchmark tools [3], many keep default configurations [8], and work on actionable-alert identification and false-positive mitigation [33, 36, 41] focuses on warning volume and triage. We add a temporal concern: settings that teams rarely revisit, such as extraction mode and recovery policy, can decide whether new code is analyzed at all, and large warning volumes can produce high label-level scores with chance-level discrimination. Longitudinal studies of static analysis. Earlier longitudinal work follows the warning population: how statically detected vulnerabilities decay [26], how security warnings persist in C++ projects [2], how PMD warnings evolve in Apache projects [56], and how warning history can guide prioritization [42]. The closest work, by Noirot Ferrand et al., runs 114 CodeQL versions on 3,993 CVEs and finds that detections are not monotonic across versions [49]. They vary the detector; we fix it and vary the age of the code, for three analyzers, with exact project–label matching, within-project trajectories, operational ledgers, and repair-level checks. Because detectors and code evolve together, the two designs complement each other.

40

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

Temporal evaluation in security and defect prediction. Building on concept-drift research [32], malware studies showed that classifiers age and that mixing time periods inflates results [4, 40, 51, 60]. Learning-based vulnerability and bug detectors degrade on realistic or shifted data [18, 27, 35, 52], vulnerability prediction depends on realistic labelling over time [38], and defect-prediction conclusions change with the evaluation period [6]. These works motivate chronological, compositionaware evaluation. We bring it to SAST, where parsers, builds, rule selection, and recovery can drift without any learned model. The concept of sustainability in security defense was earlier proposed [15] and later studied [13] for malware detection, including characterization concerning its underlying driving mechanisms due to software evolution [14], but all for mobile apps, as opposed to the sustainability of static security analyzers as focused in this paper. Vulnerability data and buildability. Vulnerability-fix datasets such as Big-Vul, Devign, CVEfixes, DiverseVul, and PrimeVul [11, 20, 27, 30, 61] supply the vulnerable/fixed pairs that designs like ours depend on, and studies of security patches and vulnerability lifetimes [1, 44] characterize the repairs. Data-quality studies report label noise and duplication in such datasets [22]; our label-conflict gate, scope-alignment analysis, and the duplicate libxml2 snapshot show the same concern for SAST scoring. Work on rebuilding past snapshots [48, 57] explains why build-dependent analyzers such as Infer and traced CodeQL lose historical targets. Graph-based query frameworks such as code property graphs [59] and analyzers that add LLM-inferred specifications to CodeQL [46] inherit the front-end and query dependencies that we show to be time-sensitive. 11

Conclusion

This paper studies whether SAST detectors remain effective, executable, and affordable as the code they analyze evolves. Using yearly populations, project-matched year pairs, and within-project trajectories over a decade of C/C++ vulnerabilities, we found that temporal score changes can arise from broader alerting, shifting evaluation scope, and alerts unrelated to the labeled repair, while selected genuine misses arise from rule/query mismatch and incomplete analysis coverage. These results show that label-level score changes alone cannot establish corresponding changes in vulnerability-mechanism coverage. SAST sustainability should therefore be assessed by replaying fixed detector configurations on project histories, with repair-level evidence and full reporting of analysis coverage. Future work will extend the study to other languages, commercial and learningbased detectors, and labels finer than the changed-file proxy. Data Availability Our replication package (https://anonymous.4open.science/r/sast-sustainability) contains the frozen case ledger, the canonical project map, Study-B/C membership files, normalized findings, the reliability registries, analysis and figure scripts, a data dictionary, and a claim-to-evidence manifest that links every number in this paper to its source artifact. Third-party source snapshots and tool distributions are referenced by commit and version and redistributed only where licenses permit. References [1] Nikolaos Alexopoulos, Manuel Brack, Jan Philipp Wagner, Tim Grube, and Max Mühlhäuser. 2022. How Long Do Vulnerabilities Live in the Code? A Large-Scale Empirical Measurement Study on FOSS Vulnerability Lifetimes. In Proceedings of the 31st USENIX Security Symposium. USENIX Association, 359–376. [2] Bushra Aloraini, Meiyappan Nagappan, Daniel M. German, Shinpei Hayashi, and Yoshiki Higo. 2019. An Empirical Study of Security Warnings from Static Application Security Testing Tools. Journal of Systems and Software 158 (2019), 110427. doi:10.1016/j.jss.2019.110427 [3] Amit Seal Ami, Kevin Moran, Denys Poshyvanyk, and Adwait Nadkarni. 2024. “False Negative – That One Is Going to Kill You”: Understanding Industry Perspectives of Static Analysis Based Security Testing. In Proceedings of the 2024 IEEE Symposium on Security and Privacy (S&P). IEEE.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

41

[4] Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and Don’ts of Machine Learning in Computer Security. In Proceedings of the 31st USENIX Security Symposium. USENIX Association, 3971–3988. [5] Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. 2016. QL: Object-Oriented Queries on Relational Data. In Proceedings of the 30th European Conference on Object-Oriented Programming (ECOOP) (LIPIcs, Vol. 56). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2:1–2:25. doi:10.4230/LIPIcs.ECOOP.2016.2 [6] Abdul Ali Bangash, Hareem Sahar, Abram Hindle, and Karim Ali. 2020. On the Time-Based Conclusion Stability of Cross-Project Defect Prediction Models. Empirical Software Engineering 25, 6 (2020), 5047–5083. doi:10.1007/s10664020-09878-9 [7] Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software. In Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 470–481. doi:10.1109/SANER.2016.105 [8] G. Bennett, T. Hall, S. Counsell, and T. Shippey. 2024. Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A Large-Scale Empirical Study. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, 454–460. doi:10.1145/3674805.3690750 [9] G. Bennett, T. Hall, E. Winter, and S. Counsell. 2024. Semgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE). ACM, 614–623. doi:10.1145/3661167.3661262 [10] Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World. Commun. ACM 53, 2 (2010), 66–75. doi:10.1145/1646353.1646374 [11] Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE). ACM. doi:10.1145/3475960.3475985 [12] Leyla Bilge and Tudor Dumitraş. 2012. Before We Knew It: An Empirical Study of Zero-Day Attacks in the Real World. In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS). ACM, 833–844. doi:10.1145/2382196.2382284 [13] Haipeng Cai. 2020. Assessing and improving malware detection sustainability through app evolution studies. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 2 (2020), 1–28. [14] Haipeng Cai. 2020. Embracing mobile app evolution via continuous ecosystem mining and characterization. In Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems. 31–35. [15] Haipeng Cai and John Jenkins. 2018. Towards sustainable android malware detection. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. 350–351. [16] Cristiano Calcagno, Dino Distefano, Jérémy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter W. O’Hearn, Irene Papakonstantinou, Jim Purbrick, and Dulma Rodriguez. 2015. Moving Fast with Software Verification. In NASA Formal Methods (NFM) (LNCS, Vol. 9058). Springer, 3–11. doi:10.1007/978-3-319-17524-9_1 [17] Cristiano Calcagno, Dino Distefano, Peter W. O’Hearn, and Hongseok Yang. 2011. Compositional Shape Analysis by Means of Bi-Abduction. J. ACM 58, 6 (2011), 1–66. doi:10.1145/2049697.2049700 [18] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Transactions on Software Engineering 48, 9 (2022), 3280–3296. doi:10.1109/TSE.2021 .3087402 [19] Wachiraphan Charoenwet, Patanamon Thongtanunam, Van-Thuan Pham, and Christoph Treude. 2024. An Empirical Study of Static Analysis Tools for Secure Code Review. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 691–703. doi:10.1145/3650212.3680313 [20] Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses (RAID). ACM. doi:10.1145/3607199.3607242 [21] Maria Christakis and Christian Bird. 2016. What Developers Want and Need from Program Analysis: An Empirical Study. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, 332–343. doi:10.1145/2970276.2970347 [22] Roland Croft, M. Ali Babar, and M. Mehdi Kholoosi. 2023. Data Quality for Software Vulnerability Datasets. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 121–133. doi:10.1109/ ICSE48619.2023.00022 [23] Roland Croft, Dominic Newlands, Ziyu Chen, and M. Ali Babar. 2021. An Empirical Study of Rule-Based and LearningBased Approaches for Static Application Security Testing. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, 1–12. doi:10.1145/3475716.3475781

42

Junhua Tang, Zhengsong Zhang, and Haipeng Cai

[24] Oege de Moor, Mathieu Verbaere, Elnar Hajiyev, Pavel Avgustinov, Torbjörn Ekman, Neil Ongkingco, Damien Sereni, and Julian Tibble. 2007. Keynote Address: .QL for Source Code Analysis. In Proceedings of the 7th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE. [25] Aurelien Delaitre, Paul E. Black, Damien Cupif, Guillaume Haben, Loembe Alex-Kevin, Vadim Okun, and Yann Prono. 2023. SATE VI Report: Bug Injection and Collection. NIST Special Publication 500-341. National Institute of Standards and Technology. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.500-341.pdf [26] Massimiliano Di Penta, Luigi Cerulo, and Lerina Aversano. 2008. The Evolution and Decay of Statically Detected Source Code Vulnerabilities. In Proceedings of the 8th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 101–110. doi:10.1109/SCAM.2008.20 [27] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability Detection with Code Language Models: How Far Are We?. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE. doi:10.1109/ICSE5534 7.2025.00038 [28] Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W. O’Hearn. 2019. Scaling Static Analyses at Facebook. Commun. ACM 62, 8 (2019), 62–70. doi:10.1145/3338112 [29] Matteo Esposito, Valentina Falaschi, and Davide Falessi. 2024. An Extensive Comparison of Static Application Security Testing Tools. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE). ACM, 69–78. doi:10.1145/3661167.3661199 [30] Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR). ACM, 508–512. doi:10.1145/3379597.3387501 [31] Tom Fawcett. 2006. An Introduction to ROC Analysis. Pattern Recognition Letters 27, 8 (2006), 861–874. doi:10.1016/j. patrec.2005.10.010 [32] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A Survey on Concept Drift Adaptation. Comput. Surveys 46, 4 (2014), 44:1–44:37. doi:10.1145/2523813 [33] Zhaoqiang Guo, Tingting Tan, Shiran Liu, Xutong Liu, Wei Lai, Yibiao Yang, Yanhui Li, Lin Chen, Wei Dong, and Yuming Zhou. 2023. Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities. IEEE Transactions on Software Engineering 49, 12 (2023), 5154–5188. doi:10.1109/TSE.2023.3329667 [34] Andrew Habib and Michael Pradel. 2018. How Many of All Bugs Do We Find? A Study of Static Bug Detectors. In Proceedings of the 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, 317–328. doi:10.1145/3238147.3238213 [35] Jingxuan He, Luca Beurer-Kellner, and Martin Vechev. 2022. On Distribution Shift in Learning-Based Bug Detectors. In Proceedings of the 39th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 162). PMLR, 8559–8580. [36] Sarah Heckman and Laurie Williams. 2011. A Systematic Literature Review of Actionable Alert Identification Techniques for Automated Static Code Analysis. Information and Software Technology 53, 4 (2011), 363–387. doi:10.101 6/j.infsof.2010.12.007 [37] Nasif Imtiaz, Akond Rahman, Effat Farhana, and Laurie Williams. 2019. Challenges with Responding to Static Analysis Tool Alerts. In Proceedings of the 16th International Conference on Mining Software Repositories (MSR). IEEE, 245–249. doi:10.1109/MSR.2019.00049 [38] Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman. 2019. The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM. doi:10.1145/3338906.3338941 [39] Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs?. In Proceedings of the 35th International Conference on Software Engineering (ICSE). IEEE, 672–681. doi:10.1109/ICSE.2013.6606613 [40] Roberto Jordaney, Kumar Sharad, Santanu K. Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting Concept Drift in Malware Classification Models. In Proceedings of the 26th USENIX Security Symposium. USENIX Association, 625–642. [41] Hong Jin Kang, Khai Loong Aw, and David Lo. 2022. Detecting False Alarms from Automatic Static Analysis Tools: How Far Are We?. In Proceedings of the 44th International Conference on Software Engineering (ICSE). ACM. doi:10.1145/3510003.3510214 [42] Sunghun Kim and Michael D. Ernst. 2007. Which Warnings Should I Fix First?. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 45–54. doi:10.1145/1287624.1287633

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

43

[43] Meir M. Lehman. 1980. Programs, Life Cycles, and Laws of Software Evolution. Proc. IEEE 68, 9 (1980), 1060–1076. doi:10.1109/PROC.1980.11805 [44] Frank Li and Vern Paxson. 2017. A Large-Scale Empirical Study of Security Patches. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2201–2215. doi:10.1145/3133956.3134072 [45] Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu, and Yixiang Chen. 2023. Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 921–933. doi:10.1145/3611643.3616262 [46] Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. In Proceedings of the 13th International Conference on Learning Representations (ICLR). [47] Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An Empirical Study on the Effectiveness of Static C Code Analyzers for Vulnerability Detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 544–555. doi:10.1145/3533767.3534380 [48] Michel Maes-Bermejo, Micael Gallego, Francisco Gortázar, Gregorio Robles, and Jesús M. González-Barahona. 2022. Revisiting the Building of Past Snapshots – A Replication and Reproduction Study. Empirical Software Engineering 27, 3 (2022), 65. doi:10.1007/s10664-022-10117-6 [49] Jean-Charles Noirot Ferrand, Kyle Domico, Yohan Beugin, and Patrick McDaniel. 2026. Longitudinal Analyses of SAST Tools: A CodeQL Case Study. arXiv preprint arXiv:2605.07900 (2026). doi:10.48550/arXiv.2605.07900 [50] Peter W. O’Hearn. 2018. Continuous Reasoning: Scaling the Impact of Formal Methods. In Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS). ACM. doi:10.1145/3209108.3209109 [51] Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2019. TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time. In Proceedings of the 28th USENIX Security Symposium. USENIX Association, 729–746. [52] Niklas Risse and Marcel Böhme. 2024. Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection. In Proceedings of the 33rd USENIX Security Symposium. USENIX Association, 4247–4264. [53] Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018. Lessons from Building Static Analysis Tools at Google. Commun. ACM 61, 4 (2018), 58–66. doi:10.1145/3188720 [54] Semgrep. 2026. Static Analysis and Rule-Writing Glossary. Semgrep documentation, https://semgrep.dev/docs/writingrules/glossary. Accessed 2026-10-01. [55] Elizabeth A. Stuart. 2010. Matching Methods for Causal Inference: A Review and a Look Forward. Statist. Sci. 25, 1 (2010), 1–21. doi:10.1214/09-STS313 [56] Alexander Trautsch, Steffen Herbold, and Jens Grabowski. 2020. A Longitudinal Study of Static Analysis Warning Evolution and the Effects of PMD on Software Quality in Apache Open Source Projects. Empirical Software Engineering 25, 6 (2020), 5137–5192. doi:10.1007/s10664-020-09880-1 [57] Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2017. There and Back Again: Can You Compile That Snapshot? Journal of Software: Evolution and Process 29, 4 (2017), e1838. doi:10.1002/smr.1838 [58] Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Harald C. Gall, and Andy Zaidman. 2020. How Developers Engage with Static Analysis Tools in Different Contexts. Empirical Software Engineering 25, 2 (2020), 1419–1457. doi:10.1007/s10664-019-09750-5 [59] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In Proceedings of the 2014 IEEE Symposium on Security and Privacy (S&P). IEEE, 590–604. doi:10.1109/SP.2014.44 [60] Limin Yang, Wenbo Guo, Qingying Hao, Arridhana Ciptadi, Ali Ahmadzadeh, Xinyu Xing, and Gang Wang. 2021. CADE: Detecting and Explaining Concept Drift Samples for Security Applications. In Proceedings of the 30th USENIX Security Symposium. USENIX Association, 2327–2344. [61] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In Advances in Neural Information Processing Systems 32 (NeurIPS).

Related documents

• CauSec: Unboxing the Causal Drivers of Static Vulnerability Analysis Performance#1326387
• CauSec: Unboxing the Causal Drivers of Static Vulnerability Analysis Performance#467075
• SNIPTEST: Fuzzing Multi-Level Code Slices for Validating Vulnerabilities#463293
• An Improved Static Analysis Approach For Detecting Input Validation Vulnerabilities In Web Application#241318
• VulWeaver: Weaving Broken Semantics for Grounded Vulnerability Detection#10432