arXiv:2604.19031v1 [cs.CR] 21 Apr 2026
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection ZHENGYANG SHAN, Shandong University, China XU QIAN, Shandong University, China JIAYUN XIN, Shandong University, China MINGHUI XU∗ , Quancheng Laboratory, China and Shandong University, China YUE ZHANG∗ , Quancheng Laboratory, China and Shandong University, China ZHEN YANG, Shandong University, China HAO WU, Nanjing University, China XIUZHEN CHENG, Shandong University, China Software vulnerabilities are a primary threat to modern infrastructure. While static analysis and Graph Neural Networks have long served as the foundation for vulnerability detection, the emergence of Large Language Models (LLMs) has introduced a transformative paradigm driven by superior semantic reasoning and cross-environment generalization. However, in the context of LLM-based vulnerability detection, we identify a fundamental bottleneck in these models termed Signal Submersion: a state where features related to vulnerability are activated internally but numerically overwhelmed by dominant functional semantics. To address this, we propose SAGE (Signal-Amplified Guided Embeddings), a framework that shifts from passive signal submersion to active signal recovery. SAGE integrates task-conditional Sparse Autoencoders (SAEs) to isolate and amplify these faint vulnerability signals. Extensive evaluations on BigVul, PrimeVul, and PreciseBugs demonstrate that SAGE achieves state-of-the-art performance. Notably, SAGE mitigates Signal Submersion by increasing the internal Signal-to-Noise Ratio (SNR) by 12.7× via sparse manifold projection. This mechanistic intervention enables a 7B model to achieve up to 318% Matthews Correlation Coefficient (MCC) gains on unseen distributions and a 319% gain on classic datasets. By maintaining robust performance across 13 programming languages and outperforming 34B baselines, SAGE establishes a more efficient and scalable path to software security than simple parameter scaling. CCS Concepts: • Security and privacy → Software and application security. Additional Key Words and Phrases: Vulnerability Detection, Large Language Models, Sparse Autoencoders, Mechanistic Interpretability ∗ Corresponding author.
Authors’ Contact Information: Zhengyang Shan, [email protected], Shandong University, Qingdao, Shandong, China; Xu Qian, Shandong University, Qingdao, Shandong, China, [email protected]; Jiayun Xin, Shandong University, Qingdao, Shandong, China, [email protected]; Minghui Xu, Quancheng Laboratory, Jinan, Shandong, China and Shandong University, Qingdao, Shandong, China, [email protected]; Yue Zhang, Quancheng Laboratory, Jinan, Shandong, China and Shandong University, Qingdao, Shandong, China, [email protected]; Zhen Yang, Shandong University, Qingdao, Shandong, China, [email protected]; Hao Wu, [email protected], Nanjing University, Nanjing, Jiangsu, China; Xiuzhen Cheng, Shandong University, Qingdao, Shandong, China, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. In Proceedings of the 35rd ACM SIGSOFT International Symposium on SoftwareTesting and Analysis (ISSTA ’26)„ Oakland, CA © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX , Vol. 1, No. 1, Article . Publication date: April 2018.
2
Trovato et al.
ACM Reference Format: Zhengyang Shan, Xu Qian, Jiayun Xin, Minghui Xu, Yue Zhang, Zhen Yang, Hao Wu, and Xiuzhen Cheng. 2018. SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (In Proceedings of the 35rd ACM SIGSOFT International Symposium on SoftwareTesting and Analysis (ISSTA ’26),). ACM, New York, NY, USA, 24 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
The rapid expansion of the digital attack surface has led to a surge in software vulnerabilities, with global databases like the NIST National Vulnerability Database (NVD) and the MITRE CVE list consistently logging thousands of new entries monthly [5]. These flaws, often stemming from subtle coding oversights, can trigger a cascade of consequences–from localized data exfiltration and unauthorized privilege escalation to systemic compromises and catastrophic service disruptions [57]. To combat this, detection methods have evolved through several distinct paradigms: starting with traditional static analysis [46], moving toward Graph Neural Networks (GNNs) [9, 27, 28, 64], and more recently, adopting pre-trained models such as CodeBERT and UniXcoder [18, 20]. The current state-of-the-art has shifted toward Large Language Models (LLMs), which represent a leap in vulnerability detection. Unlike previous iterations, LLMs possess sophisticated semantic comprehension and reasoning capabilities. This allows them to interpret the latent intent of code and trace complex execution logic across extensive sequences. By internalizing cross-lingual patterns and nuanced software environments without the need for manual rule crafting, LLMs offer a more generalized and robust solution to identifying vulnerabilities. However, despite these impressive capabilities, even the most advanced models available today still face critical limits in their performance. Prior evaluations show that, on realistic vulnerabilitydetection benchmarks, frontier LLMs often perform substantially worse than their general coding ability would suggest, and may approach random guessing when the distinction between safe and vulnerable code is subtle and the label distribution is highly imbalanced [15, 56]. In many cases, these frontier models struggle to identify the tiny, crucial differences between safe and unsafe code, leading to results that are sometimes no better than random guessing [43, 50]. The primary reason for this failure is that the signals of a vulnerability are often very weak and its code always hidden inside hundreds of lines of normal, working code [11, 34]. Because the surrounding code looks perfectly correct and follows all the rules of the programming language, it creates a lot of background noise. Even with their advanced reasoning, LLMs find it hard to focus on the tiny parts of the code that cause a security risk [43, 50]. We identify this phenomenon as Signal Submersion. Our analysis reveals that while features related to vulnerabilities are activated in LLMs, their activation intensity remains weak compared to the dominant global functional semantics. Consequently, these subtle signals become numerically dominated by the model’s standard processing, creating a performance bottleneck where vulnerability signals in final layers are too weak to enable discriminative predictions. Motivated by this, we propose SAGE (Signal-Amplified Guided Embeddings), a framework that transitions from passive signal submersion to active signal recovery. SAGE integrates taskconditional Sparse Autoencoders (SAEs) trained independently at each layer, optimizing the SAE and classifier via a joint loss function. This design explicitly constructs a sparse manifold to amplify vulnerability signals while suppressing the semantic background, capturing features at their peak magnitude before Signal Submersion occurs. Crucially, across the tested decoder-only backbones, SAGE can be integrated without modifying the backbone parameters. By skipping the need to run the full LLM during training, it achieves state-of-the-art results with faster speed. , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
3
We evaluated SAGE across four model architectures on refined versions of BigVul [17], PrimeVul [15], and PreciseBugs [22], benchmarking against 10 state-of-the-art methods and commercial frontier models including GPT-5.2-Codex [2], Gemini 3 pro [4] and Claude Opus 4.5 [3]. Results demonstrate that SAGE consistently outperforms baselines and exhibits superior robustness. Mechanistic analysis confirms our approach effectively amplifies the internal signal-to-noise ratio. Our contributions are summarized as follows: • Identification of Signal Submersion: We identify Signal Submersion as an important bottleneck for LLM-based vulnerability detection in the tested model families. Our empirical analysis reveals that vulnerability features are not absent but are numerically overwhelmed, constituting less than 5% of the total activation magnitude in deep layers. • The SAGE Framework: We propose SAGE, a framework that transitions from passive submersion to active amplification. By projecting entangled states onto a sparse manifold, SAGE reverses Signal Submersion, boosting the signal-to-noise ratio (SNR) by up to 12.7 times compared to standard inference. • State-of-the-Art Performance: We conduct an extensive evaluation across refined security benchmarks, including PrimeVul, BigVul, and PreciseBugs. SAGE establishes state-of-the-art (SOTA) performance, delivering a 318% improvement in Matthews Correlation Coefficient (MCC) on unseen distributions and a 319% gain on classic datasets. Crucially, we demonstrate that a 7B model integrated with SAGE significantly outperforms 34B parameter baselines and even larger commercial frontier model, showing positive transfer across 13 programming languages, while results on low-resource language subsets should be interpreted cautiously. 2 2.1
Related Work Traditional Vulnerability Detection Methods
The field of vulnerability detection has undergone significant evolution, beginning with foundational static analysis tools [6, 33, 59] and rule-based systems [31, 54]. These early approaches established the essential baseline for automated security auditing by encoding known vulnerability patterns. To move beyond syntax-level checks, dynamic techniques such as fuzzing and symbolic execution [36] were introduced, significantly enhancing program state exploration and interaction depth. While effective in validating execution-time behaviors, these methods often encounter scalability challenges when navigating expansive path spaces or identifying flaws hidden under rare edge conditions [12, 32, 45]. To bridge the gap between efficiency and deep semantic understanding, subsequent research has shifted towards learning-based paradigms. Recent advancements leverage graph neural networks to operate on structured representations, such as Code Property Graphs [9] or Program Dependence Graphs [26]. By capturing long-range dependencies and intricate controlflow relationships, these structure-based methods have demonstrated a superior ability to model the complex semantics inherent in modern software vulnerabilities. 2.2
LLM-based Vulnerability Detection
Prompt-based Methods. Zero-shot and few-shot prompting strategies leverage the generalization capabilities of LLMs without the need for extensive weight updates [58]. Research has explored various prompt engineering techniques [61] and established unified evaluation frameworks [51] to assess these capabilities. Studies indicate that while LLMs show promise, they primarily rely on token-level patterns, and distinguishing subtle vulnerabilities remains a complex challenge [62]. Comprehensive evaluations suggest that standard prompting strategies face difficulties in achieving high precision on complex real-world datasets, indicating the complexity of the task [50]. , Vol. 1, No. 1, Article . Publication date: April 2018.
4
Trovato et al.
Fine-tuning-based Methods. Fine-tuning pre-trained models, such as LineVul [19] and CodeBERT [18], marked a step forward in detection performance. Following these advancements, the community has increasingly focused on the robustness of evaluation protocols. Recent studies note that random splitting on historical datasets [17] can lead to data leakage, where models might retain specific sample features rather than learning generalizable patterns [8, 11, 43]. Similarly, investigations into models like PDBERT [30] emphasize the importance of distinct training and testing distributions to ensure genuine comprehension. Addressing these insights, recent works like VulLLM have adopted multi-task instruction tuning [16] and strict temporal splitting protocols [22, 60] (e.g., MSIVD) to better align model training with realistic, evolving vulnerability landscapes. Reinforcement Learning-based Methods. To align models with non-differentiable security objectives, researchers have integrated Reinforcement Learning (RL). Methodologies range from optimizing prompts via policy gradients [42] to continuous prefix tuning for generating secure code [21, 47]. More recently, efforts have shifted towards internalizing reasoning capabilities. Inspired by advancements in reasoning models [13], approaches like those by Simoni et al. [49] utilize Group Relative Policy Optimization (GRPO) to enhance detection logic. Frameworks such as ReVD [56] further employ curriculum learning and synthetic reasoning traces to improve the model’s sensitivity to subtle code distinctions. Agent-based Methods. Multi-Agent Systems (MAS) have been proposed to simulate collaborative auditing processes. Frameworks like MuCoLD [37] utilize multi-role debates to refine consensus, while PentestGPT [14] facilitates automated penetration testing. To address the computational overhead of multi-agent interactions, adaptive frameworks like ConColl [53] dynamically balance single-agent efficiency with multi-agent collaboration. Furthermore, systems such as AutoCodeRover [63] extend these capabilities beyond detection, exploring autonomous program repair. 2.3
Internal Representations for Vulnerability Analysis
Complementing external prompting and fine-tuning, mechanistic interpretability analyzes internal LLM states to understand code processing. Research indicates that attention mechanisms inherently encode both syntactic structures (ASTs) [55] and semantic dependencies (CFGs) [35], enabling vulnerability localization methods that leverage these internal signals. Building on the linear representation hypothesis [40], activation-steering methods modify model behavior by adding dense steering vectors to hidden states during inference [25]. These approaches are lightweight and effective when a single global direction is sufficient to bias model behavior. However, for vulnerability detection, the target signal is often sparse, weak, and entangled with dominant functional semantics. In this setting, a fixed dense intervention may not adequately separate vulnerability-specific cues from the broader semantic background. SAGE differs from steeringbased methods in two ways: it learns a sparse, task-conditional representation rather than applying a fixed dense offset, and it explicitly selects intermediate layers where the vulnerability signal is most separable before deep-layer dilution occurs. Recently, Sparse Autoencoders (SAEs) [1] have emerged as a powerful tool for disentangling polysemantic features. While Melo et al. [39] pioneered the application of SAEs to bug detection using general-purpose models [29], there remains an opportunity to further align these sparse representations with the specific, fine-grained semantics required for security tasks. This motivates the exploration of task-specific optimization for internal feature extraction. In summary, while existing approaches have evolved from static analysis to advanced deep learning, they share limitations in handling the subtle nature of vulnerability signals. Table 1 provides a systematic comparison of these paradigms. Current vulnerability detection methods mainly rely on , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
5
parsing code graphs or optimizing model weights through fine-tuning and reinforcement learning. However, these approaches either rely on heavy external priors or succumb to signal submersion by optimizing on diluted final-layer representations. SAGE works as an orthogonal framework that complements these existing methods. Instead of updating the model parameters, SAGE keeps the backbone frozen and intervenes directly at the intermediate layers. This approach is not limited by the model’s previous training or alignment history. By focusing on these internal layers, we can find and use hidden features that are usually lost during standard end-to-end training. Table 1. Comparison of Technical Paradigms. SAGE is an orthogonal framework enhancing internal representations, compatible with SFT, RL, and Agents. Category & Rep. Works
Core Mechanism
Graph-based
Explicit Structure Modeling. Utilizes Graph Neural Networks (GNNs) to learn from intermediate representations like Code Property Graphs (CPG) or PDGs.
[9, 26]
Supervised Fine-Tuning [19, 60]
Weight Optimization. Updates global model parameters via end-to-end backpropagation to align final-layer probability distributions with ground truth.
Reinforcement Learning Policy Alignment. Optimizes the reasoning generation policy using reward signals (e.g., COPO/GRPO) derived from correct detection outcomes. [56] Agent-based [37, 53]
Activation Engineering [25]
Dict. Learning [39]
SAGE (Ours)
3
Workflow Orchestration. Simulates multi-role debates or sequential retrieval-augmented workflows to decompose complex detection tasks. Dense Intervention. Modifies behavior by adding a fixed, dense steering vector to the activation space during inference. Unsupervised Extraction. Decomposes activations into sparse features using Autoencoders (e.g., Gemma Scope) on general corpora. Pan-Layer Perception + Task-Conditional SAE. Relation: Orthogonal to SFT/RL (operates with frozen backbone parameters, compatible with any model regardless of its alignment history) and Complementary to Agents (enhancing atomic capability).
Method
As we illustrated, the core challenge in vulnerability detection is Signal Submersion, where critical intermediate features are progressively overwhelmed by global semantics. Critically, post-training approaches [19, 49, 56, 60] do not explicitly model the internal processing. By ignoring the evolution of intermediate features and calculating loss on representations where signals are already diluted, these methods may rely on spurious correlations, leading to poor generalization. To overcome this, SAGE actively intervenes in the model’s intermediate information flow. By projecting entangled activations into a sparse manifold via a task-conditional Sparse Autoencoder (SAE), SAGE explicitly disentangles and amplifies faint vulnerability signals. This section begins by formally defining the Signal Submersion problem, followed by a detailed technical breakdown of SAGE’s three core components: Pan-Layer Latent Perception, Sparse Manifold Projection, and Task-Conditional Latent Alignment. 3.1
Motivation and Problem Formulation
𝑁 denote the dataset, where 𝑥 is the source code and 𝑦 ∈ {0, 1} is Formally, let D = {(𝑥𝑖 , 𝑦𝑖 )}𝑖=1 the label (0 for safe, 1 for vulnerable). For a Transformer-based LLM with 𝐿 layers, let ℎ𝑙 (𝑥) ∈ R𝑑 represent the latent activation of input 𝑥 at layer 𝑙.
, Vol. 1, No. 1, Article . Publication date: April 2018.
6
Trovato et al. (𝑙 ) We first categorize the activations at layer 𝑙 into two sets based on their labels: 𝐻𝑠𝑎𝑓 = {ℎ𝑙 (𝑥) | 𝑒
(𝑙 ) (𝑙 ) (𝑙 ) (𝑙 ) (𝑥, 𝑦) ∈ D, 𝑦 = 0} and 𝐻 𝑣𝑢𝑙𝑛 = {ℎ𝑙 (𝑥) | (𝑥, 𝑦) ∈ D, 𝑦 = 1}. Let 𝐻𝑎𝑙𝑙 = 𝐻𝑠𝑎𝑓 ∪ 𝐻 𝑣𝑢𝑙𝑛 denote the 𝑒 entire population of activations. Drawing on the Linear Representation Hypothesis [25, 40], we introduce two dataset-level geometric statistics at layer 𝑙: a semantic centroid 𝑠𝑙 and a vulnerability direction 𝑣𝑙 . Importantly, we do not regard these quantities as an exact sample-wise causal decomposition of every activation. Instead, they define a descriptive coordinate system for characterizing how vulnerability-relevant information is organized relative to the dominant semantic background: (𝑙 ) (𝑙 ) (𝑙 ) 𝑠𝑙 ≜ Mean 𝐻 all , 𝑣𝑙 ≜ Mean 𝐻 vuln − Mean 𝐻 safe .
For an individual sample 𝑥, we express its activation relative to this coordinate system as ℎ𝑙 (𝑥) = 𝑠𝑙 + 𝛼𝑙 (𝑥) 𝑣𝑙 + 𝑟𝑙 (𝑥),
(1)
where
⟨ℎ𝑙 (𝑥) − 𝑠𝑙 , 𝑣𝑙 ⟩ ∥𝑣𝑙 ∥ 22 measures the alignment of sample 𝑥 with the vulnerability direction, and 𝑟𝑙 (𝑥) denotes the residual component not explained by this direction. Under this formulation, 𝑠𝑙 represents the dominant semantic background shared across samples, while 𝛼𝑙 (𝑥)𝑣𝑙 captures the class-discriminative component aligned with vulnerability-related variation. This formulation is descriptive rather than causal: throughout the paper, we use the term “vulnerability signal” to refer to the discriminative component aligned with 𝑣𝑙 , not to a uniquely identifiable latent factor. 𝛼𝑙 (𝑥) =
d LLM
Inference
Attention Weight Matrix
Input: Code with a Subtle Vulnerability
The
Standar
model pays attention to the global context, assigning low weights f
to the speci ic bug.
1.000
0
0
0
0
0
0.989
0.011
0
0
0
0
0.558
0.371
0.301
0
0
0
(Irrelevant to detection)
Decoded Feature Matrix
Sparse
...
0.327
0.270
0.035
0.368
0
AGE
S
General Semantics
0
0.07
0
0.12
0
0
1
0
0
0.7
0
0
0
0
0.93
0.96
0
0
0
0
0
0.85
0
0
0
0
0
0
0
0.98
0
0
0.89
0
0.15
0
Auto Encoder
0
... 0.232
0.289
0.029
0.297
0.153
0
Vulnerability
Patch
0.241
0.201
0.185
0.015
0.211
0.147
Signal Dilution via
minance of Global mantics
Global Attention
Value Matrix
Do Se
0.92
-0.10
0.34
0.55
1.10
0.22
0.18
0.41
MLP Classifier
gnal Submersion
Si
f
minated by global semantics, leaving the f
The inal representation is do
...
Vulnerability Features
(High activation indicates vulnerability)
Global Semantics(0.85)
0.90 Vulnerability Signal(0.15)
vulnerability signal invisible to the inal output layer.
Global Semantics(0.98)
0.15
0.02
0.01
0.08
-0.02
0.87
-0.32
0.45
0.29
0.32
0.73
0.11
0.39
0.62
0.48
Yes:
Vulnerability Signal(0.02)
No:
Global Semantics (0.99)
Vulnerability Signal(0.01)
No security vulnerability.
Token Sequence
A security vulnerability
Global Semantics (0.92)
Output
Vulnerability Signal(0.08)
0.95
-0.04
0.27
0.51
Attention Output
, massive amount of 'safe' context merically overwhelms the single vulnerability token.
During aggregation the
Output
detected.
Unlike standard inference where the defect is drowned out, SAGE recovers the submerged signal. The Decoded Feature Matrix explicitly isolates the
maly, providing a -fidelity signal that model to flip the f m "Safe" to
security ano
,
clean high
enables the
prediction ro
"Vulnerable.
nu
Fig. 1. Illustration of Signal Submersion and SAGE. The self-attention mechanism assigns dominant weights to global functional context (𝑠𝑙 ), causing the sparse vulnerability features (𝑣𝑙 ) to be diluted during aggregation. SAGE addresses this by using a Sparse Autoencoder (SAE) to amplify the submerged vulnerability signal from the functional background.
The Signal Submersion Phenomenon. While 𝑣𝑙 may be distinct in shallow layers, it suffers from a dilution effect as depth 𝑙 increases. As illustrated in Fig. 1, the self-attention mechanism aggregates long-range functional context. Even if a specific token captures a defect, its contribution is numerically suppressed by the massive weight of global functional tokens during the weighted sum calculation. Consequently, the vulnerability signal becomes negligible compared to the accumulating functional background in the final layer: ||𝑠𝐿 || 2 ≫ ||𝑣 𝐿 || 2 . , Vol. 1, No. 1, Article . Publication date: April 2018.
(2)
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
7
Our empirical analysis confirms this, showing the relative magnitude of 𝑣𝑙 drops below 5% in deeper layers. This inequality reveals a bottleneck for conventional methods that rely on the final representation ℎ𝐿 . Since the discriminative component aligned with v𝐿 is submerged by the dominant semantic component centered around s𝐿 , the optimization objective degenerates. Models effectively either performance degrade to a majority-class prior (ignoring the weak 𝑣 𝐿 ) or overfit to stylistic heuristics within 𝑠𝐿 as proxies for vulnerability. This necessitates our SAGE framework, which actively intervenes at an intermediate layer 𝑙 ∗ to recover 𝑣𝑙 before the Signal-to-Noise Ratio (SNR) deteriorates. 3.2
Pan-Layer Latent Perception
Process-Based Perception via Intermediate Intervention. The Signal Submersion analysis reveals that the final representation ℎ𝐿 operates in a structurally low SNR regime. We actively intervene in the model’s inference trajectory, extracting latent states at intermediate layers (𝑙 < 𝐿) to recover vulnerability artifacts before they are submerged by global semantics. Hidden State Extraction. To intervene effectively, we must aggregate the sequence of hidden states 𝐻 (𝑙 ) into a representative vector 𝑧 (𝑙 ) . We leverage the autoregressive properties of decoderonly large language models to extract the activation state of the last token in a token sequence. The causal attention mask ensures that the last token is the unique position that mathematically aggregates the receptive field of the entire code context {𝑡 1, . . . , 𝑡𝑇 }. Accordingly, we define the layer-wise representation as: 𝑧 (𝑙 ) = T (𝐻 (𝑙 ) ) ≜ ℎ𝑡(𝑙 )
. 𝑡 =𝑇
(3)
This operator preserves the cumulative reasoning trajectory encoded by the causal chain. While other aggregation methods (e.g., Mean Pooling, Max Pooling) exist, we prioritize the last token to avoid smoothing out high-frequency anomaly signals. A detailed empirical comparison demonstrating the superiority of this extraction strategy over others is provided in Section 5.2. Layer-wise Independent Probing. Analyzing the model state at a specific layer is insufficient because the signals indicating a vulnerability vary across different depths. These signals may be strong in some layers but weak in others. To address this, SAGE employs task-conditional Sparse Autoencoders (SAEs) to analyze the representation 𝑧 (𝑙 ) at each layer individually. Because the backbone is frozen, these layer-wise SAEs are trained on cached hidden states rather than through repeated end-to-end backpropagation. The overhead therefore scales linearly with the number of probed layers, but avoids the substantially higher cost of updating the full backbone. This approach allows SAGE to scan through the model and identify exactly which layer provides the clearest and most separable vulnerability features. 3.3
Sparse Manifold Projection via SAE
Separating Vulnerabilities. While the extracted latent states 𝑧 (𝑙 ) record the inference process, they represent multiple different concepts mixed together within a dense vector [1]. In this dense representation, strong features such as syntax often dominate the weaker signals associated with vulnerabilities. Consequently, the vulnerability features are not readily separable using linear methods. To address this, we hypothesize that mapping the state into a space with significantly more dimensions (𝑑𝑠𝑎𝑒 ≫ 𝑑𝑖𝑛 ) will separate these mixed features. We employ an SAE to isolate the vulnerability signal as an independent and distinct feature. JumpReLU for Magnitude Preservation. Standard sparse coding methods typically rely on 𝐿1 regularization. However, this approach tends to reduce the value of all activations to achieve sparsity. Since the vulnerability signals are already weak (Eq. 2), this reduction may reduce important , Vol. 1, No. 1, Article . Publication date: April 2018.
8
Trovato et al.
information along with the noise. To address this limitation, we utilize a JumpReLU SAE [41]. Let ℎ𝑙 ∈ R𝑑 be the input activation vector. We project ℎ𝑙 using a learnable encoder weight matrix 𝑊𝑒𝑛𝑐 ∈ R𝑑𝑠𝑎𝑒 ×𝑑 and a bias vector 𝑏𝑒𝑛𝑐 ∈ R𝑑𝑠𝑎𝑒 , followed by a thresholding operation: 𝑧 = 𝜎𝜃 (𝑊𝑒𝑛𝑐 ℎ𝑙 + 𝑏𝑒𝑛𝑐 ),
where 𝜎𝜃 (𝑢)𝑖 = 𝑢𝑖 · I(𝑢𝑖 > 𝜃 𝑖 ).
(4)
Here, 𝜃 is a learnable threshold vector that acts as a filter. This mechanism ensures magnitude preservation. Once a feature 𝑧𝑖 exceeds its threshold 𝜃 𝑖 , its original value is kept unchanged. Consequently, SAGE effectively filters out noise by setting it to zero without weakening the retrieved vulnerability signal. Optimization Objective. We train the SAE to minimize a combined loss function. This function balances two goals: accurately reconstructing the input activation ℎ𝑙 and maintaining sparsity. Unlike standard methods that penalize the magnitude of activations, we aim to minimize the number of active features. Since directly counting non-zero elements is difficult for optimization, we use a sigmoid function 𝜎𝑠𝑖𝑔 to approximate this count: 𝑑𝑠𝑎𝑒 1 ∑︁ 𝑢𝑖 − 𝜃 𝑖 L𝑆𝐴𝐸 = ||ℎ𝑙 − 𝑊𝑑𝑒𝑐 𝑧|| 22 +𝜆 · . 𝜎𝑠𝑖𝑔 𝑑𝑠𝑎𝑒 𝑖=1 𝜏 | {z } | {z } L𝑟𝑒𝑐𝑜𝑛
(5)
L𝑠𝑝𝑎𝑟𝑠𝑒 (≈𝐿0 )
Here, 𝑊𝑑𝑒𝑐 ∈ R𝑑 ×𝑑𝑠𝑎𝑒 is the decoder weight matrix that maps the sparse feature 𝑧 back to the original dimension. The parameter 𝜏 controls how sharply the function distinguishes between active and inactive features. By minimizing L𝑠𝑝𝑎𝑟𝑠𝑒 , SAGE suppresses noise and improves the SNR. However, it is important to note that this optimization is unsupervised. The model aims to reconstruct the dominant semantic structure of the code and does not prioritize vulnerabilityspecific features. Since vulnerability signals are often faint and appear infrequently, the model tends to discard them to minimize the overall reconstruction error. Consequently, the standard SAE may fail to isolate these critical signals. To address this limitation, we introduce Task-Conditional Latent Alignment. 3.4
Task-Conditional Latent Alignment
Gradient Injection via Weighted Supervision. To transform the SAE from a passive feature extractor into a guided discovery engine, we introduce Task-Conditional Latent Alignment. We attach a linear probe 𝑔𝜙 directly to the sparse feature 𝑧. Crucially, to address the extreme class imbalance where safe samples vastly outnumber vulnerabilities, we employ Inverse-Frequency Risk Minimization. We formulate the supervision as a Weighted Cross-Entropy loss: 𝐵
L𝑐𝑙𝑎𝑠𝑠 = −
𝐶
1 ∑︁ ∑︁ 𝜔𝑐 · 𝑦𝑖,𝑐 log(𝑦ˆ𝑖,𝑐 ), 𝐵 𝑖=1 𝑐=1
(6)
where 𝑛𝑐 denotes the number of training samples in class 𝑐, and 𝐶 · (1/𝑛𝑐 ) 𝜔𝑐 = Í𝐶 𝑗=1 (1/𝑛 𝑗 ) is the normalized inverse-frequency weight for class 𝑐, with 𝐶 being the number of classes. This normalization ensures that the average class weight equals 1, so that minority classes receive larger gradients without changing the overall loss scale. , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
9
We integrate this objective into a unified optimization framework: L𝑡𝑜𝑡𝑎𝑙 = L𝑟𝑒𝑐𝑜𝑛 + 𝜆 · L𝑠𝑝𝑎𝑟𝑠𝑒 +𝛾 · L𝑐𝑙𝑎𝑠𝑠 . |{z} | {z }
(7)
Unsupervised Structure Supervised Guidance
This architecture establishes a direct connection for gradient flow. During backpropagation, the weighted gradients 𝛾 ∇L𝑐𝑙𝑎𝑠𝑠 from the classifier are propagated back to the SAE encoder. This process guides the JumpReLU activation mechanism. Unlike standard unsupervised learning, which separates all features indiscriminately, these gradients provide a supervised signal. These gradients guide the encoder to prioritize vulnerability patterns, regardless of their signal strength. At the same time, the model is encouraged to down-weight strong but task-irrelevant features, such as coding style. We do not claim that this fully eliminates shortcut learning. Rather, SAGE reduces this risk through three design choices: strict de-duplication and temporal partitioning in the data pipeline, a sparse bottleneck that penalizes diffuse correlations, and evaluation on distribution-shifted datasets where superficial cues transfer poorly. [48]. 4
Experimental Design
In this section, we present a comprehensive evaluation of the SAGE framework, utilizing refined and deduplicated versions of three rigorously curated benchmarks: BigVul [17], PrimeVul [15], and PreciseBugs [22]. Our experimental design extends beyond standard performance metrics; we aim to empirically validate the existence of the Signal Submersion phenomenon and demonstrate SAGE’s capability to recover these latent vulnerability signals. Specifically, we structure our analysis around four Research Questions (RQs) that progress from macroscopic performance comparisons to microscopic mechanistic interpretations, concluding with an investigation into the relationship between general coding capabilities and security sensitivity: RQ1 (Overall Effectiveness): How does SAGE perform on standard vulnerability detection benchmarks compared to state-of-the-art baselines? Given the challenge of detecting minute semantic deviations within complex code, we establish a rigorous comparison against diverse baselines, ranging from graph-based methods to recent LLMs. This validates whether SAGE successfully amplifies critical vulnerability signals to achieve superior detection accuracy. RQ2 (Mechanistic Verification): Does SAGE quantitatively and qualitatively mitigate Signal Submersion via robust feature decoupling? To validate the structural grounding of our approach, we analyze latent space properties through Internal Signal-to-Noise Ratio (SNR) and Top-𝐾 feature attribution. These analyses confirm that SAGE effectively recovers submerged signals via robust feature decoupling rather than relying on dense noise. RQ3 (Component Efficacy & Stability): How do different components contribute to SAGE, and is the performance robust? We assess the necessity of key components through a comprehensive ablation study, isolating the contributions of Sparse Manifold Projection and task-alignment modules. Additionally, we conduct hyperparameter sensitivity analysis to ensure that the reported performance gains are robust and stable across varying configurations. RQ4 (Impact of Model Scale): How does the Signal Submersion effect and SAGE’s effectiveness evolve across different model scales? We investigate whether the Signal Submersion effect persists or diminishes as model parameters and complexity increase. This analysis determines if SAGE remains essential for large-scale foundation models or if scaling naturally resolves the dominance of global semantics. , Vol. 1, No. 1, Article . Publication date: April 2018.
10
4.1
Trovato et al.
Studied LLMs
Table 2 lists our studied LLMs with their categories, model architectures, release timelines, parameter sizes, and effective context window limits. To ensure a comprehensive evaluation across the current landscape of LLMs, we systematically selected representative models from three distinct categories: (1) General-Purpose Open-Weights LLMs (i.e., Mistral-7B [24] and Llama-3.1-8B [52]), which serve as strong baselines for generalized reasoning; (2) Code-Specialized LLMs (i.e., the CodeLlama family [44] and Qwen2.5-Coder family [23]), which are explicitly optimized for programming tasks; and (3) Proprietary Frontier Models (i.e., GPT-5.2-Codex [2], Gemini 3 Pro [4], and Claude Opus 4.5 [3]), which represent the current state-of-the-art in closed-source commercial performance. Table 2. Studied LLMs in this paper Category
Model
Model Type
Time
Size
Training Base
In/Out (Tokens)
Commercial
GPT-5.2-Codex [2] Gemini 3 Pro [4] Claude Opus 4.5 [3]
Instruction Model Instruction Model Instruction Model
2025-12 2025-11 2025-11
/ / /
/ / /
400k / 128k 1M / 64k 200k / 64k
General
Llama-3.1-8B [52] Mistral-7B [24]
Instruction Model Base Model
2024-07 2023-10
8B 7B
15 trillion /
128k / – 32k / –
CodeLlama-7B [44] CodeLlama-13B [44] CodeLlama-34B [44] Qwen2.5-Coder-7B [23] Qwen2.5-Coder-14B [23] Qwen2.5-Coder-32B [23]
Instruction Model Instruction Model Instruction Model Instruction Model Instruction Model Instruction Model
2023-08 2023-08 2023-08 2024-09 2024-09 2024-11
7B 13B 34B 7B 14B 32B
500 billion 500 billion 500 billion 5.5 trillion 5.5 trillion 5.5 trillion
16k / – 16k / – 16k / – 128k / – 128k / – 128k / –
Code
Φ In the Training Base column, “/” denotes undisclosed proprietary data. For CodeLlama, the 500 billion tokens represent code-specific
training on top of the Llama 2 base. For open-weights models, the maximum output length is dynamically determined by the remaining context window. Therefore, we report the total context limit as the input capacity and mark the output as “–” to indicate this dynamic dependency.
4.2
Datasets and Preprocessing
To ensure ecological validity and mitigate data leakage [8, 11], we employ distinct preprocessing protocols to curate refined versions of BigVul [17], PrimeVul [15], and PreciseBugs [22]. Fuzzy De-duplication Protocol. Since BigVul and PrimeVul are historical snapshots likely present in LLM pre-training corpora, standard splitting is ineffective. To counter memorization, we implement a strict semantic filtering mechanism. We define an abstraction mapping 𝜓 to neutralize trivial perturbations and utilize SimHash with Jaccard coefficient J for efficient retrieval. We construct a purified training set 𝐷𝑡𝑟𝑎𝑖𝑛 by enforcing a similarity threshold 𝜏 = 0.75 against the test set: 𝑟𝑎𝑤 𝐷𝑡𝑟𝑎𝑖𝑛 = {𝑥 ∈ 𝐷𝑡𝑟𝑎𝑖𝑛 | ∀𝑦 ∈ 𝐷𝑡𝑒𝑠𝑡 , J (𝜓 (𝑥),𝜓 (𝑦)) < 𝜏 }, (8) | where J (𝐴, 𝐵) = |𝐴∩𝐵 |𝐴∪𝐵 | . This ensures no training sample is semantically isomorphic to any held-out test sample. Temporal Partitioning. For the continuously updated PreciseBugs, we adopt a temporal splitting protocol [60] to simulate zero-day detection. We set a global cutoff 𝑇𝑐𝑢𝑡𝑜 𝑓 𝑓 at January 2023, aligning with standard baselines like CodeLlama [44] and strictly predating evaluation samples for most studied models. The evaluation set comprises post-cutoff samples: 𝑃𝐵 𝑃𝐵 𝐷𝑡𝑒𝑠𝑡 = {𝑥 ∈ 𝐷𝑡𝑜𝑡𝑎𝑙 | Time(𝑥) > 𝑇𝑐𝑢𝑡𝑜 𝑓 𝑓 }.
(9)
This formulation minimizes memorization bias and establishes a consistent benchmark across model generations. Dataset Statistics and Diversity. Table 3 summarizes the statistics of the processed datasets. Regarding linguistic diversity, while BigVul and PrimeVul predominantly feature C/C++ vulnerabilities typical of systems software, PreciseBugs is inherently multi-lingual (comprising C/C++, Java, , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
11
Python, and JavaScript). This inclusion allows us to evaluate SAGE’s cross-language generalization capabilities beyond the single-language limitations of traditional benchmarks. Table 3. Detailed statistics of the refined datasets. To conserve space, complexity metrics (Tokens, LoC, CCN) are reported as Mean ± Std, excluding median/p95 values. Dataset
Lang.
Samples (Train / Test)
Balance Ratio (Pos:Neg Train/Test)
Tokens (Mean ± Std)
LoC (Mean ± Std)
CCN (Mean ± Std)
BigVul PrimeVul
C/C++ C/C++
16296 / 18864 9157 / 24788
1:1 / 1:17 1:1 / 1:44
471.3 ± 1242.7 693.0 ± 2459.3
40.7 ± 110.4 57.1 ± 206.4
6.7 ± 12.6 8.9 ± 16.4
PreciseBugs C# C/C++ Go Java JavaScript Kotlin Objective-C PHP Python Ruby Rust Swift TypeScript unknown
C# C/C++ Go Java JS Kotlin Obj-C PHP Python Ruby Rust Swift TS N/A
111 / 53 5957 / 315 1110 / 181 908 / 180 1593 / 247 5/6 19 / 0 4452 / 946 1032 / 180 409 / 75 79 / 12 23 / 6 315 / 64 3148 / 510
1:4 / 1:2 1:4 / 1:4 1:4 / 1:4 1:4 / 1:5 1:4 / 1:3 0:1 / 1:5 1:5 / 0:0 1:4 / 1:4 1:4 / 1:4 1:4 / 1:3 1:9 / 1:11 1:2 / 1:2 1:5 / 1:3 1:4 / 1:4
4830.7 ± 10247.9 17883.9 ± 16986.8 5629.4 ± 9218.4 4955.7 ± 7583.8 36723.8 ± 239449.4 7722.7 ± 9962.7 8185.6 ± 4269.1 7943.1 ± 11725.2 8832.0 ± 11079.9 5214.6 ± 7590.1 7414.9 ± 6835.6 8163.0 ± 12955.4 3606.0 ± 5916.6 13077.9 ± 25644.2
405.5 ± 587.0 1507.1 ± 1366.2 459.6 ± 694.4 428.1 ± 639.5 715.8 ± 1074.2 719.7 ± 940.8 776.6 ± 403.6 627.5 ± 858.8 735.7 ± 864.9 489.8 ± 689.7 654.6 ± 603.9 688.9 ± 986.4 364.2 ± 586.1 812.6 ± 1162.8
1.4 ± 1.4 2.8 ± 7.9 3.1 ± 4.3 2.2 ± 3.5 2.0 ± 3.3 1.6 ± 1.0 1.1 ± 0.2 2.1 ± 5.6 1.9 ± 2.4 1.7 ± 2.5 1.8 ± 2.2 3.3 ± 9.5 1.8 ± 3.0 1.3 ± 1.9
Total
Mix
19161 / 2775
1:4 / 1:4
13660.4 ± 71581.8
894.6 ± 1145.6
2.2 ± 5.5
4.3
Evaluation Metrics
To provide a comprehensive assessment of SAGE, we report standard classification metrics including Precision, Recall, and F1-Score. However, given the extreme class imbalance inherent in vulnerability detection datasets (e.g., BigVul contains only 6% vulnerable samples [17], and PrimeVul 2.2% [15]), standard accuracy can be misleading, as a trivial model predicting the majority class would achieve high performance. To address this, we prioritize two robust metrics: Balanced Accuracy (B.Acc) [7] and the Matthews Correlation Coefficient (MCC) [38], which is defined as the arithmetic mean of sensitivity and specificity, ensuring equal weight for both minority and majority classes. Balanced Accuracy (B.Acc) calculates the arithmetic mean of sensitivity (Recall) and specificity, preventing the evaluation from being dominated by the safe class (majority): 𝑇𝑃 𝑇𝑁 1 Balanced Acc = + , (10) 2 𝑇𝑃 + 𝐹𝑁 𝑇 𝑁 + 𝐹𝑃 where 𝑇 𝑃, 𝑇 𝑁 , 𝐹 𝑃, and 𝐹 𝑁 denote True Positives, True Negatives, False Positives, and False Negatives, respectively. Furthermore, we employ the Matthews Correlation Coefficient (MCC), widely regarded as the most informative single metric for imbalanced binary classification. As demonstrated by Chicco and Jurman [10], MCC is more informative than F1-score and accuracy in evaluating binary classification problems on imbalanced datasets, as it considers all four confusion matrix categories symmetrically: MCC = √︁
𝑇𝑃 × 𝑇 𝑁 − 𝐹𝑃 × 𝐹𝑁 , (𝑇 𝑃 + 𝐹 𝑃) (𝑇 𝑃 + 𝐹 𝑁 ) (𝑇 𝑁 + 𝐹 𝑃) (𝑇 𝑁 + 𝐹 𝑁 )
(11)
MCC values range from −1 to +1, where +1 indicates perfect prediction, 0 indicates random guessing, and −1 indicates total disagreement. , Vol. 1, No. 1, Article . Publication date: April 2018.
12
4.4
Trovato et al.
Implementation Details
All experiments were conducted on NVIDIA GeForce RTX 3090 GPUs. We implemented SAGE using PyTorch and the HuggingFace ecosystem. SAGE Configuration. For the SAE component, we set the expansion factor 𝐹 = 16, mapping the hidden dimension to an overcomplete space of 𝐹 ×𝑑𝑚𝑜𝑑𝑒𝑙 . The model was trained for 10 epochs with a batch size of 256 and a learning rate of 1e−4. The coefficients for the composite loss function (Eq. 5) were set to balance the competing objectives: sparsity penalty weight 𝜆 = 0.5, and classification weight 𝛾 = 1.0. We utilized an auxiliary 𝐿0 coefficient of 1.0 to govern the sparsity thresholding. Unless otherwise stated, all task-conditional SAEs and their downstream classifiers are trained separately for each dataset-backbone pair; no sparse representation is shared across datasets. Overlength Handling. PreciseBugs contains many long files that may exceed the context limits of some backbones. To ensure a fair and reproducible comparison, we apply an explicit span-centered windowing strategy to overlong samples. Specifically, for each buggy or fixed file, we construct a fixed 4096-token input window centered on the annotated buggy or fixing span whenever possible. If the annotated span itself exceeds this budget, we anchor the window at the span start and keep the earliest 4096 tokens covering that region. The same preprocessing policy is applied consistently to SAGE and to all compared methods that operate on raw code inputs, so that performance differences are not confounded by inconsistent truncation rules. We will release this preprocessing step together with the data pipeline to facilitate reproduction. Implementation Details. We adhered to official optimal settings, adapting for hardware constraints (all models trained for 3 epochs): • LineVul [19]: Trained with learning rate (LR) 1e−5, block size 512, batch size 2, and gradient clipping norm 1.0. • MSIVD [60]: Utilized 4-bit QLoRA (𝑟 = 96, 𝛼 = 32, dropout 0.05), LR 1e−4, and sequence length 4096. • VulLLM [16]: Employed LoRA (𝑟 = 16, 𝛼 = 32, dropout 0.1) with cosine scheduler (0.1 warmup). Set LR to 2e−4, per-device batch size 2 with 8 gradient accumulation steps (FP16). • ReVD [56]: Trained using IPO objective (𝛽 = 0.1) and LoRA on all linear layers. Configured with LR 5e−6, batch size 1 with 8 accumulation steps (BF16). • SteerLLM [25]: Followed original paired-data configs. For PreciseBugs (unpaired), we derived steering vectors by subtracting mean activations of non-vulnerable from vulnerable instances. 5 5.1
Experimental Results RQ1: Overall Effectiveness
We evaluated SAGE against 11 baselines on BigVul [17], PrimeVul [15], and the zero-day benchmark PreciseBugs [22] across four model families. Due to extreme class imbalance (e.g., 2.2% positive rate in PrimeVul), we prioritize Matthews Correlation Coefficient (MCC) over Balanced Accuracy (B.Acc) and F1, as MCC provides a truthful representation of discriminative capability by symmetrically penalizing false positives and negatives [10]. Superiority and Generalization. As shown in Table 4, SAGE consistently achieves State-ofthe-Art (SOTA) performance. While structural methods (e.g., ReVeal) struggle with parsing failures and standard SFT/Prompting strategies performance degrade on the zero-day PreciseBugs benchmark (yielding near-zero or negative MCCs), SAGE exhibits improved generalization. Specifically, SAGE (CodeLlama) sets new records on BigVul (0.7874 MCC) and PrimeVul (0.2375 MCC). On PreciseBugs, SAGE (Llama-3.1) attains the highest MCC of 0.1178, outperforming the proprietary frontier model Claude Opus 4.5 [3]. Crucially, SAGE proves to be backbone-agnostic; for instance, it boosts Llama-3.1’s MCC on BigVul from negative values (in Prompting/MSIVD) to a robust 0.7506, , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
13
Table 4. Comprehensive effectiveness comparison across different paradigms and backbones. The Prompt, SFT, RL, Activation, and Ours categories are evaluated across four distinct model families: CodeLlama-7B [44], Mistral-7B [24], Llama-3.1-8B [52], and Qwen2.5-Coder-7B [23]. BigVul [17]
Category Method (Backbone)
PrimeVul [15]
PreciseBugs [22] F1 ↑
B.Acc ↑
MCC ↑
Graph
ReVeal [9] IVDetect [26]
6.68 20.13
45.94 49.62
11.67 28.65
54.01 57.32
0.0379 0.1084
3.84 13.26
78.66 25.14
7.33 17.37
55.09 58.67
0.0396 0.1287
– –
– –
– –
– –
– –
Prompt
CodeLlama-7B [44] Mistral-7B [24] Llama-3.1-8B [52] Qwen2.5-Coder-7B [23]
7.23 9.69 5.39 6.28
9.86 52.42 95.73 75.83
8.34 16.35 10.21 11.59
51.18 61.74 48.11 54.38
0.0204 0.1174 -0.1041 0.0430
5.12 3.20 1.98 2.28
22.59 48.09 87.61 79.05
8.34 6.01 3.86 4.43
56.55 57.59 44.56 51.10
0.0649 0.0474 -0.1222 0.0077
18.23 19.58 19.44 18.71
10.65 63.76 98.15 57.84
13.44 29.96 32.46 28.27
49.55 50.20 49.88 48.52
-0.0113 0.0033 -0.0072 -0.0240
LineVul (CodeLlama) [19] LineVul (Mistral) LineVul (Llama-3.1) LineVul (Qwen2.5)
10.87 12.60 13.30 10.74
90.26 72.50 93.60 75.29
19.40 21.47 23.29 18.80
72.02 70.54 77.75 68.11
0.2072 0.2043 0.2651 0.1731
4.50 4.11 4.66 5.06
84.82 88.70 78.58 79.09
8.54 7.85 8.80 9.51
69.54 68.07 68.88 70.69
0.1218 0.1126 0.1190 0.1321
12.50 20.00 14.29 7.69
1.03 1.06 1.03 1.03
1.90 2.01 1.92 1.82
49.76 50.02 49.87 49.22
-0.0153 0.0013 -0.0089 -0.0392
VulLLM (CodeLlama) [16] VulLLM (Mistral) VulLLM (Llama-3.1) VulLLM (Qwen2.5)
8.14 22.11 5.63 9.45
86.73 90.81 100.00 30.90
14.88 35.57 10.66 14.48
64.36 85.93 50.34 56.68
0.1345 0.3925 0.0196 0.0794
3.25 2.43 2.31 2.50
91.62 98.36 99.82 32.24
6.29 4.74 4.51 4.64
64.97 54.44 52.07 51.87
0.0909 0.0430 0.0303 0.0122
18.43 19.33 19.52 19.52
78.19 95.56 99.82 98.15
29.83 32.16 32.66 32.57
47.22 49.53 50.11 50.10
-0.0583 -0.0199 0.0144 0.0059
MSIVD (CodeLlama) [60] MSIVD (Mistral) MSIVD (Llama-3.1) MSIVD (Qwen2.5)
13.45 10.15 5.51 76.92
1.51 75.36 98.39 0.95
2.73 17.89 10.44 1.87
50.47 67.92 49.26 50.47
0.0272 0.1671 -0.0738 0.0832
13.45 6.35 2.11 6.39
2.91 43.72 94.90 63.39
4.79 11.10 4.14 11.61
51.24 64.56 47.70 71.18
0.0528 0.1192 -0.0874 0.1506
19.38 19.14 19.49 20.00
79.59 58.06 98.89 0.44
31.17 28.78 32.56 0.86
48.41 48.36 50.02 50.01
-0.0332 -0.0268 0.0016 0.0009
RL
ReVD (CodeLlama) [56] ReVD (Mistral) ReVD (Llama-3.1) ReVD (Qwen2.5)
18.80 66.14 7.72 33.03
93.65 71.09 93.46 6.82
31.31 68.52 14.26 11.31
84.84 84.47 63.63 53.00
0.3572 0.6664 0.1340 0.1290
5.86 3.55 3.66 5.49
40.26 42.99 70.86 4.19
10.24 6.55 6.97 4.75
62.81 58.25 64.33 51.28
0.1050 0.0548 0.0852 0.0292
19.44 19.58 19.34 19.57
92.05 97.41 97.04 97.60
32.10 32.61 32.26 32.60
49.85 50.29 49.55 50.25
-0.0044 0.0144 -0.0241 0.0121
Activation
SteerLLM (CodeLlama) [25] SteerLLM (Mistral) SteerLLM (Llama-3.1) SteerLLM (Qwen2.5)
10.68 11.02 5.41 6.54
18.77 41.52 96.11 81.33
13.61 17.42 10.25 12.11
54.73 60.83 48.32 56.24
0.0730 0.1220 -0.0914 0.0623
3.24 3.33 2.03 2.08
53.19 38.80 89.25 83.61
6.11 6.13 3.98 4.06
58.62 56.63 45.94 47.24
0.0527 0.0446 -0.0722 -0.0260
17.16 17.96 20.54 20.82
57.87 40.56 63.49 44.80
26.47 24.89 31.03 28.43
45.13 47.86 50.22 50.53
-0.0813 -0.0341 0.0036 0.0087
Agent
ConColl (CodeLlama) [53] ConColl (Mistral) ConColl (Llama-3.1) ConColl (Qwen2.5)
5.22 11.35 13.79 17.50
91.00 35.17 15.55 1.33
9.87 17.16 14.62 2.47
46.55 59.44 54.89 50.48
-0.1017 0.1146 0.0926 0.0338
1.76 4.00 14.44 7.80
75.77 47.72 27.66 2.00
3.44 7.38 18.98 3.19
39.91 60.88 47.31 50.73
-0.1432 0.0726 -0.0431 0.0287
19.12 19.40 19.44 20.92
80.40 68.20 33.43 4.22
30.89 30.21 24.58 7.02
49.06 49.82 49.96 50.18
-0.0194 -0.0030 -0.0007 0.0073
Frontier
GPT-5.2-Codex [2] Gemini 3 Pro [4] Claude Opus 4.5 [3]
10.15 8.45 18.93
52.38 71.28 30.48
17.00 15.11 23.36
62.52 62.77 61.39
0.1269 0.1175 0.1829
3.64 3.27 4.86
56.83 84.34 85.45
6.85 6.30 9.19
61.40 63.92 73.73
0.0909 0.0828 0.1433
19.11 18.87 20.99
46.38 60.48 36.18
27.07 28.76 26.57
49.54 48.90 51.68
-0.0072 -0.0179 0.0282
Ours
SAGE (CodeLlama) SAGE (Mistral) SAGE (Llama-3.1) SAGE (Qwen2.5)
SFT
Prec. ↑ Recall ↑
F1 ↑
B.Acc ↑
MCC ↑
Prec. ↑ Recall ↑
F1 ↑
B.Acc ↑ MCC ↑
Prec. ↑ Recall ↑
82.02+307% 77.82-17% 79.86+155% 88.40+4% 0.7874+120% 17.87+33% 32.42-65% 23.04+27% 64.52-12% 0.2375+66% 23.49+12% 67.70-26% 34.88+9% 54.79+6% 0.1050+272% 77.22+17% 70.05-23% 73.46+7% 84.41-0.07% 0.7206+8% 17.70+33% 31.33-68% 22.62+25% 64.01-13% 0.2125+48% 22.55+7% 84.63-13% 35.61+9% 54.01+5% 0.1104+291% 84.70+321% 68.72-31% 75.88+165% 83.99+8% 0.7506+183% 17.78+23% 32.06-68% 22.87+20% 64.35-13% 0.2157+51% 24.91+19% 69.60-30% 36.69+12% 57.15+11% 0.1178+318% 86.97+13% 69.57-14% 77.30+170% 84.48+24% 0.7664+319% 19.16+44% 31.69-63% 23.88+37% 64.33-13% 0.2245+57% 22.90+9% 73.06-26% 34.87+7% 54.12+5% 0.0986+250%
Note: To ensure a rigorous and fair comparison of architectural effectiveness, all open-source baselines and SAGE variants are evaluated using backbones within the 7B–8B parameter range: CodeLlama-7B, Mistral-7B, Qwen2.5Coder-7B, and Llama-3.1-8B. Red (e.g., +295% ) and green (e.g., -17% ) percentages denote relative improvement and reduction, respectively, compared to the best performance among methods using the same backbone, Graph-based approaches, and commercial models. Metrics marked with ↑ indicate higher is better. The best results across all methods are highlighted in bold. Results for Graph-based baselines (ReVeal, IVDetect) on PreciseBugs are omitted due to toolchain limitations: their reliance on the Joern C/C++ parser precludes execution on the multi-language composition of this dataset.
validating that the Sparse Autoencoder successfully disentangles security features regardless of the underlying architecture. The Limitation of Pre-training Knowledge. Despite the improvements, the absolute performance decreases on unseen datasets like PrimeVul and PreciseBugs. We attribute this to a lack of prior knowledge in the base model. SAGE is designed to recover hidden vulnerability signals that already exist within the model’s representations. However, it cannot compensate for the total absence of knowledge regarding new attack vectors that were not present in the pre-training data. This result suggests that even with our enhancements, the model cannot effectively reason about completely new types of attacks without continuous learning. Analysis of Baseline Failure. Table 4 reveals a issue with the baseline models. Methods like VulLLM and MSIVD exhibit near-perfect Recall (>98%) but extremely low Precision (<5%) on unseen data. This indicates that these models tend to classify almost everything as vulnerable when they handle noisy real-world data. They lower their decision thresholds to avoid missing any potential defects. While this strategy inflates Balanced Accuracy, the low MCC scores demonstrate that they fail to effectively distinguish between safe and vulnerable code. In contrast, SAGE acts as a precise detector. The sparsity constraint in our SAE serves as a strict filter. It removes ambiguous signals and reduces false alarms. Although this approach results in lower Recall, it ensures that the detections are reliable, leading to higher Precision and MCC.
, Vol. 1, No. 1, Article . Publication date: April 2018.
14
Trovato et al.
x Finding 1 (Generalization & Reliability): Existing paradigms exhibit a "High-Recall, Low-Precision" pattern on zero-day benchmarks, degrading to random guessing. SAGE overcomes this limitation by actively filtering semantic noise, sacrificing trivial recall to achieve high-precision, robust generalization across unseen distributions. Mistral-7B
0.06 -0.05 0.11
LineVul
-0.15 0.06 -0.08 -0.09 -0.13 -0.14 -0.02 -0.06 -0.24 0.05 -0.21 0.13
VulLLM
0.07 -0.04 0.01
MSIVD
0.12
0.12
0.01 -0.09 0.06
ReVD
0.11
0.01
0.05
0.00 -0.10 0.00
0.00
0.03
0.12
-0.10 0.32
0.08
0.11
0.05
0.03
0.03
SteerLLM ConColl GPT
0.08
0.02
0.00 -0.04 0.08
0.07 -0.07 0.00
0.02
0.04
0.05
0.04
0.19
0.00
0.00
0.02
0.06
0.09
0.10
0.00
0.07
0.03
0.02 -0.02 -0.00 -0.03 0.12 -0.01 -0.05 0.03
0.20
0.06 -0.01 0.02
0.07 -0.00 0.00 -0.04 0.00
0.01 -0.01 0.21
0.08 -0.37 0.15 -0.08
0.14 -0.00 0.00
0.03 -0.10 -0.07 -0.01 0.16 -0.12 -0.13 -0.06 -0.14 -0.00
0.08
0.02
0.00
0.08 -0.04
0.00
0.00
0.04
0.00
0.00
0.00
0.02
0.00
0.00
0.00
0.00
0.00
0.00 -0.06 0.04
0.00 -0.20 -0.05 -0.03
0.00
0.00
0.00
0.00
0.04
0.00
0.00
0.00
0.00
0.00
0.00
0.00 -0.01
0.00
0.00 -0.07 0.00
0.00
-0.01 0.04
0.07
0.03 -0.00 0.00 -0.01 -0.12 0.04
0.00
0.00 -0.20 0.04
0.00
0.04
0.21 -0.07
-0.18 0.29
0.00
0.32
0.08 -0.05
0.00 -0.06 0.06
0.05 -0.10 0.00 -0.00 0.00
0.08
0.00
0.06 -0.01
0.19 -0.27 0.11
0.03 -0.04 0.00
0.05 -0.18 -0.09 0.00 -0.16 0.06
0.06
0.00
0.00 -0.11 0.06 -0.24 -0.06 -0.01 0.13 0.00
0.02
0.06
0.09
0.10
0.00
0.00
0.00
0.00 -0.03
0.07 -0.02 0.12 -0.10 -0.20 0.11 -0.02
-0.02 -0.04 -0.09 -0.07 0.02 -0.08 0.02 -0.05 0.10 0.08
Qwen2.5-Coder-7B
0.00 -0.04 0.00
0.20
0.20 -0.04 -0.02
Llama3-8B
0.02 -0.08 -0.01 0.00 -0.20 0.09 -0.01
-0.20 -0.06 -0.04 0.06 -0.08 0.00
-0.12 -0.08 -0.03 0.06 -0.05 0.00 -0.05 -0.04 -0.01 0.00 0.08
0.03 -0.09
0.12 -0.08 -0.04
0.00 -0.04 0.07 -0.04 0.00
0.24
0.00
0.07
0.32 -0.20 -0.06 0.03
0.05 -0.18 -0.09 0.00 -0.16 0.06
0.00
0.00
0.00
0.00 -0.02
0.06
-0.13 -0.01 -0.05 -0.17 0.06
0.00 -0.04 -0.13 0.14
0.04 -0.07 -0.03 -0.15 -0.18 -0.07 0.03
0.07
0.21
0.09
0.01 -0.06
0.13 -0.12 -0.22 0.02
0.00
0.00
0.00
0.00 -0.00
0.75
0.00
0.00
0.00
0.00
0.00 -0.01
0.50
0.05 -0.12 0.00 -0.01 0.04
0.00
0.00
0.00
0.00
0.02
0.01 -0.09 0.04 -0.37 0.00
0.12
0.01
0.05 -0.12 0.00 -0.01 0.00 0.10
0.00
-0.07 -0.17 -0.08 -0.07 0.04
0.08 -0.01 -0.04 -0.13 -0.01 0.00 -0.00 0.01
0.00 -0.07 0.12
0.09 -0.04 0.00
0.03
0.05
0.08
0.09
0.07
0.05 -0.18 -0.09 0.00 -0.16 0.06
0.08
0.02
0.06
0.09
0.10
0.00
0.07
0.03
0.05 -0.18 -0.09 0.00 -0.16 0.06
0.02
0.06
0.10
0.11
0.00
-0.06 0.10 -0.01 -0.05 -0.03 0.00 -0.02 -0.09 0.11 -0.17 -0.06 -0.01 0.00 0.01 -0.12 -0.20 0.31
0.00
0.02
0.00
0.00 -0.03 -0.02
Gemini
-0.16 0.06 -0.02 0.10
0.04 -0.50 -0.03 -0.17 -0.03 0.08
0.00
0.14 -0.01
-0.16 0.06 -0.02 0.10
0.04 -0.50 -0.03 -0.17 -0.03 0.08
0.00
0.14 -0.01
-0.16 0.06 -0.02 0.10
0.04 -0.50 -0.03 -0.17 -0.03 0.08
0.00
0.14 -0.01
-0.16 0.06 -0.02 0.10
0.04 -0.50 -0.03 -0.17 -0.03 0.08
0.00
0.14 -0.01
Claude
0.03
0.06
0.03
0.15
0.07
0.00
0.02 -0.03 -0.15 0.67
0.00
0.42
0.00
0.03
0.06
0.03
0.15
0.07
0.00
0.02 -0.03 -0.15 0.67
0.00
0.42
0.00
0.03
0.06
0.15
0.07
0.00
0.02 -0.03 -0.15 0.67
0.00
0.42
0.00
0.03
0.06
0.03
0.15
0.07
0.00
0.42
SAGE
0.22
0.05
0.07
0.08
0.04
0.00
0.11
0.45
0.23
0.10
-0.01 0.06
0.01
0.11
0.07
0.50
0.11 -0.01 -0.02 -0.13 0.25
0.03
0.12
0.36
0.07 -0.10 0.07
0.22
0.20
0.05
0.00
0.13
0.14
0.15
0.01
0.02
0.04
0.18 -0.20 0.10 -0.04 0.04
0.11
0.19
0.00
0.03
0.12
0.09
0.16
1.00
0.01
0.00
0.02 -0.03 -0.15 0.67 0.38
0.00
0.25
MCC
Method
CodeLlama-7B Prompt
0.00
-0.25 -0.50 -0.75
0.00 -0.08 0.10
Language
C C/C # ++ Go Jav Java aS crip t Ko tlin PH Py P tho n Ru by Ru st Typ Swif eS t c un ript kn ow n
Language
C C/C # ++ Go Jav Java aS crip t Ko tlin PH Py P tho n Ru by Ru st Typ Swif eS t c un ript kn ow n
Language
C C/C # ++ Go Jav Java aS crip t Ko tlin PH Py P tho n Ru by Ru st Typ Swif eS t c un ript kn ow n
C C/C # ++ Go Jav Java aS crip t Ko tlin PH Py P tho n Ru by Ru st Typ Swif eS t c un ript kn ow n
-1.00
Language
Fig. 2. Fine-grained Language-wise Generalization on the zero-day PreciseBugs benchmark. The heatmaps visualize MCC scores across 13 programming languages for different methods and backbones. Red (warm) indicates positive correlation, while blue (cold) indicates negative correlation.
Robustness Across Different Languages. Fig. 2 exposes a second consistent limitations in baseline models. Traditional methods and even frontier models perform poorly on less common programming languages. A notable example is Gemini 3 Pro, which suffers a performance drop to -0.50 MCC on Kotlin. This suggests that these models rely heavily on the surface-level syntax of the languages they were trained on. They fail to identify defects when the syntax changes. In contrast, SAGE shows positive transfer across the 13-language PreciseBugs split. However, languages with very small test sets (e.g., Kotlin, Swift, and Rust) should be interpreted as qualitative case studies rather than stable performance estimates. By using sparse projection, SAGE effectively separates the logic of the defect from the syntactic style of the code. This capability allows a 7B model to detect defects in languages where trillion-parameter generalist models fail due to unfamiliar syntax. x Finding 2 (Generalization to Long-tail Syntax): Strong performance on mainstream languages may not generalize to diverse syntax patterns. Most methods struggle to maintain accuracy on long-tail languages like Kotlin. SAGE overcomes this limitation. It achieves consistent detection rates across all tested languages. This proves that the method successfully learns universal defect features instead of memorizing specific language syntax.
5.2
RQ2: Mechanistic Verification
Analysis of Signal Loss. To rigorously quantify the phenomenon of signal loss, we employ a dual-metric approach that analyzes the evolution of vulnerability features. This method allows us to verify two complementary aspects: the linear separability of the signal and its relative dominance against the background semantics. First, we evaluate discriminability using the Projected Signal-toNoise Ratio (SNR). Second, we measure the dominance of the signal using the Magnitude Ratio (𝜌𝑙 ), which compares the strength of vulnerability signals (𝑣𝑙 ) against global semantics (𝑠𝑙 ). Guided by the Linear Representation Hypothesis [40], we first define the class centroids at layer 𝑙 as (𝑙 ) (𝑙 ) (𝑙 ) (𝑙 ) 𝜇vul = Mean(𝐻 vuln ) and 𝜇safe = Mean(𝐻 safe ), and use their difference to construct the vulnerability (𝑙 ) (𝑙 ) direction 𝑣𝑙 = 𝜇vul − 𝜇 safe . We then normalize this direction as 𝑣ˆ𝑙 = 𝑣𝑙 /∥𝑣𝑙 ∥ 2 , and project each ′ ′ ′ , 𝜇′ activation ℎ𝑙 (𝑥) onto it via 𝑝𝑙 (𝑥) = ℎ𝑙 (𝑥) ⊤𝑣ˆ𝑙 . Accordingly, 𝜇vul and Varvul , Varsafe denote the safe , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
15
class-wise means and population variances of these scalar projections over vulnerable and safe samples, respectively. We formulate these metrics at layer 𝑙 as follows: SNR𝑙 =
′ − 𝜇′ )2 (𝜇 vul safe ′ ′ Varvul + Varsafe +𝜖
,
𝜌𝑙 =
∥𝑣𝑙 ∥ 2 . ∥𝑠𝑙 ∥ 2
(12)
Fig. 3. Layer-wise evolution of SNR and Feature Magnitude Ratio. The signal follows an "Inverted-U" trajectory, decaying in deep layers. SAGE (stars) successfully recovers the peak signal.
The Reality of Signal Submersion. Fig. 3 visualizes these metrics across four model architectures. The SNR results reveal an inverted-U trajectory where vulnerability signals peak in intermediate layers before collapsing in the deeper layers. This trend implies that standard models perform inference using weakened signals in the final layer. The Magnitude Ratio provides the explanation for this degeneration. As shown in the bottom of Fig. 3, the vulnerability activations constitute less than 5% of the total energy (𝜌𝑙 < 0.05). This low ratio indicates that the subtle vulnerability features are substantially dominated by the dominant global semantics. SAGE effectively addresses this imbalance. On the PreciseBugs benchmark using CodeLlama-7B, SAGE amplifies the SNR by 12.7× compared to the baseline peak. This significant increase validates that our method successfully recovers submerged signals by boosting their relative magnitude. Analysis of Representational Sparsity. We further investigate the efficiency of the learned sparse representation by restricting inference to the top-𝐾 vulnerability-relevant neurons. In Fig. 4, the top-𝐾 selection is performed globally at the dataset level rather than independently for each sample. Specifically, we first identify neurons that are clearly activated on vulnerable samples but not on safe samples, rank them by their average activation magnitude on vulnerable samples, and retain only the top 𝐾 such features during inference. As illustrated in Fig. 4, the performance exhibits a rapid increase followed by a stable plateau. The model recovers 95% of its peak detection capability using only about 220 active neurons. This small subset constitutes less than 1% of the total feature space. This observation indicates that vulnerability-relevant information is highly concentrated in a small number of sparse features. It further suggests that SAGE learns a precise and efficient representation, rather than relying on a large number of diffuse parameters to memorize the data. , Vol. 1, No. 1, Article . Publication date: April 2018.
16
Trovato et al.
Fig. 4. Top-𝐾 Feature Attribution. Performance saturates with fewer than 250 atomic features, confirming the compactness of the extracted vulnerability basis. x Finding 3 (Mechanism of Improvement): SAGE mitigates signal loss by refining the feature representation. Standard models often suppress vulnerability features to less than 5% of the total strength in deep layers. The sparse structure of SAGE corrects this issue. It amplifies the SNR by 12.7× and concentrates the critical information. Consequently, the complex decision logic is reduced to a concise set of approximately 250 essential neurons.
5.3
RQ3: Component Efficacy & Stability
To determine whether SAGE’s performance stems from its architectural design rather than incidental tuning, we conducted a rigorous ablation study and hyperparameter sensitivity analysis. Analysis of Components and Architecture. The results in Table 5 clarify the contribution of each specific module. Although the Linear Probe outperforms standard finetuning, its stagnation at a low Precision of approximately 25% indicates that vulnerability signals remain deeply mixed within the complex feature space. This limitation confirms that simple linear methods are insufficient to separate these entangled signals. Standard aggregation strategies similarly exhibit high instability across different datasets. Mean Pooling tends to dilute the specific vulnerability signals by averaging the features, while Max Pooling often focuses incorrectly on strong noise signals rather than subtle defects. This failure is particularly evident on the PreciseBugs dataset where Max Pooling yields an MCC near 0.00 because it prioritizes high-energy noise over the actual vulnerability patterns. In contrast, the unsupervised SAE functions as an effective noise filter and substantially improves Precision to 74.62% on the BigVul dataset. However, this method lacks specific semantic guidance and consequently suffers from a drop in Recall to 55.45%. Full SAGE resolves this trade-off by using classification gradients to refine the feature representation. This approach effectively restores Recall to 77.82% while maintaining a state-of-the-art Precision of 82.02%. Furthermore, SAGE achieves consistent gains across CodeLlama, Mistral, Llama-3.1, and Qwen2.5, demonstrating that the method learns universally transferable features rather than relying on specific model architectures. Hyperparameter Sensitivity & Structural Invariance. We further examine the robustness of the model by adjusting key hyperparameters as shown in Fig. 5a. The results indicate that the performance remains highly stable across different settings. The Sparse Loss (𝜆) reaches its optimal performance at 0.5, which confirms that a noise filter is necessary to reduce interference from unrelated features. Similarly, the Classification Weight (𝛾) peaks at 1.0, representing an ideal balance between accurately reconstructing the data and identifying defects. Additionally, increasing the Expansion Factor up to 16× leads to consistent improvements, validating that projecting features into a larger space is essential for separating the complex logic of vulnerabilities. Most importantly, Fig. 5b reveals that the signal trajectory consistently follows an inverted-U pattern regardless of the hyperparameter settings. The detection capability always peaks at the intermediate layers (Layers 4–8) before declining rapidly in the deeper layers. This empirical evidence supports the observation that vulnerability signals fade as the model processes information. While , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
17
Table 5. Ablation study of SAGE components. We compare Full SAGE with four variants: Vanilla Finetuning, Linear Probe, Unsupervised SAE, and Pooling strategies. Variant
BigVul [17]
Backbone
PrimeVul [15]
PreciseBugs [22]
Prec. ↑ Recall ↑ F1 ↑ B.Acc ↑ MCC ↑ Prec. ↑ Recall ↑ F1 ↑ B.Acc ↑ MCC ↑ Prec. ↑ Recall ↑ F1 ↑ B.Acc ↑ MCC ↑ Vanilla Finetuning
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
10.87 12.60 13.30 10.74
90.26 72.50 93.60 75.29
19.40 21.47 23.29 18.80
72.02 70.54 77.75 68.11
0.2072 0.2043 0.2651 0.1731
4.50 4.11 4.66 5.06
84.82 88.70 78.58 79.09
8.54 7.85 8.80 9.51
69.54 68.07 68.88 70.69
0.1218 0.1126 0.1190 0.1321
12.50 20.00 14.29 7.69
1.03 1.06 1.03 1.03
1.90 2.01 1.92 1.82
49.76 50.02 49.87 49.22
-0.0153 0.0013 -0.0089 -0.0392
Linear Probe
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
25.28 24.73 28.62 29.71
76.87 68.63 75.92 72.61
38.05 36.35 41.57 42.17
81.71 78.12 82.35 81.22
0.3878 0.3569 0.4183 0.4176
6.08 5.97 6.01 6.16
69.76 76.14 76.87 73.95
11.18 72.67 11.08 74.50 11.16 74.83 11.38 74.22
0.1532 0.1602 0.1622 0.1614
15.38 15.56 12.50 14.29
2.13 3.19 2.27 6.38
3.74 5.29 3.85 8.82
49.88 49.50 49.60 49.33
-0.0058 -0.0202 -0.0174 -0.0190
Unsupervised (w/o L𝑐𝑙𝑠 )
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
74.62 73.18 70.04 80.31
55.45 51.47 54.50 54.88
63.62 60.43 61.30 65.20
77.17 75.18 76.56 77.04
0.6257 0.5953 0.5983 0.6481
14.31 14.17 14.14 15.26
29.33 27.50 29.87 28.05
19.24 18.70 19.19 19.77
62.67 61.86 62.88 62.26
0.1791 0.1721 0.1795 0.1826
20.94 21.27 21.99 21.74
98.10 94.47 86.18 82.56
34.51 34.72 35.04 34.41
50.23 51.15 52.81 52.11
0.0707 0.0628 0.0697 0.0597
Pooling (Mean)
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
59.43 57.27 58.71 62.06
72.61 64.93 62.94 70.24
65.36 60.86 60.75 65.90
84.83 81.03 80.16 83.85
0.6344 0.5852 0.5838 0.6389
13.48 15.13 10.97 12.88
46.63 26.05 47.91 32.79
20.92 19.14 17.85 18.50
69.93 61.37 69.55 63.88
0.2205 0.1747 0.1947 0.1771
18.37 17.07 17.25 17.22
38.30 97.87 98.94 60.64
24.83 29.07 29.38 26.82
51.98 50.97 51.61 50.92
0.0310 0.0382 0.0634 0.0140
Pooling (Max)
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
33.21 63.11 53.68 58.01
43.98 50.43 52.51 42.56
37.85 56.06 53.09 49.10
69.37 74.34 74.91 70.37
0.3399 0.5414 0.5034 0.4719
15.50 14.70 13.84 15.22
23.31 28.78 28.60 22.40
18.62 19.46 18.66 18.13
60.22 62.50 62.28 59.79
0.1676 0.1806 0.1730 0.1623
0.00 0.00 0.00 50.00 16.85 98.94 28.79 50.22 16.79 100.00 28.75 50.00 19.79 81.91 31.88 57.48
0.0000 0.0139 0.0000 0.0000
Full SAGE
CodeLlama-7B Mistral-7B Llama-3.1-8B Qwen2.5-Coder-7B
82.02 77.22 84.70 86.97
77.82 70.05 68.72 69.57
79.86 73.46 75.88 77.30
88.40 0.7874 17.87 84.41 0.7206 17.70 83.99 0.7506 17.78 84.48 0.7664 19.16
32.42 31.33 32.06 31.69
23.04 22.62 22.87 23.88
64.52 64.01 64.35 64.33
0.2375 0.2125 0.2157 0.2245
23.49 22.55 24.91 22.90
0.1050 0.1104 0.1178 0.0986
67.70 84.63 69.60 73.06
34.88 35.61 36.69 34.87
54.79 54.01 57.15 54.12
Note: Linear Probe results are derived from L2-regularized logistic regression on frozen representations. The best results across all methods are highlighted in bold.
Lambda ( ) - Sparse Loss Weight
MCC
0.790 Impact: 2.97
Gamma ( ) - Classification Loss Weight
0.790
0.7842
Impact: 1.50
0.790 Impact: 0.68
Expansion Factor
0.790 Impact: 0.59
0.780
0.780
0.770
0.770
0.760
0.760
0.760
0.760
0.750
0.750
0.750
0.750
0.740
0.740
0.740
0.740
0.730
0.730
0.730
0.730
0.720
0.720
0.720
0.0
0.05
0.1
0.2
0.5
Lambda ( )
1.0
0.780
0.7702
0.5
1.0
0.7683
0.770
2.0
3.0
Mean MCC
±1 Std Dev
Gamma ( )
L0 Coefficient
0.780
4
8
16
Expansion Factor
Max MCC
0.7670
0.770
0.720
0.5
1.0
5.0
10.0
L0 Coefficient
Min MCC
(a) Impact of key hyperparameters (𝜆, 𝛾, 𝐿0 coeff, and Expansion Factor) on model performance at the optimal intervention depth. Lambda ( )
0.800
=0.0 =0.2 =0.5 =1.0
0.775
Best MCC
0.750
Gamma ( )
0.800
=0.5 =1.0 =2.0 =3.0
0.775 0.750
Expansion Factor
0.800
exp=4 exp=8 exp=16
0.775 0.750
0.750
0.725
0.725
0.725
0.700
0.700
0.700
0.700
0.675
0.675
0.675
0.675
0.650
0.650
0.650
0.650
0.625
0.625
0.625
0.625
0.600
0.600
0.600
5
10
15
20
Layer Index
25
30
0
5
10
15
20
Layer Index
25
30
L0=0.5 L0=1.0 L0=5.0 L0=10.0
0.775
0.725
0
L0 Coefficient
0.800
0.600 0
5
10
15
20
Layer Index
25
30
0
5
10
15
20
Layer Index
25
30
(b) Layer-wise performance trajectory across different hyperparameter settings. The consistent "Inverted-U" pattern validates the structural nature of Signal Submersion. Fig. 5. Hyperparameter Sensitivity and Layer Dynamics Analysis. All experiments were conducted using the CodeLlama-7B backbone on the BigVul dataset.
the deeper layers of large language models focus on generating text and integrating context, the specific signals required for detecting defects are strongest in the middle layers. Consequently, SAGE effectively extracts these critical signals before they are overwhelmed by the general objectives of the model. , Vol. 1, No. 1, Article . Publication date: April 2018.
18
Trovato et al.
x Finding 4 (Synergy and Stability): Sparsity contributes significantly to achieving high Precision, while task-specific alignment ensures high Recall. SAGE combines these elements to refine the signal more effectively than standard dense methods. Moreover, the consistent inverted-U trajectory across various settings proves that intervening at the intermediate layers is structurally necessary for accurate detection.
5.4
RQ4: Impact of Model Scale
To determine if Signal Submersion stems from limited capacity or architectural bottlenecks, we evaluated CodeLlama (7B-34B) and Qwen2.5-Coder (7B-32B) against SAGE. We designate standard Supervised Fine-Tuning (SFT) models as the Baseline to isolate the impact of mechanistic intervention.
0.6
MCC Score
Qwen2.5-Coder: Performance
Baseline 7B Baseline 13B Baseline 34B SAGE
CodeLlama: SNR Scaling
Baseline 7B Baseline 14B Baseline 32B SAGE
0.8
0.6
Qwen2.5-Coder: SNR Scaling
Baseline 7B Baseline 13B Baseline 34B SAGE
7.1×
4
3
2.5 2.0
0.4
0.4
1.8×
2.3×
2
1.5
0.2
0.2
Baseline 7B Baseline 14B Baseline 32B SAGE
4.4×
3.0
1.0
11.1×
1 10.4×
0.0
0.0
BigVul
PrimeVul
PreciseBugs
BigVul
PrimeVul
PreciseBugs
0
BigVul
PrimeVul
SNR Peak
CodeLlama: Performance 0.8
PreciseBugs
0.5 0.0
BigVul
PrimeVul
PreciseBugs
Fig. 6. Scalability and Signal Quality Analysis. Standard scaling (blue) exhibits diminishing marginal returns in both performance and signal-to-noise ratio, whereas the SAGE algorithm (red) consistently amplifies signals above the noise baseline. Notably, despite a fivefold increase in parameters, the baseline still shows an asymptotic plateau.
The Saturation of Scaling Laws. Our results examine the assumption that simply increasing model size solves the problem of missed detections. As shown in Table 4, specialized smaller models often outperform massive general-purpose models like GPT-5.2 [2], Gemini 3 [4], and Claude 4.5 [3]. For instance, SAGE-7B achieves a higher score than Gemini 3 Pro on the BigVul dataset, whereas GPT-5.2 even shows a negative correlation on PreciseBugs. This performance gap suggests that larger models tend to prioritize general fluency and coherence, which causes them to overlook the subtle details associated with vulnerabilities. Fig. 6 further confirms this limitation by revealing a performance plateau as models scale from 7B to 34B parameters. This stagnation indicates that adding more parameters does not necessarily improve the ability of a model to recognize vulnerability features. Consequently, while larger models may store more global patterns, they do not inherently possess the sharp perception required to identify fine-grained security anomalies. Table 6. Efficiency Analysis. SAGE achieves superior efficiency, reducing training time by up to 26× compared to RL methods, despite a higher parameter count due to its frozen backbone architecture.
Method & Backbone
LineVul (7B)
VulLLM (7B)
MSIVD (7B)
ReVD (7B)
LineVul (13B)
LineVul (34B)
SAGE (7B)
SAGE (13B)
SAGE (34B)
Params Time (h)
16.8M 13.6
167.7M 34.8
240.0M 40.5
167.7M 69.6
26.2M 25.6
67.1M 62.4
537.0M 839.0M 2.6 5.3
2.15B 15.8
The Cost-Efficiency Paradox & Frozen Backbone Advantage. Standard scaling incurs quadratic computational costs for negligible gains (Table 6). In contrast, SAGE-7B outperforms the Baseline-34B while reducing training latency by 80.8% compared to LineVul-7B and 26.7× compared to ReVD. Although SAGE possesses more trainable parameters (due to the wide Sparse Autoencoder), it operates on a Frozen Backbone regime. By avoiding expensive backpropagation , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
19
through the deep transformer layers, SAGE effectively decouples the cost of detection from the cost of reasoning, enabling a standard RTX 3090 to harness the power of 34B-scale representations. Mechanistic Explanation for Frontier Model Failures. These findings offer a mechanistic explanation for why massive proprietary models, such as GPT-5.2 and Gemini 3, underperformed in our RQ2 analysis. The observed performance plateau indicates that simply scaling up parameters yields diminishing returns in distinguishing vulnerability signals. This suggests that the issue is not incidental but stems from an architectural limitation: as models grow, their enhanced focus on general functional semantics tends to overshadow the sparse indicators of security vulnerabilities. Consequently, even trillion-parameter models struggle to prioritize these subtle artifacts over global code patterns. Addressing this requires a shift in how models represent data—such as employing sparse projection to isolate relevant features—rather than continuing to rely on increased model size. x Finding 6: The Limitations of Scale. We observe a notable gap between model capacity and vulnerability sensitivity. Standard approaches that simply increase model size result in high computational costs with almost no improvement in detection signal quality. SAGE overcomes this limitation by demonstrating that optimizing the model structure is far more effective than relying on parameter scale. Notably, a 7B model equipped with SAGE outperforms a standard 34B baseline while requiring only a fraction of the computational resources.
6
Implications
The findings in this work extend beyond the performance improvements of SAGE. They offer important lessons for applying Large Language Models in software engineering. We identified that the main problem is the suppression of weak signals within the model architecture. Therefore, we examine the common assumption that simply increasing the size of a model or using standard post-training methods like reinforcement learning is the best way to improve detection. Instead, we propose a design approach that focuses on optimizing the internal processing mechanisms of the model. Implications for Researchers. First, we identify the limits of model scaling and standard training. Many researchers believe that larger models automatically reason better. However, our results show that scaling has a limit. While larger models understand global context, the main features of the code still hide the small signals of a vulnerability. Standard fine-tuning also fails because it only optimizes the final output where the signal is already weak. Researchers should focus on architectures that protect these subtle signals in the intermediate layers. Second, this work connects model interpretability to performance. Usually, researchers only use interpretability to analyze a model after training. SAGE shows that interpretability tools can actively improve performance during detection. We should stop treating models as a black box systems. Instead, we should use strategies that help the model separate its internal features rather than only checking the final answer. Third, we show that sparsity is essential for reliability. Standard representations often mix vulnerability logic with irrelevant coding styles. This makes models fail when the style changes. Our results show that simplifying internal representations is necessary for generalization. This allows the model to learn the actual nature of a bug instead of just memorizing the training data. Implications for Practitioners. For industry practitioners, our findings highlight a practical opportunity for cost efficiency. We demonstrate that a standard 7B model can outperform larger 34B generalist models by simply attaching a lightweight, external module while keeping the original model’s parameters frozen. Since this approach involves very few trainable parameters and does not alter the base model’s architecture, it suggests that effective security analysis may not require , Vol. 1, No. 1, Article . Publication date: April 2018.
20
Trovato et al.
massive proprietary models or expensive full-model training. Instead, using a small, specialized addon with a standard model offers a more practical solution for scalable and real-time vulnerability scanning in environments with limited resources. Furthermore, the study emphasizes the need for stability when facing diverse code. In real-world scenarios, codebases vary greatly and frequently undergo changes or restructuring. Traditional fine-tuning often produces models that perform poorly when facing minor syntactic variations. The ability of SAGE to focus on the consistent core features of a vulnerability provides a guide for building resilient security tools. Practitioners should prioritize models that demonstrate the ability to ignore irrelevant changes. This approach reduces the risk of missing detections when facing new vulnerability patterns or complex code structures. 7
Conclusion and Future Work
In this work, we identify Signal Submersion as a significant architectural limitation that limits the ability of Large Language Models to fine-grained security defects, necessitating a shift from passive feature extraction to active signal amplification. By projecting entangled representations onto a high-dimensional sparse manifold, our proposed SAGE framework successfully recovers low-magnitude vulnerability signals, achieving state-of-the-art performance and better robustness against distribution shifts while challenging the universality of parameter scaling for anomaly detection tasks. Looking forward, we plan to extend this method beyond simple detection. Future work will focus on translating these sparse features into natural language to explain the root causes of vulnerabilities. We also aim to use these refined signals to guide models in automatically fixing code. Additionally, we will develop adaptive methods to capture defects at different levels of complexity. Ultimately, SAGE demonstrates that targeted intervention in the internal mechanisms of LLMs offers a more efficient and rigorous path to software security than simply increasing model size. Data Availability We have open-sourced the code and dataset at: https://github.com/BDS-SDU/SAGE. Acknowledgement We thank the anonymous reviewers for their time and helpful comments. This study is supported by the National Natural Science Foundation of China (No. U25A20425, 62232010, 62302266, U23A20302, U24A20244, U24B20149), the Research Project of Quancheng Laboratory, China under Grant No. QCL20250106, the Natural Science Foundation of Shandong Province (Grant No. ZR2024QF093), and the Young Talent of Lifting Engineering for Science and Technology in Shandong, China (Grant No. SDAST2025QTB031).. References [1] 2023. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. https://www.anthropic. com/news/towards-monosemanticity-decomposing-language-models-with-dictionary-learning [Online; accessed 2025-12-08]. [2] 2025. ChatGPT-5.2-Codex. https://openai.com/index/introducing-gpt-5-2-codex/ [Online; accessed 2025-12-23]. [3] 2025. Claude-Opus-4.5. https://www.anthropic.com/news/claude-opus-4-5 [Online; accessed 2025-12-23]. [4] 2025. Gemini-3-pro. https://deepmind.google/models/gemini/pro/ [Online; accessed 2025-12-23]. [5] 2025. NIST. https://nvd.nist.gov/general/nvd-dashboard [Online; accessed 2025-12-08]. [6] Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A few billion lines of code later: using static analysis to find bugs in the real world. Commun. ACM 53, 2 (Feb. 2010), 66–75. doi:10.1145/1646353.1646374 , Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
21
[7] Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M. Buhmann. 2010. The Balanced Accuracy and Its Posterior Distribution. In 2010 20th International Conference on Pattern Recognition. 3121–3124. doi:10.1109/ICPR.2010.764 [8] Partha Chakraborty, Krishna Kanth Arumugam, Mahmoud Alfadel, Meiyappan Nagappan, and Shane McIntosh. 2024. Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets. IEEE Trans. Software Eng. 50, 8 (2024), 2163–2177. doi:10.1109/TSE.2024.3423712 [9] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Transactions on Software Engineering 48, 9 (2022), 3280–3296. doi:10.1109/TSE.2021. 3087402 [10] Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 1 (02 Jan 2020), 6. doi:10.1186/s12864-019-6413-7 [11] Roland Croft, M. Ali Babar, and M. Mehdi Kholoosi. 2023. Data Quality for Software Vulnerability Datasets. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 121–133. doi:10.1109/ICSE48619.2023.00022 [12] Lei Cui, Jiancong Cui, Zhiyu Hao, Lun Li, Zhenquan Ding, and Yongji Liu. 2022. An empirical study of vulnerability discovery methods over the past ten years. Comput. Secur. 120 (2022), 102817. doi:10.1016/J.COSE.2022.102817 [13] DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. CoRR abs/2501.12948 (2025). arXiv:2501.12948 doi:10.48550/ARXIV.2501.12948 [14] Gelei Deng, Yi Liu, Victor Mayoral Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2023. PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. CoRR abs/2308.06782 (2023). arXiv:2308.06782 doi:10.48550/ARXIV.2308.06782 [15] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David A. Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability Detection with Code Language Models: How Far are We?. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1729–1741. doi:10.1109/ICSE55347.2025.00038 [16] Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. 2024. GeneralizationEnhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 10507–10521. doi:10.18653/v1/2024.findings-acl.625 [17] Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories (Seoul, Republic of Korea) (MSR ’20). Association for Computing Machinery, New York, NY, USA, 508–512. doi:10.1145/3379597.3387501 [18] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536–1547. doi:10.18653/V1/2020.FINDINGS-EMNLP.139 [19] Michael Fu and Chakkrit Tantithamthavorn. 2022. LineVul: a transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 608–620. doi:10.1145/3524842.3528452 [20] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 7212–7225. doi:10.18653/V1/2022.ACL-LONG.499 [21] Jingxuan He and Martin Vechev. 2023. Large Language Models for Code: Security Hardening and Adversarial Testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23). ACM, 1865–1879. doi:10.1145/3576915.3623175 [22] Y. He, Z. Chen, and C. Le Goues. 2023. PreciseBugCollector: Extensible, Executable and Precise Bug-Fix Collection: Solution for Challenge 8: Automating Precise Data Collection for Code Snippets with Bugs, Fixes, Locations, and Types. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, Los Alamitos, CA, USA, 1899–1910. doi:10.1109/ASE56229.2023.00163 [23] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. CoRR abs/2409.12186 (2024). arXiv:2409.12186 doi:10.48550/ARXIV.2409.12186 [24] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B.
, Vol. 1, No. 1, Article . Publication date: April 2018.
22
Trovato et al.
CoRR abs/2310.06825 (2023). arXiv:2310.06825 doi:10.48550/ARXIV.2310.06825 [25] Jiayuan Li, Lei Cui, Jie Zhang, Haiqiang Fei, Yu Chen, and Hongsong Zhu. 2025. Steering Large Language Models for Vulnerability Detection. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1–5. doi:10.1109/ICASSP49660.2025.10887736 [26] Yi Li, Shaohua Wang, and Tien N. Nguyen. 2021. Vulnerability detection with fine-grained interpretations. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’21). ACM, 292–303. doi:10.1145/3468264.3468597 [27] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2022. SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities. IEEE Transactions on Dependable and Secure Computing 19, 4 (2022), 2244–2258. doi:10.1109/TDSC.2021.3051525 [28] Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In Proceedings 2018 Network and Distributed System Security Symposium (NDSS 2018). Internet Society. doi:10.14722/ndss.2018.23158 [29] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. CoRR abs/2408.05147 (2024). arXiv:2408.05147 doi:10.48550/ARXIV.2408.05147 [30] Zhongxin Liu, Zhijie Tang, Junwei Zhang, Xin Xia, and Xiaohu Yang. 2024. Pre-training by Predicting Program Dependencies for Vulnerability Analysis Tasks. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 151:1–151:13. doi:10.1145/3597503.3639142 [31] Benjamin Livshits and Monica S. Lam. 2005. Finding Security Vulnerabilities in Java Applications with Static Analysis. In USENIX Security Symposium. https://api.semanticscholar.org/CorpusID:8766314 [32] Benjamin Livshits, Manu Sridharan, Yannis Smaragdakis, Ondřej Lhoták, J. Nelson Amaral, Bor-Yuh Evan Chang, Samuel Z. Guyer, Uday P. Khedker, Anders Møller, and Dimitrios Vardoulakis. 2015. In defense of soundiness: a manifesto. Commun. ACM 58, 2 (Jan. 2015), 44–46. doi:10.1145/2644805 [33] V. Benjamin Livshits and Monica S. Lam. 2005. Finding Security Vulnerabilities in Java Applications with Static Analysis. In Proceedings of the 14th USENIX Security Symposium, Baltimore, MD, USA, July 31 - August 5, 2005, Patrick D. McDaniel (Ed.). USENIX Association. https://www.usenix.org/conference/14th-usenix-security-symposium/findingsecurity-vulnerabilities-java-applications-static [34] Changhua Luo, Wei Meng, and Shuai Wang. 2024. Strengthening Supply Chain Security with Fine-grained Safe Patch Identification. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 89, 12 pages. doi:10.1145/3597503.3639104 [35] Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, and Yang Liu. 2024. Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities. ACM Trans. Softw. Eng. Methodol. 33, 7, Article 169 (Aug. 2024), 29 pages. doi:10.1145/3664606 [36] Valentin J.M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2021. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering 47, 11 (2021), 2312–2331. doi:10.1109/TSE.2019.2946563 [37] Zhenyu Mao, Jialong Li, Dongming Jin, Munan Li, and Kenji Tei. 2024. Multi-Role Consensus Through LLMs Discussions for Vulnerability Detection. In 24th IEEE International Conference on Software Quality, Reliability, and Security, QRS Companion, Cambridge, United Kingdom, July 1-5, 2024. IEEE, 1318–1319. doi:10.1109/QRS-C63300.2024.00173 [38] B.W. Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure 405, 2 (1975), 442–451. doi:10.1016/0005-2795(75)90109-9 [39] Rui Melo, Cláudia Mamede, Andre Catarino, Rui Abreu, and Henrique Lopes Cardoso. 2025. Are Sparse Autoencoders Useful for Java Function Bug Detection? CoRR abs/2505.10375 (2025). arXiv:2505.10375 doi:10.48550/ARXIV.2505.10375 [40] Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The Linear Representation Hypothesis and the Geometry of Large Language Models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview.net/forum?id=UGpGkLzwpP [41] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. CoRR abs/2407.14435 (2024). arXiv:2407.14435 doi:10.48550/ARXIV.2407.14435 [42] Zilong Ren, Xiaolin Ju, Xiang Chen, and Hao Shen. 2024. ProRLearn: boosting prompt tuning-based vulnerability detection by reinforcement learning. Automated Software Engineering 31, 2 (2024), 38. doi:10.1007/s10515-024-00438-9 [43] Niklas Risse and Marcel Böhme. 2024. Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection. In 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024, Davide Balzarotti and Wenyuan Xu (Eds.). USENIX Association. https://www.usenix.org/conference/usenixsecurity24/presentation/risse [44] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron
, Vol. 1, No. 1, Article . Publication date: April 2018.
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
23
Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. CoRR abs/2308.12950 (2023). arXiv:2308.12950 doi:10.48550/ARXIV.2308.12950 [45] Hossain Shahriar and Mohammad Zulkernine. 2012. Mitigating program security vulnerabilities: Approaches and challenges. ACM Comput. Surv. 44, 3, Article 11 (June 2012), 46 pages. doi:10.1145/2187671.2187673 [46] Qingkai Shi, Xiao Xiao, Rongxin Wu, Jinguo Zhou, Gang Fan, and Charles Zhang. 2018. Pinpoint: fast and precise sparse value flow analysis for million lines of code. SIGPLAN Not. 53, 4 (June 2018), 693–706. doi:10.1145/3296979.3192418 [47] Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. Execution-based Code Generation using Deep Reinforcement Learning. Trans. Mach. Learn. Res. 2023 (2023). https://openreview.net/forum?id=0XBuaxqEcG [48] Huizhen Shu, Xuying Li, and Zhuo Li. 2025. LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation. CoRR abs/2509.19839 (2025). arXiv:2509.19839 doi:10.48550/ARXIV.2509.19839 [49] Marco Simoni, Aleksandar Fontana, Giulio Rossolini, and Andrea Saracino. 2025. Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization. CoRR abs/2507.03051 (2025). arXiv:2507.03051 doi:10.48550/ARXIV.2507.03051 [50] Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Earl T. Barr, and Wei Le. 2024. A Comprehensive Study of the Capabilities of Large Language Models for Vulnerability Detection. CoRR abs/2403.17218 (2024). arXiv:2403.17218 doi:10.48550/ARXIV.2403.17218 [51] Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, and Yang Liu. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. CoRR abs/2401.16185 (2024). arXiv:2401.16185 doi:10.48550/ARXIV.2401.16185 [52] Llama Team. 2024. The Llama 3 Herd of Models. CoRR abs/2407.21783 (2024). arXiv:2407.21783 doi:10.48550/ARXIV. 2407.21783 [53] Chung-Nan Tsai, Xin Wang, Cheng-Hsiung Lee, and Ching-Sheng Lin. 2025. A Sequential Multi-Stage Approach for Code Vulnerability Detection via Confidence- and Collaboration-based Decision Making. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 21151–21157. doi:10. 18653/v1/2025.emnlp-main.1071 [54] J. Viega, J.T. Bloch, Y. Kohno, and G. McGraw. 2000. ITS4: a static vulnerability scanner for C and C++ code. In Proceedings 16th Annual Computer Security Applications Conference (ACSAC’00). 257–267. doi:10.1109/ACSAC.2000.898880 [55] Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2022. What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ACM, 2377–2388. doi:10.1145/3510003.3510050 [56] Xin-Cheng Wen, Yijun Yang, Cuiyun Gao, Yang Xiao, and Deheng Ye. 2025. Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Data. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025 (Findings of ACL, Vol. ACL 2025), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, 8935–8949. https://aclanthology.org/2025.findings-acl.467/ [57] Xin-Cheng Wen, Xinchen Wang, Cuiyun Gao, Shaohua Wang, Yang Liu, and Zhaoquan Gu. 2023. When Less is Enough: Positive and Unlabeled Learning Model for Vulnerability Detection. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 345–357. doi:10.1109/ASE56229.2023.00144 [58] Fangzhou Wu, Qingzhao Zhang, Ati Priya Bajaj, Tiffany Bao, Ning Zhang, Ruoyu Wang, and Chaowei Xiao. 2023. Exploring the Limits of ChatGPT in Software Security Applications. CoRR abs/2312.05275 (2023). arXiv:2312.05275 doi:10.48550/ARXIV.2312.05275 [59] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy. 590–604. doi:10.1109/SP.2014.44 [60] Aidan Z. H. Yang, Haoye Tian, He Ye, Ruben Martins, and Claire Le Goues. 2024. Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models. CoRR abs/2406.05892 (2024). arXiv:2406.05892 doi:10.48550/ARXIV.2406.05892 [61] Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. 2024. Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 276–277. doi:10. 1145/3639478.3643065 [62] Jie Zhang, Wei Ma, Qiang Hu, Shangqing Liu, Xiaofei Xie, Yves Le Traon, and Yang Liu. 2023. A Black-Box Attack on Code Models via Representation Nearest Neighbor Search. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 (Findings of ACL, Vol. EMNLP 2023), Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9706–9716. doi:10.18653/V1/2023.FINDINGS-EMNLP.649
, Vol. 1, No. 1, Article . Publication date: April 2018.
24
Trovato et al.
[63] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, 1592–1604. doi:10.1145/3650212.3680384 [64] Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 10197–10207. https://proceedings.neurips.cc/paper/2019/hash/ 49265d2447bc3bbfe9e76306ce40a31f-Abstract.html
, Vol. 1, No. 1, Article . Publication date: April 2018.