Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda

Large Language Models at the Intersection of Software Engineering and Software Security: An Evidence-Centered Structured Survey and Research Agenda

arXiv:2608.21107v1 [cs.AI] 21 Aug 2026

Wei Lin, Tao Zhou, Zhaofei Xie, and Changgui Hong Nanjing Liancheng Intelligent Technology Group, Nanjing, China {linwei, zhoutao, xiezhaofei, hongchanggui}@chinaliancheng.com

Abstract Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and participate in security-sensitive workflows. The evidence for these systems, however, remains divided between software engineering evaluations centered on functional task completion and software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation. This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design. In addition to a task taxonomy, we introduce an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review shows that execution feedback and repository access can substantially improve engineering task completion, but do not by themselves establish security; conversely, static-analysis labels or vulnerability-classification scores rarely establish deployable correctness. We identify recurring validity threats—weak test oracles, duplicated and temporally leaked data, changing agent harnesses, proxy-only security checks, and under-reported budgets and human intervention—and derive a minimum reporting protocol for cross-study comparison. The resulting research agenda prioritizes jointly secure-and-functional benchmarks, repository-scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation. The central conclusion is that model capability should be judged as an assurance case supported by task-appropriate evidence, rather than by a single benchmark score.

Keywords: large language models, software engineering, software security, code generation, vulnerability detection, automated program repair, agentic systems

1

Introduction

The emergence of Large Language Models has substantially changed software development research and practice. Models such as GPT-4, CodeLlama [1], StarCoder [2], and DeepSeek-Coder [3] can understand, generate, and reason about source code. Their applications span code generation, automated program repair (APR), testing, code review, documentation, and vulnerability analysis [4]. Practitioner evidence also suggests productivity benefits for selected development activities, although reported gains depend strongly on task, developer experience, codebase context, and evaluation design [5]. This progress has been accompanied by a proliferation of survey papers, summarized in Table 1. Hou et al. [4] provide a systematic mapping of LLM4SE, including some security-related tasks 1

within the broader software engineering landscape. Guo et al. [6] focus on benchmarks and solutions for LLM-empowered agentic systems, while Khan et al. [7] examine code-review benchmarks and evaluation practices. On the security axis, Xu et al. [8] review LLM applications across cybersecurity, Zhou et al. [9] focus on vulnerability detection and repair, Sheng et al. [10] synthesize vulnerability-detection techniques, and Basic and Giaretta [11] trace evidence from insecure generation to detection and remediation. These reviews establish substantial task-specific knowledge, but they leave open a cross-domain question: what evidence is sufficient when a development artifact must be functional, secure, reproducible, and produced by an agent with non-trivial authority? Our survey adopts an explicitly intersectional perspective. First, it treats software engineering and software security as co-equal analytical pillars rather than presenting security only as a subtask of software engineering. Second, it covers the emergence of agentic systems, including SWE-benchoriented architectures [12], conversational development paradigms [13], and autonomous security workflows [8]. Third, it connects the two domains by examining how code-generation reliability affects security, how security analysis enters development workflows, and which evaluation criteria are required when functionality and security must be satisfied simultaneously. We do not claim exhaustive coverage of either field; instead, the goal is to provide a structured bridge between them. Table 1: Comparison with representative surveys. ◦ = covered as a sub-topic; • = primary focus; – = not emphasized. Survey

Year SE Security Agentic Benchmarks Intersection

Hou et al. [4] 2024 Xu et al. [8] 2025 Zhou et al. [9] 2024 Guo et al. [6] 2025 Gao et al. [14] 2025 Basic and Giaretta [11] 2026 This survey 2026

• – – • • ◦ •

◦ • • – ◦ • •

– ◦ – • – ◦ •

◦ ◦ ◦ • – ◦ •

◦ ◦ ◦ – ◦ ◦ •

This survey makes four contributions: (i) a unified taxonomy that relates engineering and security tasks to artifact granularity, context requirements, adaptation mechanisms, and agent authority; (ii) an assurance-centered framework that distinguishes functional, security, operational, reproducibility, and governance evidence instead of treating task success as a scalar score; (iii) a structured comparison of benchmarks and metrics, including a claim–evidence ladder and a minimum reporting protocol for model and agent evaluations; and (iv) a testable research agenda connecting secure generation, repository-scale analysis, human oversight, maintainability, and standardized cross-task evaluation.

2

Review Methodology

2.1

Scope and Research Questions

This article is an evidence-centered structured narrative and tertiary survey rather than a completed protocol-driven systematic literature review. The search cutoff is May 31, 2026. We prioritize foundational model and benchmark papers, recent systematic reviews, empirical evaluations, and work 2

that directly connects development with software security. The unit of synthesis is not the paper alone, but the claim supported by a particular artifact, dataset, execution environment, metric, and threat model. Because the underlying studies use different model versions, prompts, tools, retry budgets, repository snapshots, and benchmark variants, reported scores are treated as study-specific observations rather than as inputs to a pooled meta-analysis. This scope is deliberately transparent: the review provides a traceable conceptual synthesis and an extensible evidence package, but does not claim exhaustive recall of the literature. The review is guided by four research questions: RQ1: Which software engineering and software security tasks are addressed by LLM-based systems, and what artifacts and context do they require? RQ2: How do prompting, fine-tuning, retrieval, and agentic tool use change the capabilities and failure modes observed across these tasks? RQ3: Which benchmarks and metrics provide credible evidence, and which validity threats prevent direct cross-study comparison? RQ4: Where do engineering and security requirements intersect, and what evaluation and governance mechanisms are needed for integrated agents?

2.2

Search, Eligibility, and Coding

Sources were identified through backward chaining from recent domain surveys and targeted searches for task names, adaptation strategies, models, and benchmark names in arXiv, ACM, and IEEE indexes. The protocol defines four concept blocks: language or code models; software engineering; software security; and adaptation or agent mechanisms. A source was retained when it introduced a consequential model, dataset, benchmark, or method; reported empirical evidence relevant to one of the research questions; synthesized a clearly bounded subfield; or exposed a validity threat with direct implications for evaluation. Opinion-only articles, duplicate versions, sources without enough methodological detail to calibrate their claims, and general cybersecurity work without a software artifact were excluded from the core synthesis. The manuscript’s seed corpus is generated automatically from citation keys in the source file and deduplicated by DOI, arXiv identifier, and normalized title. Coding fields include publication status, task family, artifact granularity, programming language, model or system, adaptation strategy, context source, tools, benchmark, split, metric, principal result, limitations, and the SE– security intersection. A separate quality rubric scores research-question clarity, dataset provenance, split validity, configuration reporting, baseline quality, metric validity, uncertainty analysis, leakage analysis, reproducibility, external validity, and security validation. These artifacts support audit and extension; reviewer judgments and screening agreement are intentionally left for real human coding rather than generated automatically.

2.3

Synthesis and Reproducibility

The analysis combines task-centered narrative synthesis with structured comparison. Findings are not pooled statistically because the studies differ in models, data, prompts, tool access, and evaluation protocols. Instead, evidence is compared at the level of claims that a benchmark can support: function-level correctness, repository-level task completion, vulnerability classification, secure generation, or operational agent behavior. Positive results are interpreted together with negative findings, proxy limitations, and stated threats to validity. The source package includes 3

a prospective review protocol, machine-readable seed corpus, search-log and extraction templates, a quality rubric, and scripts for merging, deduplication, and descriptive statistics. This package separates what has already been synthesized from what a future exhaustive update must still screen.

2.4

Claim Calibration

We calibrate conclusions using four evidence levels. E1 (illustrative) consists of qualitative examples, demonstrations, or small curated tasks and supports feasibility claims only. E2 (controlled benchmark) adds repeatable datasets, baselines, and task-aligned metrics, supporting comparative claims within a fixed configuration. E3 (repository and execution) requires realistic project context, executable tests or exploits, environment capture, and regression checks, supporting system-level claims on the sampled projects. E4 (operational) additionally requires longitudinal use, humaneffort or cost measurement, incident or maintenance outcomes, and governance controls. Evidence at a higher level does not automatically imply better model quality, but it supports a broader deployment claim. Throughout the survey, we therefore distinguish “performs on a benchmark” from “is reliable or secure in practice.” The remainder of this paper is organized as follows. Section 3 provides background on LLM architectures and adaptation strategies for code. Section 5 surveys LLM applications across core software engineering tasks. Section 6 examines LLM-driven approaches for software security. Section 7 compares representative evaluation benchmarks. Section 8 discusses persistent challenges and open problems, and Section 9 concludes with key findings and future directions.

3

Background and Preliminaries

3.1

LLM Architectures for Code

The application of LLMs to code spans three dominant architectural paradigms. Encoder-only models, exemplified by CodeBERT [15], employ bidirectional attention to produce contextual representations suitable for understanding tasks such as vulnerability detection, clone detection, and code search. Encoder-decoder architectures, such as CodeT5 [16], combine bidirectional encoding with autoregressive decoding for tasks including summarization, translation, and repair. Decoderonly models support open-ended generation and tool-mediated interaction. Codex provided an early demonstration [17], while CodeLlama [1], StarCoder [2], and DeepSeek-Coder [3] established widely used open model families. Architectural labels alone do not determine performance: training data, context length, adaptation, and execution feedback are often equally consequential.

3.2

LLM Adaptation Strategies for Code

Four principal adaptation strategies have emerged. Zero-shot and few-shot prompting leverages the model’s pre-trained knowledge without parameter updates, using natural language instructions or input-output examples to guide code generation [17]. Fine-tuning adapts models to specific code domains through full parameter updates or parameter-efficient methods such as LoRA and QLoRA, which are particularly important for domain-specific languages [18] and security-sensitive tasks [10]. Retrieval-Augmented Generation (RAG) enhances generation quality by retrieving relevant code snippets, documentation, or issue reports from repositories, significantly improving repositorylevel code generation [19]. Agent-based approaches decompose complex software tasks into subtasks handled by specialized LLM agents with tool-use capabilities, enabling end-to-end automation of the software development lifecycle [20].

4

3.3

Task Taxonomy

We organize the LLM4SE+Security landscape along three intersecting dimensions: software engineering tasks – code generation/synthesis, automated program repair, software testing, code review, and documentation/maintainability; software security tasks – vulnerability detection, vulnerability repair, malware analysis, and security testing/fuzzing; and adaptation strategies – the four approaches described above. Figure 1 illustrates this taxonomy as a hierarchical tree spanning all three dimensions. Figure 2 situates key developments along a historical timeline from CodeBERT (2020) through the emergence of agentic SE systems (2024–2026). LLM4SE & Security

Automated Repair

Software Engineering

Software Security

Code Gen & Synthesis Testing & Fuzzing

Vuln. Detection

Vuln. Repair

Malware Analysis

Security Testing

Maintainability

Code Review

Adaptation Strategies Fine-tuning

Prompting

RAG

Agent-based

Figure 1: Unified taxonomy of LLM4SE and LLM4Security. The hierarchy separates engineering tasks, security tasks, and adaptation strategies; colors distinguish node roles and adaptation families.

CodeBERT 2020

CodeT5 2021

Codex 2021

GPT-3.5/4 2022-23

Encoder-only

Encoder-Decoder

StarCoder DeepSeekCodeLlama Coder 2024 2023 Decoder-only Agentic

Agentic SE 2024-26

Figure 2: Timeline of key LLM developments for code (2020–2026). Colors denote architectural paradigms.

4

An Assurance-Centered Framework

Existing evaluations often compress a system into one number even though an LLM-based development workflow is a compound intervention. A model receives a context selected by a retriever or agent, acts through a harness, observes tool feedback, and is judged by tests or analyzers executed in a particular environment. We represent an evaluation configuration as E = ⟨A, C, M, T, V, B, H⟩,

(1)

where A is the software artifact and task, C the available context, M the model and adaptation, T the tools and permissions, V the validators, B the inference and retry budget, and H the human intervention. Two scores are directly comparable only when the differences among these components are either controlled or explicitly analyzed. This representation explains why a model-only label is insufficient for an agent result and why benchmark leaderboards can change when the harness or environment changes.

5

4.1

Joint Functional and Security Success

For an output y on task i, let Fi (y) denote satisfaction of functional tests and Si (y) denote satisfaction of threat-model-aligned security checks. The primary intersectional outcome is J@k =

N i 1 X h (k) I ∃y ∈ Yi : Fi (y) ∧ Si (y) , N

(2)

i=1

(k)

where Yi is the set of up to k generated candidates. Reporting F and S separately remains necessary because the same joint score can arise from different failure profiles. The conjunction is nevertheless important: averaging functional and security scores can conceal a catastrophic failure in either dimension. For repair and agent tasks, the conjunction should be extended with regression freedom, environment reproducibility, and policy compliance. This formulation also separates security from mere absence of analyzer warnings. A security predicate may be supported by dynamic proof-of-vulnerability tests, exploit reproduction, expert review, property checking, or a documented combination of analyzers. The validator must match the claimed threat model. Recent repository-level benchmarks move in this direction by pairing developer tests with dynamic security checks or by evaluating proof-of-concept generation and patching in executable environments [21, 22, 23].

4.2

Evidence Ladder

Table 2 connects evidence design to the strongest defensible claim. The levels are cumulative in realism but not interchangeable: a large controlled benchmark may provide more precise comparative evidence than a small operational deployment, while the deployment provides stronger externalvalidity evidence. A mature assurance case should therefore combine complementary levels rather than replace one score with another. Table 2: Claim–evidence ladder for LLM-based software engineering and security systems. Level

Evidence setting

Minimum evidence

Defensible claim

Typical failure

E1

Demonstration or curated examples

Feasibility on illustrated cases

Cherry-picking and no denominator

E2

Controlled benchmark

Comparative performance under the stated benchmark

Leakage, weak tests, and metric mismatch

E3

Repository and executable environment

System effectiveness on sampled projects and environments

Hidden harness effects and irreproducible infrastructure

E4

Operational or longitudinal setting

Prompt, outputs, manual inspection, and explicit examples Frozen data and split, baselines, repeated runs, taskaligned metric, configuration disclosure Versioned repository, build and tests, dynamic or threatmodel-aligned checks, regression evidence, tool and budget logs Real users and workflow, time and cost, maintenance or incident outcomes, access controls, audit trail, and failure escalation

Utility, risk, and sustainability in the observed deployment context

Selection bias, organizational confounding, and rare-event uncertainty

task

6

4.3

Authority and Governance

Agent authority is a separate evaluation dimension. We distinguish four practical tiers: G0 , readonly explanation or retrieval; G1 , patch proposal without execution; G2 , sandboxed editing and tool execution; and G3 , actions that can affect shared repositories, dependencies, credentials, or deployment. Evidence sufficient for G0 is not sufficient for G3 . As authority increases, evaluation should add least-privilege tool scopes, network and secret isolation, action logging, approval gates, rollback, and incident handling. This tiering avoids treating all “agents” as equivalent and makes governance requirements proportional to potential impact.

5

LLMs for Software Engineering

5.1

Code Generation and Synthesis

Code generation represents one of the most extensively studied LLM4SE applications. Early benchmarks such as HumanEval [17] and MBPP [24] evaluate function-level generation in Python. Reported performance has improved substantially, but HumanEval+ [25], which augments the original tasks with many additional tests, demonstrates that solutions passing limited test suites may still fail on edge cases. This distinction between benchmark success and robust functional correctness is central to evaluating generated code. Repository-level code generation introduces significantly greater complexity. SWE-bench [12] tasks systems with resolving real GitHub issues spanning entire codebases. The original study reported a 1.96% resolution rate for its strongest baseline, whereas a recent secondary review reports a result of 78.4% for a later agentic configuration [20]. These endpoints illustrate rapid progress, but they should not be interpreted as a controlled longitudinal experiment: benchmark variants, model versions, scaffolding, tool access, and verification procedures changed over time. Table 3 therefore presents the results as illustrative source-reported observations rather than a unified leaderboard. Table 3: Illustrative SWE-bench results reported by primary and secondary sources. The rows are not directly comparable because evaluation configurations differ. Source/System Setting Resolved (%) Interpretation Jimenez et al. [12]

Original benchmark baseline

1.96

Bhati et al. [20]

Later agentic result reported in review

78.4

Limited repository-level issue resolution Stronger models, retrieval, tools, and iterative execution

Retrieval-Augmented Generation (RAG) is a common mechanism for moving from isolated function generation toward repository-aware synthesis. Tao et al. [19] categorize RAG-for-code approaches by generation strategy, retrieval modality, and integration stage. Multilingual code generation remains uneven: Python dominates evaluation, while less represented languages and language-specific toolchains receive substantially less evidence [26]. Domain-specific generation, including hardware description languages such as Verilog, further illustrates that syntax, verification infrastructure, and domain constraints materially shape evaluation [18].

7

The emergence of “vibe coding” – a paradigm where developers provide high-level intent and iteratively refine LLM outputs through natural language feedback – represents a shift toward speculative, conversational code generation. Ge et al. [13] synthesize this emerging literature and propose a taxonomy spanning unconstrained iteration to context-enhanced development workflows.

5.2

Automated Program Repair

LLM-based APR has become a prominent research direction alongside template-based, search-based, and constraint-based repair. Unlike methods restricted to predefined transformations, generative models can propose diverse patches, but this flexibility also increases the space of plausible yet incorrect repairs [27]. A typical pipeline combines fault localization, context construction, patch generation, and validation through compilation, tests, or additional program analyses. Recent work has explored reinforcement learning and iterative repair, where rewards or feedback encode compilation, test outcomes, and patch properties [28]. Such feedback can improve a particular pipeline, but gains depend on test quality and retry budgets. Key benchmarks include Defects4J [29], BugsInPy [30], and QuixBugs [31]. Persistent risks include regression introduction, test-suite overfitting, and semantically vacuous patches that suppress symptoms without satisfying the intended specification.

5.3

Software Testing and Fuzzing

LLMs have been applied to test generation and to the automation of testing activities. Execution feedback can help models repair imports, assertions, and setup code, but generated tests may reproduce implementation behavior rather than validate the intended specification. TestGenEval evaluates real-world test authoring, completion, and coverage improvement across Python repositories [32]; project-level benchmarks further expose compilation and cascading-context failures that function-level tasks do not capture. Metamorphic testing, a technique for detecting software faults by verifying relations across program executions, has entered a bidirectional relationship with LLMs. Zheng et al. [33] distinguish metamorphic testing applied to LLMs themselves from LLMs applied to generating metamorphic relations and test oracles. Fuzzing has been extended by LLM-guided input generation. Rather than relying solely on mutation-based strategies, LLM-guided fuzzers can generate semantically meaningful inputs for structured formats and protocols. Yu et al. [34] review human-in-the-loop fuzzing and identify interaction modes including monitoring, strategy steering, and collaborative exploration. Wienczkowski et al. [35] describe a “structural-adaptive fragmentation” in which program analysis, DevSecOps, and LLM-based testing remain insufficiently integrated.

5.4

Code Review and Documentation

Code review automation has expanded from static analysis and pattern matching to generative review. Khan et al. [7] document systems that produce comments, estimate change quality, or propose refinements. CodeReviewer provides data and tasks for change-quality estimation, review-comment generation, and code refinement [36]. CRUXEval instead probes input and output prediction for short programs [37]; it is useful for execution reasoning but should not be treated as a direct codereview benchmark. Code summarization and documentation generation are common LLM applications. Models can produce function-level docstrings, file-level overviews, and architectural descriptions, but linguistic fluency is not evidence of factual consistency. Evaluation should therefore check claims against 8

signatures, call sites, tests, and repository evolution, particularly when documentation describes API behavior or security assumptions.

5.5

Software Design and Maintainability

Beyond the coding phase, LLMs are increasingly applied to requirements engineering, architecture design, and design-pattern recommendation. Yu et al. [38] identify a mismatch between academic emphasis on code-level tasks and industrial demand for requirements, architecture, and reliability support. Matias et al. [39] similarly identify potential maintainability benefits alongside risks from hallucinations, inconsistent context, and weak lifecycle evidence. Maintainability claims require different evidence from one-shot correctness. A patch that passes tests today may increase coupling, duplicate logic, weaken an abstraction boundary, or create documentation debt. Useful longitudinal outcomes include subsequent defect density, review and rework effort, code churn, dependency risk, architectural conformance, and the rate at which generated changes are reverted. Current evidence is much stronger for immediate task completion than for these lifecycle outcomes, so productivity and maintainability should not be used as synonyms.

5.6

Repository-Level Agents and Human Work

Repository-level agents combine issue interpretation, search, planning, editing, execution, and revision. Their apparent capability is therefore a property of the whole configuration E, not only the base model. Retrieval quality determines which code and documentation become visible; the harness determines editing and tool behavior; the environment determines which tests can run; and retry limits determine how much search is permitted. Evaluations should include component ablations and report failed trajectories, not only the final successful patch. Human involvement also changes the task. An agent that independently resolves an issue is not directly comparable to a copilot whose user selects files, diagnoses failures, or approves intermediate plans. Human effort should be measured through intervention count, active time, review burden, correction type, and acceptance or rollback decisions. This makes it possible to distinguish automation from productivity support and to identify cases where faster code production merely transfers effort to review, security triage, or maintenance.

6

LLMs for Software Security

6.1

Vulnerability Detection

Detecting security vulnerabilities in source code is one of the most critical and challenging LLM4Security applications. Unlike general bug detection, vulnerability detection requires reasoning about subtle semantic properties – improper input validation, buffer overflows, injection flaws, and access control violations – that may span multiple functions or files [8]. For analysis, we group vulnerability-detection approaches into three overlapping families rather than strict chronological generations. Representation-learning approaches fine-tune encoder models such as CodeBERT [15] for function-level classification on datasets including Devign [40] and Big-Vul [41]. Structure-aware approaches incorporate control-flow, data-flow, or syntax representations. Generative reasoning approaches prompt decoder-only models to produce classifications, explanations, or candidate data-flow traces, sometimes with retrieval and tool support. The families differ in interpretability and context capacity, but all remain sensitive to dataset construction and repository context.

9

Sheng et al. [10] survey LLM-based vulnerability detection techniques, covering model architectures, adaptation strategies, and evaluation protocols. Zhou et al. [9] organize vulnerability detection and repair approaches from direct prompting and fine-tuning to multi-step reasoning. Detection granularity ranges from function-level through slice-level to repository-level analysis. Cross-language and multimodal approaches integrate code with commits, issue reports, and security advisories. Representative resources include Big-Vul and DiverseVul [41, 42], Devign [40], ReVeal [43], and PrimeVul [44]. These datasets differ substantially in labeling quality, class balance, deduplication, and temporal splitting; consequently, scores across them should not be treated as interchangeable. Gao et al. [45] further show that strong benchmark results do not by themselves establish production readiness.

6.2

Vulnerability Repair and Patch Generation

Automated vulnerability repair extends APR to the security domain with heightened correctness requirements: a faulty patch may fail to remove the vulnerability or may introduce a new attack surface. Context-aware repair therefore benefits from information about the vulnerability category, affected paths, and existing mitigations [9]. CVEfixes provides a corpus linking disclosed vulnerabilities to real fixes [46]; it can support repair research but is not itself a complete repair-evaluation protocol. SecurityEval instead evaluates generated code from a software-security perspective and should be classified under secure code generation rather than vulnerability repair [47]. Integration of vulnerability repair with general APR pipelines represents an emerging trend. Rather than treating security bugs as a separate category, unified repair frameworks aim to localize a weakness, generate a patch, preserve intended behavior, and validate exploit resistance within one pipeline. SEC-bench, for example, evaluates proof-of-concept generation and vulnerability patching in reproducible task environments [22]; repository-level secure-generation benchmarks similarly combine project context with functional and dynamic security validation [21, 23]. These designs better align the evidence with the claim, although their project, language, and vulnerability coverage remains bounded.

6.3

Malware Analysis and Threat Intelligence

Beyond source code analysis, LLMs have been applied to binary-level and behavioral security tasks. Malware detection and classification leverage LLMs’ pattern recognition capabilities to identify malicious code patterns, while behavioral analysis uses LLMs to reason about execution traces and system call sequences. Xu et al. [8] document LLM applications in threat intelligence, incident response, and security log analysis, noting the emergence of autonomous security agents that orchestrate multi-step security workflows. Hybrid systems combining traditional program analysis with LLM reasoning represent a particularly promising direction. Static and dynamic analysis provide precise structural information about code and execution, while LLMs contribute semantic understanding and natural language reasoning. This synergy is especially powerful for tasks like vulnerability reachability analysis – determining whether a detected vulnerability is actually exploitable in a given deployment context.

6.4

Security Testing and Exploit Generation

LLM-guided fuzzing augments conventional fuzzers with semantically informed seeds, harness components, or protocol-aware inputs. Wienczkowski et al. [35] organize adaptive security testing around structural analysis, DevSecOps integration, feedback-driven fuzzing, LLM-based test generation, and hybrid systems. They identify insufficient integration among program analysis, devel10

opment pipelines, and LLM-based generation as a central obstacle. Claims of improved coverage must therefore separate the contribution of the LLM from the coverage guidance and execution infrastructure supplied by the underlying fuzzer. Exploit generation – constructing inputs that demonstrate practical exploitability – is inherently dual use. LLM-based security agents may assist with triage, incident analysis, and response orchestration, but evidence for fully autonomous deployment remains limited [8]. False positives can increase analyst workload, while false negatives can miss critical threats. Human approval, audit logs, scoped tool permissions, and reproducible execution evidence are therefore important design requirements for security-agent evaluations.

6.5

Cross-Domain Integration

The intersection of software engineering and security is most visible at four points in the development workflow. First, generated code should be evaluated for both functional correctness and security properties rather than unit-test passage alone. Second, repair systems should verify that a patch removes the targeted weakness without breaking behavior or creating a new vulnerability. Third, repository-level agents should consume feedback from tests, static analyzers, dependency scanners, and policy checks within the same refinement loop. Fourth, agent authority should depend on risk: read-only analysis, patch proposal, test execution, and deployment are materially different capabilities. These integration points motivate evaluations that report secure-and-functional success, evidence provenance, tool permissions, and the amount of human intervention required.

7

Benchmarks and Evaluation

7.1

Code Generation Benchmarks

The code generation landscape is served by a diverse ecosystem of benchmarks, summarized in Table 4. HumanEval [17] (164 hand-written Python problems) and MBPP [24] (∼1,000 crowd-sourced Python problems) remain common function-level evaluations, typically measured via pass@k. HumanEval+ [25] adds substantially more tests to expose solutions that pass weak suites. DS-1000 targets realistic data-science problems across major Python libraries [48], while BigCodeBench evaluates complex instructions and diverse library calls [49]. LiveCodeBench continuously incorporates newly released contest problems and also evaluates self-repair, execution, and output prediction [50]. MultiPL-E translates HumanEval and MBPP to 18 additional programming languages [51]. At repository scale, SWE-bench [12] evaluates issue resolution on real GitHub projects. These benchmarks measure different capabilities and should be selected according to the artifact and deployment claim under study.

11

Table 4: Key benchmarks for LLM4SE and LLM4Security.

Category

Benchmark

Primary Task

Code Generation

HumanEval/HumanEval+ MBPP SWE-bench Verified DS-1000 BigCodeBench/LiveCodeBench

Function-level (Python) Function-level (Python) Repository-level Data science code Multi-library/Continuous

Repair

Defects4J BugsInPy QuixBugs

Real-world Java bugs Real-world Python bugs Multilingual bug fixing

Security Detection

Big-Vul Devign/ReVeal DiverseVul PrimeVul

C/C++ vulnerability data Function-level detection Diverse vulnerability types Deduplicated temporal evaluation

Security Repair Secure Generation Secure Generation Security Agents Security Testing

CVEfixes SecurityEval SecRepoBench/RealSec-bench SEC-bench OSS-Fuzz

Vulnerability–fix corpus Prompt-level security Repository-level joint evaluation Executable PoC and patching tasks Real-world fuzzing targets

Understanding

CodeXGLUE/CodeSearchNet

Multi-task/Search

7.2

Code Repair and Testing Benchmarks

Automated program repair is commonly evaluated on Defects4J [29], BugsInPy [30], and QuixBugs [31]. These resources differ in language, project realism, fault complexity, and test adequacy, so a repair result on short algorithmic programs does not establish repository-scale effectiveness. TestGenEval evaluates test authoring and completion on real Python repositories [32]. For fuzzing research, OSS-Fuzz offers real open-source targets and operational bug histories [52]; nevertheless, coverage or crash counts remain sensitive to time budgets, harness quality, and the underlying fuzzer configuration.

7.3

Security Benchmarks

Security evaluation relies on resources with different purposes. Big-Vul [41], Devign [40], ReVeal [43], PrimeVul [44], and DiverseVul [42] support vulnerability-detection research but differ in construction, labeling, balance, and split strategy. PrimeVul emphasizes deduplication, chronological splitting, and realistic class imbalance, illustrating why scores on older datasets may not transfer to deployment settings. CVEfixes links vulnerabilities to fixing commits and code changes [46]. SecurityEval contains 130 Python prompts covering 75 CWE types and evaluates generated code from a security perspective [47]. Newer resources increasingly test functionality and security together. SecRepoBench supplies repository context, developer-written tests, and dynamic proof-of-vulnerability inputs for secure code completion [21]; RealSec-bench targets secure generation in real Java repositories and explicitly uses a joint secure-and-functional outcome [23]; SEC-bench evaluates agent behavior on executable 12

proof-of-concept and patching tasks [22]. These benchmarks narrow the construct-validity gap, but they should still not be collapsed into a single leaderboard because they evaluate different languages, artifacts, authority levels, and threat models.

7.4

Metric Limitations and Evaluation Gaps

Despite the breadth of available benchmarks, several evaluation gaps persist. First, test-suite adequacy: pass@k estimates the probability that at least one of k generated programs passes the available unit tests [17]; it measures functional correctness only with respect to those tests and does not establish full specification compliance or security. Second, benchmark saturation: strong HumanEval scores reduce its discriminative power for newer systems. Third, data leakage: trainingdata contamination may allow models to reproduce memorized solutions. Fourth, security-specific validity: accuracy, recall, and false-positive rate do not capture triage cost, exploitability, or time to remediation. Static analyzers are valuable validators but imperfect ground truth; human-validated comparison has exposed substantial per-sample disagreement between analyzer reports and actual security labels [53]. Fifth, agentic evaluation: multi-step systems [6] combine retrieval, reasoning, editing, tool use, and execution, making component attribution and reproducibility difficult. Sixth, selection by repeated sampling: best-of-k results can improve while cost, latency, and the number of insecure discarded candidates also increase.

7.5

Minimum Reporting Protocol

To make results interpretable and reproducible, we propose the reporting protocol in Table 5. It applies to both model-centric and agentic studies; fields may be marked not applicable, but should not be silently omitted. The protocol operationalizes the evaluation configuration in Section 4 and allows reviewers to identify whether a reported gain comes from the model, context, harness, tools, budget, validator, or human assistance.

13

Table 5: Minimum reporting protocol for LLM-based software engineering and security evaluations. Dimension

Required disclosure

Reason for inclusion

Task and artifact

Task definition, artifact granularity, language, repository version, and intended deployment claim Exact model and date/version, decoding parameters, prompts, examples, fine-tuning data, and retrieval configuration Agent framework and commit, tools, permissions, network access, sandbox, memory, and stopping rules Samples, retries, tokens, wall time, monetary or compute cost, interventions, and approval decisions Dataset version, deduplication, temporal and project split, overlap checks, and unavailable training-data caveats Functional tests, coverage or mutation evidence, security oracle, threat model, regressions, and manual adjudication Repeated runs, confidence intervals where meaningful, negative results, failure categories, and sensitivity analyses Code, prompts, logs, containers, dependencies, seeds, generated artifacts, and archival identifier

Prevents function-level evidence from being generalized to repositories or operations

Model and adaptation

Harness tools

and

Budget and human input Data and contamination Validation

Uncertainty and failures Reproducibility

Separates model capability from prompt, data, and context effects

Makes agent scores attributable and exposes authority-related risk Prevents hidden search and human labor from being counted as autonomous capability Calibrates memorization and transfer claims

Aligns validators with correctness and security claims Avoids conclusions based only on a favorable run or aggregate score Enables independent verification after services and repositories change

Table 6 summarizes the dominant adaptation strategies and their applicability across the SE and security task landscape, highlighting how different tasks benefit from different LLM utilization paradigms. Table 6: LLM adaptation strategies and their applicability across SE and security tasks. • = primary strategy; ◦ = commonly used; – = rarely applied.

Task Code Generation (function) Code Generation (repo-level) Automated Program Repair Test Generation Code Review Vulnerability Detection Vulnerability Repair Fuzzing

Prompting Fine-tuning RAG Agent-based • ◦ ◦ • • ◦ ◦ •

◦ – • ◦ ◦ • • –

14

◦ • ◦ ◦ ◦ • ◦ –

– • ◦ ◦ – – ◦ •

7.6

Cross-Task Synthesis

Table 7 consolidates the evidence across task families. Three patterns answer the research questions. First, the required context expands from a prompt and function signature to repository state, build systems, issue history, and organizational policy; adaptation correspondingly shifts from prompting and fine-tuning toward retrieval and tool-mediated agents (RQ1–RQ2). Second, evaluation validity depends on the claimed unit of deployment: function tests cannot justify repository-level or security claims, and detector accuracy cannot substitute for exploitability or analyst-cost evidence (RQ3). Third, the SE–security intersection is not a separate terminal task but a set of constraints on generation, repair, testing, and agent authority (RQ4). A high-quality evaluation should therefore report not only whether a task completed, but also which context and tools were available, which functional and security checks passed, what regressions occurred, and how much human intervention was required. Table 7: Cross-task synthesis of artifacts, adaptation strategies, evidence, and principal validity threats. Task family

Artifact and con- Common text tation

adap- Typical evidence

Principal validity threat

Function syn- Specification, sig- Prompting; fine- pass@k and expanded Weak tests, contamination, thesis nature, local tests tuning unit tests and limited API realism Repository is- Repository, issue, Retrieval; planning; Resolved issues and re- Changing benchmark vergression tests sions, retry budgets, and sue resolution build and test tool-using agents hidden infrastructure environment Program re- Fault location, Fine-tuning; iter- Plausible and correct Test-suite overfitting and pair buggy code, tests, ative prompting; patches behavior regressions execution feedback agents Coverage, mutation Oracle weakness and conTest gener- Code under test, Prompting; score, crashes, valid founding from the underlyruntime execution-guided ation and harness, ing fuzzer tests agents feedback fuzzing Review and Diff, repository Prompting; re- Comment relevance Subjective labels and fludocumentation context, conven- trieval; task- and human judgment ent but unsupported explations, APIs specific tuning nations Precision, recall, F1, lo- Duplicates, temporal leakVulnerability Function, slice, Fine-tuning; calization age, unrealistic balance, detection data flow, reposi- structure-aware and label noise models; retrieval tory context Secure genera- Specification, Prompting; fine- Functional tests plus Proxy analyzers, incomtion and repair CWE/CVE con- tuning; constrained security checks plete threat models, and text, tests, analyz- or iterative generanew vulnerabilities ers tion Security agents Multi-tool environ- Retrieval; planning; Task completion, evi- Unsafe actions, irreproment, permissions, tool use; human ap- dence trace, analyst ef- ducible environments, and logs, policies proval fort unbounded authority

8

Challenges and Open Problems

8.1

Reliability, Hallucination, and Calibration

Code hallucinations—outputs that are syntactically plausible but semantically incorrect, insecure, or inconsistent with the repository—remain a fundamental reliability challenge. Gao et al. [14] relate these failures to training data, autoregressive generation, and insufficient context. At repository scale, the failure surface expands to nonexistent symbols, stale APIs, misunderstood build 15

assumptions, incompatible dependency changes, and patches that address the visible symptom while violating an implicit invariant. Execution feedback reduces some errors but creates a risk of validator overfitting. An agent may learn to satisfy the observed tests without implementing the intended behavior, or remove an analyzer warning without eliminating exploitability. Reliability therefore requires diverse evidence: specification-derived tests, regression suites, mutation analysis, static and dynamic checks, differential behavior, and targeted human review. Uncertainty should also be calibrated at the level at which action is taken. A confidence statement attached to a fluent explanation is less useful than calibrated probabilities for patch acceptance, vulnerability presence, or the need for escalation.

8.2

Dataset Realism, Leakage, and Benchmark Governance

Dataset construction can dominate apparent model progress. Duplicate or near-duplicate functions inflate random-split performance; vulnerability-fixing commits contain unrelated edits; negative examples may be sampled from unrealistically clean code; and labels based on a commit message or analyzer can be noisy. PrimeVul demonstrates the importance of deduplication, chronological splitting, and realistic imbalance [44]. Similar discipline is needed for generative tasks because benchmark problems, tests, solutions, and repository histories may all be present in pretraining or retrieval corpora. A benchmark should consequently be governed as a versioned scientific instrument. Its release should identify provenance, licenses, filtering, label adjudication, known contamination channels, test coverage, supported claims, and change history. For agent benchmarks, the repository image, dependency registry state, container, network policy, and evaluator must also be frozen. Periodic refreshes can reduce memorization, but they should not silently replace an existing test set because longitudinal comparisons then lose meaning. A stable core, a sequestered evaluation set, and a documented refresh track provide a more defensible compromise.

8.3

Reproducibility, Cost, and Environmental Drift

Proprietary model updates, nondeterministic sampling, disappearing APIs, mutable repositories, and evolving package registries make exact replication difficult. Agent evaluations add further variation through search indexes, tool versions, timeouts, concurrency, network access, and retry policies. A result should therefore include both a logical specification of the task and an executable environment whenever licensing permits. Containers alone are insufficient if external services, model snapshots, or package artifacts remain mutable. Cost is also part of effectiveness. A system that obtains a higher best-of-k success rate through many samples or long repair loops may be unsuitable for routine use. Relevant quantities include tokens, model calls, wall time, compute or monetary cost, tool executions, failed candidates, and human review. Reporting a Pareto frontier over quality, security, cost, and latency is more informative than ranking systems by task success alone. For security triage, analyst time spent dismissing false positives may dominate model inference cost.

8.4

Adversarial Robustness and Agent Governance

Security systems face distribution shift and active adversaries. Semantics-preserving renaming, control-flow restructuring, dead-code insertion, misleading comments, poisoned retrieval documents, and tool-output manipulation can alter model judgments. Repository-level agents additionally process untrusted files and issue text that may contain prompt injections. Evaluations should

16

distinguish robustness to natural code evolution from robustness to deliberate manipulation and should test both the model and the surrounding tool chain. Governance becomes essential when agents can execute commands or modify shared assets. The authority tiers in Section 4 imply progressively stronger controls: isolation and content provenance for read-only analysis; mandatory diff review for proposed patches; sandboxing and egress restrictions for tool execution; and explicit approval, rollback, and audit requirements for changes to production or security infrastructure. Safety refusal rates alone do not measure these system properties. The relevant question is whether unsafe actions are prevented, detected, contained, and recoverable under the stated threat model.

8.5

The Academic–Industry Evidence Gap

Yu et al. [38] identify mismatches in task emphasis, scale realism, and integration. Academic work is concentrated on code-level tasks with automatically checkable outcomes, whereas industrial adoption must also address requirements, architecture, migration, reliability, compliance, ownership, and coordination. Million-line repositories, private dependencies, legacy build systems, organizationspecific conventions, and regulated data cannot be represented fully by small public benchmarks. The gap is not solved simply by obtaining proprietary data. Industrial studies also require credible counterfactuals: comparison teams or periods, developer-experience stratification, workflow instrumentation, and measurement of downstream review, incidents, and maintenance. Matias et al. [39] emphasize that long-term maintainability remains uncertain. Longitudinal studies should follow generated changes beyond merge time and should report rejected and reverted outputs, not only accepted suggestions.

8.6

Threats to the Validity of This Survey

Selection validity is limited because this is a structured narrative review built from a traceable seed corpus rather than a completed exhaustive search and dual-reviewer screening process. Relevant work may be absent, and highly visible English-language papers and preprints may be overrepresented. The protocol and templates make this limitation auditable but do not remove it. Construct validity is threatened by inconsistent uses of terms such as correctness, agent, autonomy, vulnerability detection, and secure generation; the taxonomy and assurance framework reduce but cannot eliminate these differences. Conclusion validity is limited by heterogeneous tasks and configurations, which is why source-reported scores are not pooled. External validity remains constrained by the dominance of a small set of languages, open-source repositories, curated datasets, and short evaluation windows. Temporal validity is unusually fragile because models, APIs, scaffolds, and leaderboards change rapidly. The cutoff date and benchmark versions should therefore accompany any reuse of the conclusions.

8.7

A Testable Research Agenda

Table 8 translates broad directions into falsifiable questions and minimum evidence. The agenda favors studies that isolate mechanisms, preserve failure evidence, and jointly evaluate engineering value and security risk.

17

Table 8: Research agenda derived from the evidence gaps. Direction

Testable research question

Minimum credible evaluation

Joint secure generation

Does an intervention improve J@k rather than shifting failures between functionality and security?

Repository context and agents

Which gains come from retrieval, the model, the harness, tools, or additional retries?

Verifier ensembles and human feedback

When do tests, analyzers, dynamic exploits, and expert review provide complementary rather than correlated evidence? How does increasing authority change utility, unsafe-action probability, and oversight cost? Do LLM-authored changes reduce total engineering effort without increasing future defects or debt? Do methods transfer across time, languages, ecosystems, and newly disclosed weakness patterns?

Paired functional and threat-model-aligned tests; separate F , S, and joint outcomes; fixed sampling budget; regression and failure analysis Versioned repositories and environments; controlled component ablations; equalized budgets; trajectory and tool logs; multiple projects Human-adjudicated subset; precision, recall, calibration, disagreement analysis, analyst time, and sensitivity to the chosen oracle

Risk-bounded tonomy

au-

Lifecycle maintainability Continuous and multilingual evaluation

9

Evaluation across G0 –G3 tiers; adversarial tasks; least-privilege controls; approval and rollback logs; incident-oriented metrics Longitudinal repository study; matched controls; review and rework time; defects, reversions, churn, and architecture measures Temporal and project-held-out splits; multiple language/toolchain families; benchmark refresh protocol; contamination audit and uncertainty reporting

Conclusion

This survey connected software engineering and software security through a shared taxonomy, an explicit evaluation configuration, and an assurance-centered view of evidence. The synthesis answers the four research questions as follows. For RQ1, the tasks range from function-level generation and classification to repository issue resolution, security patching, fuzzing, and tool-mediated analysis; the decisive difference is often the artifact and context available rather than the task label alone. For RQ2, prompting and fine-tuning remain important for bounded tasks, while retrieval, execution feedback, and agents become more consequential as repository context and iteration increase. These mechanisms change both capability and failure modes, and their effects cannot be attributed to the base model without controlled ablations. For RQ3, credible evidence must match the deployment claim. Unit tests support only testrelative functional claims; vulnerability labels support classification claims; and repository or operational claims require versioned environments, realistic context, regression checks, security oracles, budgets, and human-effort evidence. The proposed configuration E, evidence ladder, and minimum reporting protocol make these requirements explicit. For RQ4, security is not a downstream task appended to development. It is a constraint on generation, repair, testing, review, and agent authority. Joint outcomes such as J@k should therefore accompany separate functional and security results. The literature supports cautious optimism rather than unconditional automation. Repository access and iterative tool use can improve task completion, yet they also introduce harness dependence, larger attack surfaces, higher cost, and reproducibility challenges. Static analyzers, tests, and LLM judges are useful components of an assurance case but are not individually sufficient ground truth. Long-term maintainability, multilingual transfer, adversarial robustness, and the 18

organizational cost of oversight remain under-evidenced. The central research priority is consequently not another isolated leaderboard, but a cumulative evidence infrastructure: versioned benchmarks, executable environments, contamination-aware splits, joint functional and security validation, transparent agent permissions, complete trajectory and budget logs, and longitudinal human-centered outcomes. Progress should be claimed in proportion to the evidence. Under this standard, an LLM-based software system is trustworthy not because it produces a plausible patch or achieves a high scalar score, but because its behavior is supported by a reproducible, threat-model-aligned, and governance-aware assurance case.

References [1] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, et al. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. [2] Raymond Li et al. StarCoder: May the source be with you! Transactions on Machine Learning Research, 2023. arXiv:2305.06161. [3] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, et al. DeepSeekCoder: When the large language model meets programming. arXiv preprint arXiv:2401.14196, 2024. [4] Xinyi Hou et al. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 2024. arXiv:2308.10620. [5] Vincent Gurgul, Robin Gubela, and Stefan Lessmann. The state of generative AI in software development: Insights from literature and a developer survey. arXiv preprint arXiv:2603.16975, 2026. [6] Jiale Guo, Yanjie Zhao, Yue Liu, Li Li, Kailong Wang, and David Lo. A comprehensive survey on benchmarks and solutions in software engineering of LLM-empowered agentic system. arXiv preprint arXiv:2510.09721, 2025. [7] Taufiqul Islam Khan et al. A survey of code review benchmarks and evaluation practices in Pre-LLM and LLM era. arXiv preprint arXiv:2602.13377, 2026. [8] Hanxiang Xu et al. Large language models for cyber security: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 2025. arXiv:2405.04760. [9] Xin Zhou, Ting Zhang, and David Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead. arXiv preprint arXiv:2404.02525, 2024. [10] Ze Sheng, Zhicheng Zhan, Zhenyu Qin, Sichen He, and Jian Huang. LLMs in software security: A survey of vulnerability detection techniques and insights. arXiv preprint arXiv:2502.07049, 2025. [11] Enna Basic and Alberto Giaretta. From vulnerabilities to remediation: A systematic literature review of LLMs in code security. arXiv preprint arXiv:2412.15004, 2026. [12] Carlos Ernesto Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. 19

[13] Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, Baolong Bi, Fangda Guo, Jiafeng Guo, Shenghua Liu, and Xueqi Cheng. A survey of vibe coding with large language models. arXiv preprint arXiv:2510.12399, 2025. [14] Cuiyun Gao, Guodong Fan, Chun Yong Chong, Shizhan Chen, Chao Liu, David Lo, Zibin Zheng, and Qing Liao. A systematic literature review of code hallucinations in LLMs: Characterization, mitigation methods, challenges, and future directions for reliable AI. arXiv preprint arXiv:2511.00776, 2025. [15] Zhangyin Feng et al. CodeBERT: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1536–1547, Online, 2020. Association for Computational Linguistics. [16] Yue Wang et al. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8696–8708, Online, 2021. Association for Computational Linguistics. [17] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [18] Guang Yang, David Lo, et al. Large language model for Verilog code generation: Literature review and the road ahead. arXiv preprint arXiv:2512.00020, 2025. [19] Yicheng Tao et al. Retrieval-augmented code generation: A survey with focus on repositorylevel approaches. arXiv preprint arXiv:2510.04905, 2025. [20] Happy Bhati. Agentic AI in the software development lifecycle: Architecture, empirical evidence, and the reshaping of software engineering. arXiv preprint arXiv:2604.26275, 2026. [21] Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. SecRepoBench: Benchmarking code agents for secure code completion in real-world repositories. arXiv preprint arXiv:2504.21205, 2026. [22] Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. SEC-bench: Automated benchmarking of LLM agents on real-world software security tasks. arXiv preprint arXiv:2506.11791, 2025. [23] Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, and Zibin Zheng. RealSec-bench: A benchmark for evaluating secure code generation in real-world repositories. arXiv preprint arXiv:2601.22706, 2026. [24] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [25] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems (NeurIPS), 36:21558–21572, 2023.

20

[26] Chao Jiang et al. Large language models for multilingual code intelligence: A survey. arXiv preprint arXiv:2604.25960, 2026. [27] Avinash Anand, Akshit Gupta, Nishchay Yadav, and Shaurya Bajaj. A comprehensive survey of AI-driven advancements and techniques in automated program repair and code generation. arXiv preprint arXiv:2411.07586, 2024. [28] Md Rayhanul Masud et al. Reward engineering for reinforcement learning in software tasks. arXiv preprint arXiv:2601.19100, 2026. [29] René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA), pages 437–440, New York, NY, USA, 2014. Association for Computing Machinery. [30] Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, et al. BugsInPy: A database of existing bugs in python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 1556–1560, New York, NY, USA, 2020. Association for Computing Machinery. [31] Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. QuixBugs: A multilingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, pages 55–56, New York, NY, USA, 2017. Association for Computing Machinery. [32] Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. TestGenEval: A real world unit test generation and test completion benchmark. Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. arXiv:2410.00752. [33] Zheng Zheng et al. Bidirectional empowerment of metamorphic testing and large language models: A systematic survey. arXiv preprint arXiv:2605.13898, 2026. [34] Jiongchi Yu et al. Human in the loop for fuzz testing: Literature review and the road ahead. arXiv preprint arXiv:2603.13411, 2026. [35] Michael Wienczkowski et al. Adaptive and AI-augmented security testing: A systematic survey. arXiv preprint arXiv:2604.27000, 2026. [36] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, et al. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 1035–1047, New York, NY, USA, 2022. Association for Computing Machinery. [37] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024. [38] Hang Yu, David Lo, et al. Aligning academia with industry: An empirical study of industrial needs and academic capabilities in AI-driven software engineering. arXiv preprint arXiv:2512.15148, 2025. 21

[39] Bruno Claudino Matias et al. A survey on large language model impact on software evolvability and maintainability. arXiv preprint arXiv:2601.20879, 2026. [40] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in Neural Information Processing Systems (NeurIPS), Vol. 32, 2019. [41] Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. Big-vul: A C/C++ vulnerability dataset with comprehensive code representations. arXiv preprint arXiv:2105.14511, 2021. [42] Yi Ding, Chang Sun, Shaoying Liu, and Hui Gao. DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. arXiv preprint arXiv:2304.00409, 2023. [43] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48(9):3280–3296, 2022. arXiv:2009.07235. [44] Yangruibo Ding et al. Vulnerability detection with code language models: How far are we? In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE), pages 1729–1741, Los Alamitos, CA, USA, 2025. IEEE Computer Society. arXiv:2403.18624. [45] Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420, 2023. [46] Guru Prasad Bhandari, Amara Naseer, and Leon Moonen. CVEfixes: Automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), pages 30–39, New York, NY, USA, 2021. Association for Computing Machinery. arXiv:2107.08760. [47] Mohammed Latif Siddiq and Joanna C. S. Santos. SecurityEval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, pages 29–33, New York, NY, USA, 2022. Association for Computing Machinery. [48] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2211.11501. [49] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, et al. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. arXiv:2406.15877. [50] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. [51] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, et al. MultiPL-E: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022. 22

[52] Zhen Yu Ding and Claire Le Goues. An empirical study of OSS-Fuzz bugs. arXiv preprint arXiv:2103.11518, 2021. [53] Ehsan Firouzi and Mohammad Ghafari. Persistent human feedback, LLMs, and static analyzers for secure code generation and vulnerability detection. arXiv preprint arXiv:2602.05868, 2026.

23

Related documents

• Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda#662279
• Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda#661229
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models#1141431
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#1013799
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500