arXiv:2606.28125v1 [cs.SE] 26 Jun 2026
How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests Pien Rooijendijk
Christoph Treude
Mairieli Wessel
Radboud University The Netherlands
Singapore Management University Singapore
Radboud University The Netherlands
Abstract—Developers may reference vulnerabilities in pull request discussions through both explicit identifiers, such as CVEs or GHSAs, and implicit security-related language (e.g., “unauthorized access” or “SQL injection”). Prior work has primarily focused on explicit identifiers, potentially overlooking vulnerability discussions that lack formal references. Bots and coding agents are becoming more common in pull requests, raising new questions about how different accounts communicate about vulnerabilities. In this registered report, we describe our planned study of vulnerability communication in pull requests by humans, bots, and coding agents. Building on the AIDev-pop dataset, we analyze explicit vulnerability references and implicit security-related signals across pull request titles, descriptions, review comments, commit messages, and timeline discussions. We further investigate whether these references are associated with vulnerabilities introduced or fixed in the modified code and how they relate to pull request review activity and outcomes. This study contributes a large-scale empirical investigation of vulnerability communication practices in modern software development. Index Terms—Software Security, Vulnerability Identifiers, Bots, Coding Agents
I. I NTRODUCTION Software vulnerabilities are routinely identified, cataloged, and communicated through standardized identifier systems such as Common Vulnerabilities and Exposures (CVE), Common Weakness Enumeration (CWE), GitHub Security Advisories (GHSA), and ecosystem-specific databases. These identifiers support coordination across projects, tools, and communities by providing shared references for discussing weaknesses, tracking disclosures, and linking fixes to known issues. Prior work has shown that such identifiers are frequently referenced in software repository artifacts, including commits, issues, and pull requests [1, 2, 3]. However, developers do not communicate about vulnerabilities only through formal identifiers. Security concerns may also be discussed through implicit language, such as “unauthorized access,” “SQL injection,” or “insecure behavior,” without referencing a CVE or GHSA. Prior work suggests that vulnerability discussions often occur before formal disclosure and that communication practices vary across repositories and development contexts [4, 5, 6, 7]. Previous work has primarily used vulnerability identifiers to study security-related development activity, including vulnerability-fixing commits, dependency management, disclosure practices, and vulnerability response times [8, 9, 10].
Other studies explored automated detection of security-related discussions using keyword-based and machine learning approaches [11, 12]. However, these studies focus on identifying security-related artifacts, such as GitHub issues, rather than understanding how vulnerabilities are communicated within pull request discussions. Bots and coding agents are becoming increasingly common in pull request workflows. Recent work showed that coding agents can generate security-related pull requests and that bots frequently reference vulnerability identifiers in automated dependency updates and security patches [13, 14, 15]. While our prior work examined how humans, bots, and coding agents use explicit vulnerability identifiers [13], it did not consider implicit security-related signals or whether such references correspond to actual vulnerabilities in the modified code. As a result, little is known about how different actors communicate about vulnerabilities beyond formal identifiers, whether these references are associated with vulnerabilities being introduced or fixed, and how they influence pull request review and outcomes. In this registered report, we describe our plans for investigating how vulnerabilities are referenced in GitHub pull requests by humans, bots, and coding agents. We build on the AIDev-pop dataset [16] and extend it with additional pull requests collected from the same repositories. We analyze explicit vulnerability references and implicit security-related signals across pull request titles, descriptions, review comments, commit messages, and timeline discussions. Explicit references are identified using regular expression matching for standardized vulnerability identifiers, while implicit signals are detected using a validated keyword-based approach for security-related language [11]. This study contributes a large-scale analysis of explicit vulnerability references and implicit security-related signals in pull requests across humans, bots, and coding agents. In addition, we provide a dataset and analysis pipeline for studying vulnerability communication in pull request discussions. II. BACKGROUND & R ELATED W ORK A. Vulnerability Signals in Software Repositories Software vulnerabilities are frequently discussed in pull requests (PRs) through both explicit vulnerability references, which directly mention standardized identifiers such as CVEs,
(a) Explicit vulnerability references
(b) Implicit security-related signals
Fig. 1: Examples of explicit and implicit references & signals in pull requests.
CWEs, GHSAs, or RUSTSEC entries, and implicit securityrelated signals, which describe security concerns without referencing a formal identifier. Explicit references provide structured and verifiable links to vulnerability databases and advisory systems and have been widely used to study vulnerability fixes and security-related development activity. Figure 1a shows an example of an explicit vulnerability reference in a pull request by Copilot.1 Not all security-related discussions include explicit identifiers. Developers may instead describe vulnerabilities through natural language, for example by referring to insecure behaviour, attack scenarios, or mitigation strategies without mentioning a known CVE, CWE, or GHSA entry [6, 7]. We refer to these as implicit security-related signals. Figure 1b presents an example where a reviewer highlights a potential authorization issue without referencing a formal vulnerability identifier.2 Prior work has investigated how vulnerabilities are referenced and discussed in software repositories. Many studies rely on explicit vulnerability identifiers such as CVEs, CWEs, and GHSAs to trace vulnerability fixes, disclosure practices, dependency management, and other security-related development activities [1, 2, 3]. Recent work also showed that vulnerability discussions frequently occur in repository artifacts such as issues, commits, and pull requests, sometimes before formal disclosure [4, 5]. While most of this work focuses on explicit vulnerability identifiers, other studies have explored how security-related content can be identified from natural-language discussions. Zhou and Sharma [11] proposed a keyword-based approach for identifying security-related discussions from commit messages and bug reports, while Cipollone et al. [12] used Transformer-
based models to identify vulnerability-related GitHub issues. Together, these studies suggest that analyses relying solely on explicit identifiers may overlook security-related discussions that lack formal vulnerability references. B. Security Related Pull Requests Research on security-related pull requests has largely focused on dependency remediation, vulnerability classification, and review outcomes. Studies of dependency management bots found that automated security pull requests are widely used but are not always merged or acted upon by developers [17, 18]. More recently, work on coding agents showed that agents increasingly participate in security-related development activities and generate pull requests involving vulnerability-related changes [14, 15]. Our prior work [13] examined how humans, bots, and coding agents use explicit vulnerability identifiers in pull requests. In contrast, the present study investigates both explicit and implicit vulnerability references, examines their relationship with vulnerabilities detected in the modified code, and analyzes their association with pull request review activity and outcomes. III. R ESEARCH Q UESTIONS We investigate how explicit and implicit vulnerability references are communicated in pull requests by humans, bots, and coding agents through the following research questions: RQ1 How are vulnerability references distributed in pull requests in terms of implicit and explicit references? RQ1a
1 https://github.com/microsoft/azurelinux/pull/13855 2 https://github.com/liam-hq/liam/pull/1598
How frequently do implicit and explicit vulnerability references occur in pull requests?
RQ1b How does the distribution of vulnerability references vary across actor types (humans, bots, agents)? RQ1c
How does the distribution of vulnerability references vary across pull request components?
Vulnerabilities may be discussed both through explicit identifiers and implicit natural-language descriptions. However, little is known about how these references are distributed across pull request artifacts and actor types. Understanding these distributions is important because repository mining studies and security tooling often rely primarily on explicit identifiers, potentially overlooking vulnerabilities that are discussed only implicitly. To answer this question, we identify explicit references using regular expression patterns and implicit references using a validated keyword-based approach applied to pull request artifacts.
GitHub Copilot, Cursor, and Claude Code. The original dataset contains more than 456k agent-authored PRs across over 61k repositories and includes rich repository, review, timeline, and commit-level metadata, with data collected up to August 1, 2025. We use AIDev-pop, which filters the dataset to include repositories with more than 500 GitHub stars, focusing on popular and active open-source projects. As shown in Table I, AIDev-pop contains 33,078 pull requests across 2,807 repositories authored by five coding agents. AIDev-pop includes pull request titles and descriptions, review comments, commit messages, timeline events, and commit level metadata such as patches and file changes. TABLE I: Distribution of pull requests and repos by agent. Agent
RQ2 How are implicit and explicit vulnerability references associated with static-analysis-detectable security changes of vulnerability detection, as reflected by indicators of actual vulnerabilities in code changes?
Recent studies reported that coding agents frequently generate security-related pull requests, but also produce false positives and security-related claims that do not correspond to actual vulnerabilities [19]. This raises the question of whether the way vulnerabilities are referenced in pull requests is associated with the presence of actual security-relevant code changes. We therefore compare vulnerability references against static analysis results from the code before and after the pull request changes. RQ3 How are implicit and explicit vulnerability references associated with the social response to pull requests?
Pull Requests
Repositories
OpenAI Codex Devin GitHub Copilot Cursor Claude Code
21,779 4,827 4,970 1,541 459
1,248 288 1,012 327 213
Total
33,078
2,807
To account for vulnerability references beyond agentauthored PRs, we collect all pull requests created during the same observation period as AIDev-pop across all repositories in the dataset. This extension allows us to compare vulnerability references across humans, bots, and coding agents within the same repository ecosystems. V. E XECUTION P LAN Figure 2 provides an overview of the study design. A. Vulnerability Signals Detection
RQ3a
How are implicit and explicit vulnerability references associated with pull request review activity?
RQ3b How are implicit and explicit vulnerability references associated with pull request outcomes?
The way vulnerabilities are communicated may influence how pull requests are reviewed and evaluated. Explicit identifiers provide direct links to external advisories and vulnerability databases, while implicit references require reviewers to interpret and validate security-related claims from the discussion context. To investigate whether these differences affect pull request evaluation, we analyze review activity, discussion patterns, response times, and pull request outcomes across different vulnerability reference types and actor categories. IV. DATASET Our study builds on the AIDev-pop dataset, a curated subset of the AIDev dataset introduced by Li et al. [16]. AIDev captures pull requests (PRs) authored by autonomous coding agents on GitHub, including OpenAI Codex, Devin,
To identify vulnerability references, we analyze pull request artifacts at a fine-grained level rather than classifying pull requests as a whole. Specifically, we analyze pull request titles, descriptions, review comments, commit messages, and timeline messages individually. Each detected reference is associated with both the artifact in which it appears and the account responsible for authoring that artifact. This enables us to study how vulnerability references are communicated across different interaction contexts and account types. TABLE II: Regex patterns used for explicit vulnerability signal detection. ID CVE CWE GHSA GO RUSTSEC OSV MAL USN
Regex pattern CVE−\d{4} − \d{2, 7} CWE−\d{2, 4} GHSA−[a−z0−9]{4}−[a− z0 − 9]{4} − [a − z0 − 9]{4} GO−\d{4} − \d{2, 4} RUSTSEC−\d{4} − \d{4, 7} OSV−\d{4} − \d{4, 7} MAL−\d{4} − \d{4, 7} USN−\d{4} − \d{1, 2}
Source cve.mitre.org cwe.mitre.org github.com/advisories vuln.go.dev rustsec.org osv.dev/list github.com/ossf/malicious-packages ubuntu.com/security/cves
Fig. 2: Our study’s execution plan
1) Explicit References: Explicit vulnerability references are comparatively easier to identify because they follow standardized textual patterns. We therefore detect explicit references using regular expression matching. We use a keyword-based approach to identify candidate security-related signals, which are subsequently validated through LLM-assisted annotation and manual review. We apply the regular expression patterns shown in Table II independently to all analyzed pull request artifacts. In this study, we rely on the Open Source Vulnerability (OSV) specification3 , which defines a common interchange format for vulnerabilities and supports identifiers from multiple vulnerability databases and ecosystems. The AIDev-pop dataset references eight OSV-supported identifier schemes, consistent with prior work [13]. Each detected explicit reference is stored with the matched identifier, the pull request artifact where it was found, the associated pull request, the surrounding text, the author’s account classification, and the GitHub URL for verification. When multiple identifiers appear within the same artifact, each identifier is recorded separately. 2) Implicit Signals: Unlike explicit references, implicit vulnerability signals do not follow fixed identifier patterns and instead rely on natural-language descriptions of securityrelated concerns. To detect them, we adopt the keywordbased methodology proposed by Zhou and Sharma [11]. The approach combines security-related keywords with contextual filtering to distinguish security discussions from generic bugfixing language. We apply this approach to the same pull request artifacts analyzed for explicit references. Each detected implicit reference is stored with the same metadata collected for explicit references, including the artifact type, associated pull request, surrounding text, actor classification, and verification URL. Pull requests may contain both explicit and implicit references across different artifacts. 3 https://ossf.github.io/osv-schema
B. Manual Evaluation Following Rabbi et al.’s [20] LLM-based validation method, we validate the found explicit and implicit vulnerability signals using the Gemini-2.0-flash model. The model outputs a binary label, yes or no, if the signal references or communicates about a vulnerability. If Gemini-2.0-flash is no longer available at the time of data collection, we will use the closest available successor model and report this change in the final manuscript. To assess the reliability of the LLM annotations, we draw two representative samples of 360 pull requests each, one containing explicit signals and one containing implicit signals. This number is based on the 5,465 pull requests containing explicit security signals identified in our previous work. This sample size is determined under the assumption of a 95% confidence level and a 5% margin of error. The samples are stratified across actor types and signal categories. Two authors independently code this subset as having a vulnerability signal or not, based on the complete pull request contents. The interrater reliability is measured by Cohen’s κ [21], which indicates the agreement level between the human annotators. Cohen’s κ is computed before disagreement resolution. Disagreements will be discussed to agree on a shared coding scheme. The resulting human annotations serve as ground truth and are compared against the LLM-generated labels to evaluate the reliability of the automated annotations using accuracy, precision, recall, and F1-score [22]. The results are reported alongside the main findings. C. Account Type Classification We classify the author of each detected vulnerability reference as a human, bot, or coding agent. Coding agent accounts: To identify coding agent accounts, we use the user accounts associated with autonomous coding agents (e.g., devin-ai-integration[bot]), which were annotated in the original AIDev-pop dataset by Li et al. [16]. References appearing in artifacts authored by these accounts were classified as agent-generated references.
Bot accounts: Among the remaining accounts, we identified bots using three high-precision signals commonly used in prior work [13, 23, 24]. First, we identified accounts explicitly tagged by GitHub as bots, such as dependabot[bot]. Second, we identified usernames containing the string “bot.” Third, we matched accounts against a curated list of 385 wellknown bot accounts reported by Chidambaram et al. [24]. We additionally performed manual verification to improve the reliability of the classification, where we classified a representative random sample of 242 accounts. This is based on the unique 654 bot accounts found in our previous work, using a 95% confidence level and a 5% margin of error. Human accounts: All remaining accounts that are not classified as coding agents or bots are then classified as human contributors. D. Data Analysis The analyses are organized around the three research questions (RQ1–RQ3). Comparisons across actor types, pull request components, and other categories provide additional context for interpreting the results. Given the exploratory nature of this study, subgroup analyses are interpreted with caution and reported together with effect sizes. 1) RQ1: Distribution of Vulnerability References: In RQ1 , we compute the distribution of explicit and implicit vulnerability references across all PRs in AIDev-pop, by PR component. We report the proportion of security-mentioning PRs that use only implicit language, only explicit identifiers, or both. To compare the distribution of vulnerability references across actor types and pull request components, we use chi-square tests of independence and report Cramér’s V as an effect size. For repository-level analyses, we additionally report results aggregated by repository. 2) RQ2: Quality of Vulnerability Detection: To answer RQ2 we use the static analysis and software composition analysis (SCA) tool of Semgrep4 to evaluate the original code and the changed code to assess whether the proposed change introduces or fixes a vulnerability. We configure Semgrep with the default rule sets and the four severity levels: critical, high, medium and low. We selected Semgrep because it supports the programming languages represented in AIDev-pop and enables reproducible vulnerability scans [25, 26]. For each PR containing a vulnerability signal, we perform a static analysis scan on all commits in the PR. For pull requests with multiple commits, we also scan intermediate commits to trace how vulnerabilities evolve throughout the pull request. However, the primary classification of vulnerabilities as introduced, fixed, or unchanged is based on comparisons between the pull request’s base and head commits. The before version is defined as the base commit of the PR, the commit that is at the head of the branch at the time the PR was opened, which is the state of the code before any changes were made. To track vulnerabilities across commits, findings are matched using the same vulnerability ID and relative file path to the 4 https://semgrep.dev
vulnerable code. If a vulnerability exists in the before version of the pull request but was resolved in one of the following commits within the modified lines of the PR diff, it is classified as fixed. If a vulnerability appears in the head commit but was absent in the base commit, it is classified as introduced. When multiple findings are present within a PR, we aggregate the results at the PR level based on whether the vulnerabilities were introduced, fixed, or unchanged. The after version is defined as the head commit of the PR, meaning the last commit of the PR at the time of closure, regardless of whether the PR was merged. For merged PRs, this means the code that was integrated into the branch. For closed but unmerged PRs, the head commit has the final changes and is scanned to assess whether the vulnerability signal corresponded to a vulnerability in the changed code, even if those changes were never accepted. This analysis evaluates how accurately the presence of an implicit or explicit vulnerability signal in PR descriptions reflects its reliability. A PR might contain a detailed security explanation without actually addressing a vulnerability, whereas another may describe a concrete vulnerability without referencing a formal identifier. By linking these outcomes to the type of vulnerability signal used in the PR, we can analyze whether explicit or implicit references are more strongly associated with actual vulnerability fixes, and whether this relationship varies across actor types. To assess these relationships, we fit regression models using the Semgrep outcome (introduced, fixed, or unchanged) as the dependent variable and vulnerability reference type and actor type as independent variables. The models additionally control for pull request characteristics, including the number of changed files, lines added or deleted, and the number of commits. We account for repository-level clustering to reduce the influence of repository-specific development practices. 3) RQ3: Social Response: To answer RQ3 , we compare review activity and pull request outcomes across different vulnerability reference types and account categories. Specifically, we analyze the number of comments, number of reviewers, pull request status (merged, closed, or open), and time until the first response for pull requests containing explicit and implicit vulnerability references. We consider the final state of each pull request at the cut-off date of August 1, 2025. We further investigate whether implicit references are associated with longer discussions, delayed responses, or lower merge rates compared to explicit references. Since review activity may also depend on pull request characteristics unrelated to vulnerability communication, we control for factors such as the number of changed files, the number of lines added or deleted, the repository, and the programming language. This helps account for differences caused by pull request size, complexity, or repository context. We use statistical models appropriate to each outcome variable. Pull request status is analyzed using logistic regression, review activity measures, such as the number of comments and reviewers, are analyzed using count-based regression models, and time until first response is analyzed using time-to-event models. All models
include the control variables described above and account for repository-level clustering. VI. T HREATS TO VALIDITY Internal Validity. Our identification of explicit vulnerability references relies on regular expression matching. Although identifiers such as CVEs, CWEs, GHSAs, and Go vulnerability IDs follow standardized formats, malformed identifiers may go undetected. Detecting implicit vulnerability references is more challenging because it relies on natural-language descriptions rather than fixed patterns. Although we adopt the validated keyword-based approach of Zhou and Sharma [11], implicit references may still produce false positives or false negatives. To mitigate this threat, detected signals are validated through LLM-assisted annotation and a manually reviewed sample of pull requests across different account types and reference categories. Account classification may also introduce inaccuracies. Coding agent accounts rely on the AIDev-pop annotations of Li et al. [16], while bot identification combines GitHub bot tags, username heuristics, and a curated list of known bot accounts. Consequently, some automated accounts may still be misclassified. RQ2 further depends on Semgrep and Semgrep Supply Chain to assess whether pull request changes introduce, fix, or preserve vulnerabilities. Static analysis and software composition analysis tools may generate false positives or fail to detect vulnerabilities, particularly for complex or contextdependent security issues. External Validity. We focus on popular open-source GitHub repositories with more than 500 stars. As a result, our findings may not generalize to smaller repositories, private repositories, or software projects hosted outside GitHub. The AIDev-pop dataset focuses on repositories that actively use coding agents and therefore contains a higher proportion of agent-authored pull requests than would be expected in the broader GitHub ecosystem. Consequently, our findings should be interpreted as characterizing vulnerability communication practices within repositories where coding agents are present rather than as prevalence estimates across all open-source projects. To mitigate this threat, we report observed actor distributions, rely on proportional measures rather than raw counts, and account for repository-level variation in the analyses. The prevalence and style of vulnerability references may also vary across programming languages, ecosystems, and repository communities. Finally, our analysis focuses on pull request artifacts and does not capture vulnerability communication occurring through external channels such as mailing lists, private disclosures, chat platforms, or issue trackers outside GitHub. VII. C ONCLUSION This registered report presents a study of explicit vulnerability references and implicit security-related signals in GitHub pull requests authored by humans, bots, and coding agents. We examine how these references are communicated, how they relate to vulnerabilities in the modified code, and how they are associated with review activity and pull request
outcomes. Future work could investigate whether vulnerability references correspond to incorrect or unsupported security claims, particularly in AI-generated pull requests. VIII. ACKNOWLEDGEMENTS Pien’s work is supported by the Dutch science foundation NWO through the KIC “Find2Fix” project (No. and Milo KICH1.VE05.23.008). Special thanks to Fred . R EFERENCES [1] D. Nakano, M. Yin, R. Sato, A. Hindle, Y. Kamei, and N. Ubayashi, “A Quantitative Study of Security Bug Fixes of GitHub Repositories,” arXiv preprint arXiv:2012.08053, 2020. [2] G. Bhandari, A. Naseer, and L. Moonen, “CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, ser. PROMISE 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 30–39. [Online]. Available: https://doi.org/10.1145/3475960.3475985 [3] D. Hommersom, A. Sabetta, B. Coppola, D. D. Nucci, and D. A. Tamburri, “Automated Mapping of Vulnerability Advisories onto their Fix Commits in Open Source Repositories,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 5, p. 1–28, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3649590 [4] S. Liu, J. Zhou, X. Hu, F. R. Cogo, X. Xia, and X. Yang, “An Empirical Study on Vulnerability Disclosure Management of Open Source Software Systems,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 7, pp. 1–31, Aug. 2025. [Online]. Available: https://doi.org/10.1145/3716822 [5] J. Ayala, Y.-J. Tung, and J. Garcia, “A Mixed-Methods Study of Open-Source Software Maintainers on Vulnerability Management and Platform Security Features,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 2105–2124. [6] S. Reis, R. Abreu, and C. Pasareanu, “Are security commit messages informative? Not enough!” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 196–199. [Online]. Available: https://doi.org/10.1145/3593434.3593481 [7] R. Croft, Y. Xie, M. Zahedi, M. A. Babar, and C. Treude, “An empirical study of developers’ discussions about security challenges of different programming languages,” Empirical Software Engineering, vol. 27, no. 1, p. 27, 2022. [Online]. Available: https://doi.org/10.1007/ s10664-021-10054-w [8] G. Antal, M. Keleti, and P. Hegedundefineds, “Exploring the Security Awareness of the Python and JavaScript Open Source Communities,” in
Proceedings of the 17th International Conference on Mining Software Repositories, ser. MSR ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 16–20. [Online]. Available: https://doi.org/10.1145/3379597.3387513 [9] S. H. B. I. Kumar, L. R. Sampaio, A. Martin, A. Brito, and C. Fetzer, “A Comprehensive Study on the Impact of Vulnerable Dependencies on OpenSource Software,” in 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2024, pp. 96–107. [Online]. Available: https: //doi.org/10.1109/ISSRE62328.2024.00020 [10] S. Kancharoendee, T. Phichitphanphong, C. Jongyingyos, B. Reid, R. G. Kula, M. Choetkiertikul, C. Ragkhitwetsagul, and T. Sunetnanta, “On Categorizing Open Source Software Security Vulnerability Reporting Mechanisms on GitHub,” in 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2025, pp. 751–756. [Online]. Available: https://doi.org/10.1109/SANER64311.2025.00076 [11] Y. Zhou and A. Sharma, “Automated Identification of Security Issues from Commit Messages and Bug Reports,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 914–919. [Online]. Available: https://doi.org/10.1145/3106237.3117771 [12] D. Cipollone, C. Wang, M. Scazzariello, S. Ferlin, M. Izadi, D. Kostić, and M. Chiesa, “Automating the detection of code vulnerabilities by analyzing github issues,” in 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 2025, pp. 41–48. [Online]. Available: https: //doi.org/10.1109/LLM4Code66737.2025.00010 [13] P. Rooijendijk, C. Treude, and M. Wessel, “Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests,” in 2026 IEEE/ACM 23rd International Conference on Mining Software Repositories (MSR), 2026. [Online]. Available: https://doi.org/10.1145/3793302.3793616 [14] M. L. Siddiq, X. Zhao, V. C. Lopes, B. Casey, and J. Santos, “Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub,” arXiv preprint arXiv:2601.00477, 2026. [15] F. Wang, B. Do, and J. Jermier, “Automated vs. human security patching patterns in pull requests: Evidence from the aidev dataset,” 2025. [Online]. Available: https://plg.uwaterloo.ca/∼migod/846/current/ projects/04-FelixJacieBrian-report.pdf [16] H. Li, H. Zhang, and A. E. Hassan, “The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents are Reshaping Software Engineering,” arXiv preprint arXiv:2507.15003, 2025. [17] M. Alfadel, D. E. Costa, E. Shihab, and M. Mkhallalati, “On the Use of Dependabot Security Pull Requests,” in 2021 IEEE/ACM 18th International Conference on
Mining Software Repositories (MSR), 2021, pp. 254–265. [Online]. Available: https://doi.org/10.1109/MSR52588. 2021.00037 [18] H. Mohayeji, A. Agaronian, E. Constantinou, N. Zannone, and A. Serebrenik, “Investigating the Resolution of Vulnerable Dependencies with Dependabot Security Updates,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 2023, pp. 234–246. [Online]. Available: https://doi.org/10.1109/MSR59073.2023.00042 [19] B. Steenhoek, K. Sivaraman, R. S. Gonzalez, Y. Mohylevskyy, R. Z. Moghaddam, and W. Le, “Closing the Gap: A User Study on the Realworld Usefulness of AI-powered Vulnerability Detection & Repair in the IDE,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 01–13. [Online]. Available: https://doi.org/10.1109/ICSE55347.2025.00126 [20] M. F. Rabbi, A. K. Turzo, A. I. Champa, and M. F. Zibran, “Insights into Security-Related AI-Generated Pull Requests,” arXiv preprint arXiv:2604.19965, 2026. [21] S. M. Vieira, U. Kaymak, and J. M. Sousa, “Cohen’s Kappa Coefficient as a Performance Measure for Feature Selection,” in International conference on fuzzy systems. IEEE, 2010, pp. 1–8. [Online]. Available: https://doi.org/doi={10.1109/FUZZY.2010.5584447} [22] D. Rao and C. Callison-Burch, “Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why,” arXiv preprint arXiv:2606.00093, 2026. [23] A. Abdellatif, M. Wessel, I. Steinmacher, M. A. Gerosa, and E. Shihab, “BotHunter: An Approach to Detect Software Bots in GitHub,” in Proceedings of the 19th International Conference on Mining Software Repositories, ser. MSR ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 6–17. [Online]. Available: https://doi.org/10.1145/ 3524842.3527959 [24] N. Chidambaram, A. Decan, and T. Mens, “A Dataset of Bot and Human Activities in GitHub,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 2023, pp. 465–469. [Online]. Available: https://doi.org/10.1109/MSR59073. 2023.00070 [25] K. Kuszczyński and M. Walkowski, “Comparative Analysis of Open-Source Tools for Conducting Static Code Analysis,” Sensors, vol. 23, no. 18, p. 7978, 2023. [Online]. Available: https://doi.org/10.3390/s23187978 [26] G. Bennett, T. Hall, E. Winter, and S. Counsell, “Semgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 614–623. [Online]. Available: https://doi.org/10.1145/3661167.3661262