arXiv CSopen access

Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study

Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study Zahra Mousavi1 1

Chadni Islam2 M. Ali Babar1 Kristen Moore3

Alsharif Abuadbba3

Centre for Research on Engineering Software Technologies (CREST) & Adelaide University, Australia 2 3 Edith Cowan University, Australia CSIRO’s Data61, Australia

arXiv:2607.11348v1 [cs.SE] 13 Jul 2026

Abstract AI code assistants are transforming software development, but their implications for software security remain a major concern, particularly in the context of security APIs. These APIs are critical for safeguarding software systems, yet their complexity often leads to incorrect use and serious vulnerabilities. Developing an evidence-based understanding of how AI assistants influence developers’ use of these APIs is therefore essential for informing effective mitigation strategies. While a few user studies have examined the broader impact of AI assistants on software vulnerabilities, the use of security APIs remains unexplored from a developer-centered perspective. This study addresses this gap by presenting the first empirical investigation into how AI code assistants affect professional developers’ use of security APIs. We conducted a study with 44 developers who completed security API programming tasks with and without GitHub Copilot assistance. Our findings show that, while Copilot improves functional correctness and marginally reduces certain insecure patterns, it does not significantly improve secure API usage. We also found that developers rarely raised security concerns when engaging with Copilot, and many did not recognize that their final implementations remained insecure. Finally, we offer recommendations for enhancing security awareness among developers and propose future research directions to support safer AI-assisted software development.

Keywords: Security API · AI Code Assistants · Software Security

1

Introduction

nication between a user and the application, and obtain 3 unauthorized access to the user’s information (Step ○). The underlying reasons for the prevalent misuse of security APIs include insufficient security training among developers [9–11], inadequate or unclear API documentation [12], and the inherent complexity of the security APIs themselves [3], which can overwhelm developers with a confusing range of configurations and options. These challenges often lead developers to turn to alternatives such as AI code assistants, which enable them to easily find solutions to development problems [13]. Modern AI code assistants, driven by Large Language Models (LLMs), are rapidly becoming integral to developers’ workflows, with many relying on them to streamline and accelerate their daily programming activities [13]. For example, GitHub Copilot, currently the most widely used AI code assistant, has surpassed 20 million users by mid-2025 and is used by major companies and around 90% of Fortune 100 firms [14]. This growing reliance raises serious concerns regarding the quality and security of the LLM-generated code [15– 27]. LLMs like Copilot have been trained on vast code repositories containing both secure and insecure examples, posing the risk of reproducing insecure patterns. These concerns are especially serious in the context of security APIs, where their misuse can lead to severe vulnerabilities and compromise system security [1]. Therefore, it is imperative to understand how these tools influence developers’ use of security APIs in practice. A recent study [25] revealed that approximately 70%

Security Application Programming Interfaces (APIs) play a crucial role in modern software development by providing essential functionalities, such as encryption and secure communication [1]. Developers rely heavily on these APIs to protect various types of applications against cyber threats. SSL/TLS APIs are a prominent example and are extensively integrated within a wide range of platforms, including web browsers, mobile applications, and cloud services, to ensure the confidentiality and integrity of data transmitted over networks [2]. However, using security APIs correctly remains a significant challenge for developers, resulting in their widespread misuse (i.e., incorrect use) across real-world software systems and open-source codebases [1,3–8]. Notably, security API misuse accounts for a substantial number of security vulnerabilities that expose systems to large-scale data breaches and significant financial losses [3–8]. A preliminary study on non-browser software found critical misuses of SSL/TLS APIs, such as bypassing certificate validation in security-critical applications and libraries, ranging from payment gateways to mobile banking apps [3]. Such misuse exposes software to Man-in-the-Middle (MitM) attacks, compromising the confidentiality and integrity of network communications. Fig. 1 illustrates this type of misuse. A developer uses an SSL/TLS API to establish a secure connection with 1 a server (Step ○) but incorrectly configures it to trust 2 all certificates (Step ○). This misuse enables a MitM attacker to impersonate the server, intercept the commu1

Figure 1: A misuse of an SSL/TLS API leading to the leakage of user personal information • We present the first developer-centered empirical study of AI-assisted programming with security APIs, comparing Copilot-assisted and unaided development in realistic task settings.

of security API code generated by ChatGPT exhibited misuse patterns. This analysis was based solely on model outputs generated from predefined prompts, without developer involvement. This leaves three critical gaps. First, existing studies lack real-world evaluations that capture developers’ role in interpreting, modifying, accepting, or rejecting AI-generated code—decisions that directly influence the security of the final implementation. While a few developer-centered studies have explored the broader impact of AI assistants on security vulnerabilities [19, 20, 24], the developers’ role in using security APIs with AI assistants remains unexplored. Second, it remains unclear how AI-assisted development compares to unaided coding for security APIs, both in terms of security outcomes and functional correctness, as many misuses only arise once core API calls are correctly integrated and implementations execute as intended. Third, little is known about developers’ security awareness when engaging with AI assistants in security-sensitive contexts, including whether they consider security during interaction with the tool and whether they recognize security issues in the resulting code. Therefore, conducting developer-centered studies is paramount for building an evidence-based understanding of how AI code assistants influence the use of security APIs in practice. This study presents the first empirical investigation of its kind into developer engagement with AI code assistants for implementing security API tasks in realistic settings. We focus on Copilot in this study, as it is currently the most widely adopted code assistant [14, 28, 29]. Our study investigates three key Research Questions (RQs):

• We show that Copilot can improve functional correctness, especially for more complex security API tasks, but does not enable developers to produce fully secure implementations. • We identify the types of security API misuses that persist, are introduced, or are partially mitigated in Copilot-assisted development. • We reveal limited security-aware engagement with Copilot: only two participants explicitly considered security in their prompts, and many failed to recognize that their final implementations remained insecure, despite being informed that security would be evaluated. • We further show through a post-study prompting analysis that targeted security-focused prompts can help address some misuses, although Copilot still does not reliably prevent all of them. Together, these findings underscore the need to foster a security-conscious mindset in AI-assisted development and to support developers in critically assessing the security of AI-generated code. We further discuss the implications of these findings and provide recommendations for developers, tool designers, and researchers. All study materials are available online [30]. The remainder of this paper is organized as follows: Section 2 covers background and related work. Section 3 details the methodology. Section 4 presents results, followed by analysis in Section 5. Threats to validity are discussed in Section 6, and the paper concludes in Section 7.

• RQ1: How effectively does Copilot help developers produce functionally correct code for security API tasks? • RQ2: How does Copilot affect the secure use of security APIs, and which misuse types does it introduce or mitigate?

2

Related work

Several studies have raised significant concerns about the security implications of AI assistants for code generation [15–27]. An early study by Pearce et al. [15] showed that approximately 40% of Copilot’s suggestions contained security flaws. Similar studies have confirmed that AI-generated code can inherit insecure patterns from training data [15–17, 21, 22, 27]. However, these analyses were largely conducted in controlled environments using predefined prompts without involving developers. Recognizing this gap, recent studies have shifted toward

• RQ3: To what extent do developers demonstrate security awareness when engaging with Copilot for security API tasks? We conducted a user study with 44 professional developers, where each completed two development tasks involving different security APIs, one with Copilot and one without. We performed quantitative and qualitative analyses of the resulting code artifacts and developers’ engagement with Copilot. This paper makes the following contributions: 2

Table 1: Comparison of studies on security implications of AI-assisted coding Study Sandoval et al. [19] Perry et al. [20]

AI Model Codex

Participants 58 students

Codex

Asare et al. [24]

Copilot

Fu et al. [26]

Copilot, CodeWhisp., Codeium ChatGPT

47 students/professionals 25 students/professionals None (GitHub artifacts)

Mousavi et al. [25] Our Study

Copilot

None (model-based) 44 professionals

Sec. Focus Tasks MITRE Top Linked-list 25 CWEs task (C) Web vulnera- 5 security tasks bilities (Py/JS/C)

Key Findings Improved correctness; < 10% increase in critical CWEs

13 CWEs (e.g., SQLi)

2 real-world C tasks

Improved security only for complex tasks

43 CWEs

Py/JS AI snippets

30% vulnerable snippets; includes 8 CWE Top 25

API misuse

48 Java tasks (5 APIs) 2 Java tasks

70% outputs contained API misuses

API misuse

evaluations in real-world contexts. Fu et al. [26] examined code snippets generated by developers using Copilot and two other AI tools in GitHub projects, finding that approximately 30% contained security weaknesses spanning 43 CWE types, including eight from the CWE Top 25 list. To assess Copilot’s performance relative to human developers, Asare et al. [18] used a dataset of vulnerabilities introduced in human-written code to construct prompts. They found that Copilot reproduced the same vulnerabilities in only about 33% of cases, suggesting that while it generates insecure code, it does not perform worse than developers. User studies have also examined the security implications of AI assistance during coding, though findings vary by tool and task complexity. Sandoval et al. [19] found that Codex improved functional correctness while increasing critical security bugs by at most 10%. In contrast, Perry et al. [20] found that 47 students and professionals using Codex produced significantly less secure code and were more likely to overestimate its security. Focusing on Copilot, Asare et al. [24] studied 25 students and professionals and found that Copilot improved security only for more complex problems, with no significant effect on simpler ones or on specific vulnerability categories. Notably, none of the aforementioned studies specifically addressed the critical issue of security API misuse, which remains a major concern in secure software development. Compared with general vulnerabilities, security API misuse places greater emphasis on compliance with API-specific security constraints and often leads to severe security consequences. Mousavi et al. [25] investigated this issue using ChatGPT and found misuses in 70% of generated code instances for security API tasks. However, their study relied on researcher-crafted prompts and did not involve professional developers. This is a key limitation, as developers ultimately shape the final code by accepting, modifying, or rejecting AI-generated suggestions [31, 32]. Further, its focus on ChatGPT leaves a gap in understanding how Copilot—the most widely adopted AI assistant among developers—affects security API usage in practice. Our study addresses these gaps through a developer-centered, real-world evaluation. Specifically, (1) we involve professional developers in realistic programming tasks to assess security API usage

Less secure code; security overestimation

Improved correctness; no significant security gain; security often overlooked and overestimated

in practice; (2) we focus on Copilot given its widespread adoption; and (3) we analyze developers’ consideration of security in their prompts and its impact on final code security. Table 1 outlines the key differences between our study and prior work. To our knowledge, this is the first in-depth study of how GitHub Copilot influences security API usage in real-world development settings.

3

Study Design

This section outlines the design of our empirical study, including the programming tasks, participant recruitment process, study procedure, experimental setup, and evaluation methodology. An overview of the research methodology is illustrated in Fig. 2. The following subsections elaborate on each component, along with a discussion of the ethical considerations relevant to the study.

3.1

Task Design

To design effective programming tasks for our study, we aimed to achieve three primary goals. First, tasks were crafted to reflect real-world security challenges that developers commonly face in practice. Second, they were specifically designed to expose developers to common security API misuses, allowing us to observe whether participants would address them. Finally, we ensured that tasks are feasible to complete within a reasonable timeframe. We selected Java as the target language due to its widespread adoption in software development and the inherent complexity of its security APIs, which are frequently prone to misuse by developers in practice [33, 34]. To fit within the practical time limits in a controlled user study while enabling in-depth analysis, we limited our scope to two security APIs that represent distinct security functionalities and allow observation of diverse misuse patterns across realistic scenarios. We focused on two widely used and security-critical Java APIs: (i) Java Secure Socket Extension (JSSE), which enables secure communication over SSL/TLS protocols [2, 3, 21], and (ii) Google OAuth, which enables applications to access Google user data on their behalf without exposing their credentials [7, 35]. While this 3

Figure 2: An overview of the research methodology for the user study selection does not cover the full spectrum of security APIs, it reflects two major classes of security functionality, secure communication and delegated authorization, both of which are commonly misused and associated with high-impact vulnerabilities [1]. Each of these APIs formed the basis of one of the two programming tasks assigned in our study. The first task required participants to use JSSE to implement a createSSLSocket method. This method needs to configure an SSL/TLS socket that allows a client application to securely connect to a server. This task represents one of the most frequent patterns for secure communication and exposes developers to well-known misuses such as improper protocol selection, certificate validation, and hostname verification [2, 3, 21]. The second task introduced a more complex programming scenario. Participants were required to implement an authorize method using the Google OAuth API as part of a desktop application. The application was designed to retrieve the number of unread messages in a user’s Gmail inbox, subject to user consent. This task captures a realistic OAuth integration workflow and exposes participants to frequent authorization-flow misuses identified in prior work [7, 35]. For both tasks, we provided participants with a skeleton application that included stub code for non-security-related functionalities. Participants were then asked to implement the essential security components. Full task descriptions are available in the online supplementary material [30].

3.2

recruiter service, which directly contacts freelancers with relevant skill sets and invites them to apply. Additionally, we encouraged participants to share the project with other eligible freelancers within their networks. Interested freelancers submitted bids with a payment offer and a short proposal. In total, we received 207 applications. A multi-stage screening process was implemented to select eligible candidates. First, we excluded applicants with a rating below 4.5 (unless new to the platform), and those requesting higher compensation than offered. Next, we reviewed the proposals, excluding those that were irrelevant to the project. We then verified that applicants met the eligibility requirement of at least one year of Java experience. If a developer’s profile did not clearly demonstrate Java expertise, we requested additional supporting materials, such as links to GitHub repositories showcasing their Java projects. Following this screening process, 143 applicants were invited to participate. Each invited participant received a participant information sheet detailing the study, a consent form, and a link to a registration form that collected their demographic information and allowed them to select a convenient time for participation. As a result, 75 registered, of whom 11 later withdrew from the study, and 15 failed to attend their scheduled sessions. Of the 49 who joined the study and completed the programming tasks, five were excluded due to non-compliance with study guidelines, such as suspicious use of AI tools on the task that was intended to be completed independently without AI assistance. Ultimately, data from 44 participants were included in our final analysis. Each participant who completed the study tasks received a compensation of AUD $100. Fig. 3 shows a demographic overview of the final participants. Further details on anonymized participants and recruitment materials are available online [30].

Recruitment and Participant Pool

Following prior security-focused work [24], we recruited professional developers aged 18+ with ≥1 year of Java experience. Prior security knowledge was not required as we aimed to examine how Copilot influences developers’ perceptions and experiences with security APIs regardless of their background. In addition, we required familiarity with VS Code, the IDE used in our study. We selected Freelancer as our recruitment platform, based on recommendations from prior research highlighting its effectiveness in recruiting participants for security-focused development studies [36]. Freelancer has also been used in several user studies involving security programming tasks [37–40]. Over a period of nearly three months, we posted a project on Freelancer in multiple iterations. The project description clearly outlined the eligibility criteria and provided a concise overview of the study. While we mentioned that the study involved security programming tasks, we did not disclose the specific APIs under evaluation. To expand our reach, we also used Freelancer’s

3.3

Study Procedure

We conducted a within-subject study [41] in which all participants completed tasks under both experimental and control conditions. In the experimental condition, participants completed a task with Copilot assistance, whereas in the control condition, they completed a different task without any AI tool. To mitigate learning and fatigue effects, we counterbalanced Copilot usage (enabled vs. disabled) and task order (JSSE first vs. OAuth first). This led to four groups (2 conditions × 2 task orders), labeled A, B, C, and D, with 11 participants each. For example, Group A completed the JSSE task with Copilot, followed by the OAuth task without Copilot. The remaining groups followed differ-

4

Figure 3: Participant overview (n=44): (a) Java experience and Copilot use frequency; (b) prior API experience and familiarity with the corresponding security protocol. ent combinations of task order and Copilot availability, ensuring that for each task, half of the participants used Copilot while the other half did not. Group assignment followed permuted block randomization (block size = 4), and each participant was assigned an ID based on their group and number (e.g., A1–11, B1–11). To replicate a realistic coding environment, participants in both settings were allowed to access internet resources. Those in the control condition were restricted from using any AI assistants, including Copilot, while those in the experimental condition had access to all Copilot features (e.g., inline completions, chat interface, and fixes). To ensure familiarity with the tool, all participants were asked to review a brief tutorial before their sessions. Since developers need clear prompts to write secure code [38], participants were informed that their solutions would be evaluated for both correctness and security. They were given 2 hours to complete both tasks, with an optional 30-minute break after the first one. Although full functionality was not required for compensation, they were encouraged to complete as much as possible within this time. After each task, participants completed a short survey that asked whether this was their first experience with the API used, about their familiarity with the relevant security protocol, and whether they believed they had solved the task securely. With participants’ consent, all sessions were screen-recorded for compliance monitoring and later analysis.

3.4

instance from their respective group, which they accessed remotely from their personal devices to complete the tasks during their scheduled time.

3.5

Evaluation Methodology

Our evaluation involved (i) Code Analysis (RQ1 and RQ2 ) and (ii) Security Awareness Analysis (RQ3 ), as detailed below. 3.5.1

Code Analysis.

For each task, we analyzed the functionality and security of code samples. Functionality was assessed by whether the solutions executed without errors and produced the expected outputs. Solutions that required only minor adjustments to become functional were categorized as semi-functional. Only functional and semi-functional solutions proceeded to the security evaluation. The security evaluation focused on identifying instances of security API misuse within code samples. This assessment was guided by the taxonomy of security API misuses established by Mousavi et al. [1]. Each implementation was analyzed for the presence of misuse patterns defined in this taxonomy. Several tools exist for detecting security API misuse, but they often produce high rates of false positives and negatives, limiting their applicability to our context. Preliminary tests with CryptoGuard [5], a tool recognized for its precision, failed to effectively detect JSSE misuses relevant to our study. Additionally, no automated tools currently exist to detect OAuth misuses. Given these limitations, the first author and a software security researcher from our lab, each with four years of security experience, independently conducted a manual review of all participant submissions. Each reviewer assessed the implementations against our predefined misuse criteria and recorded the presence or absence of each misuse category. We observed a high level of inter-rater agreement (κ = 0.97), indicating strong consistency between the two reviewers’ assessments. Any disagreements were subsequently resolved through discussion, with both reviewers revisiting the corresponding code and reaching a final consensus classification. To statistically examine the effects of Copilot use and participants’ background on our results, we applied logistic regression [42]. Logistic regression is well suited for modeling binary dependent outcomes (e.g., correct

Experimental Infrastructure

All participants completed the study within a controlled virtual environment using Amazon Web Services (AWS) virtual machines (VMs), each configured with 16 vCPUs, 64 GiB of RAM, and running Ubuntu 20.04 LTS. We pre-installed VS Code, the required Java packages, and GitHub Copilot (Version v1.250.0, running on GPT-4o, the latest available version at the time of conducting the study). To support the study design, we prepared four distinct VM configurations corresponding to the four participant groups. For each group, two separate VS Code instances—Copilot enabled and disabled—were provided to ensure Copilot usage was aligned with the assigned condition for each task. Each VM included a README file containing study instructions, as well as the specific task order based on the participant’s group assignment. Participants were assigned a unique VM 5

vs. incorrect) and enables us to estimate the influence of multiple predictors simultaneously. In our models, the primary predictors were the Copilot condition (with vs. without Copilot) and participants’ background characteristics. We fit separate models for each task, such that each participant contributed a single observation per model. This design ensured there were no repeated measures within a given model, and therefore satisfied the independence assumption required for logistic regression. 3.5.2

the clarity of the task instructions and the feasibility of completing the programming tasks within the allotted time. After completing the tasks, participants filled out a short survey about their experience, including any difficulties in understanding the tasks or using the study environment. We also gathered additional qualitative feedback through follow-up discussions, conducted via Freelancer chat for remote participants and in person for lab-based participants. Based on this feedback, we refined the task descriptions, added explicit completion criteria, expanded the step-by-step navigation instructions for the study environment, and addressed technical issues identified during the pilot such as screen recording interruptions.

Security Awareness Analysis.

To investigate developers’ security awareness, we analyzed (i) prompts to assess whether participants considered security during interaction, and (ii) post-task self-assessments to determine whether they recognized security problems in their final implementations.

3.7

This study was approved by our organization’s Human Research Ethics Committee and conducted in full compliance with its ethical guidelines.

Prompt Analysis. This analysis focused exclusively on the experimental condition in which participants used Copilot. Specifically, we examined the natural-language prompts participants used to interact with Copilot to assess whether they expressed any security-related intentions or concerns. Out of 44 participants, the chat session histories were available for 38. We extracted these logs directly from Copilot’s chat interface. For the remaining 6 participants, the chat histories were unavailable because the developers had closed their Copilot Chat sessions, and Copilot does not persist conversation histories across sessions. In these cases, we manually reviewed participants’ screen recordings to capture their prompts. Additionally, the participants could interact with Copilot through inline chat prompts within the code editor or by embedding natural-language instructions as comments to receive code suggestions. Since these interactions are not recorded in the chat interface, we again relied on screen recordings to manually extract such prompts. All collected prompts were manually analyzed to identify any indications of security awareness. We also reviewed Copilot’s responses and participants’ final implementations to assess how security-oriented prompts affected the generated code, particularly regarding API misuses.

4

Experimental Results

In this section, we present our findings for each research question.

4.1

RQ1: Functionality Analysis

This section addresses RQ1 by examining the extent to which Copilot helps developers produce functionally correct code. We analyzed participants’ success in completing the assigned tasks and assessed how AI assistance influenced their performance. For the simpler JSSE task, all participants produced functional code regardless of Copilot use, although those with Copilot completed the task faster on average (14 minutes vs. 26 minutes). In contrast, the OAuth task was more challenging, particularly without AI assistance. In the control condition, only 45% of participants (10/22) produced functional code, with an average completion time of 62 minutes. With Copilot, the success rate rose to 91% (20/22), while the average completion time decreased by 63% to 23 minutes. According to the logistic regression analysis, Copilot had a statistically significant positive effect on functional outcomes (coefficient = 3.36, p = 0.007), indicating that participants who used Copilot were significantly more likely to complete the task successfully than those who did not, even after controlling for background factors. Among the background predictors, programming experience (p = 0.041) and prior API experience (p = 0.046) were significantly associated with functional outcomes, whereas security familiarity and Copilot use frequency showed positive but statistically non-significant trends. We also reviewed non-functional samples and identified semi-functional implementations for inclusion in the subsequent security analysis. Fig. 4 presents the distribution of non-functional, semi-functional, and functional OAuth implementations across participant background levels.

Self-Assessment Analysis. To investigate whether participants recognized security issues in their implementations, we analyzed post-task self-assessments collected after each task. Participants rated their agreement with the statement “I believe that I solved this task securely” on a 5-point Likert scale ranging from Strongly Disagree to Strongly Agree. In this study, responses of Disagree and Strongly Disagree were interpreted as explicit recognition that the solution was insecure. These responses were then compared against the actual security outcomes identified through our misuse analysis.

3.6

Ethics Approval

Pilot Study

Prior to the main study, we conducted a pilot study with six developers, including two from our lab and four recruited via Freelancer. The pilot aimed to assess

6

Figure 4: Distribution of OAuth task functionality levels by background. Copilot use freq and security familiarity were rated on a 5-point scale and grouped into low (1–3) and high (4–5).

assisted condition implemented custom trust managers that accepted all certificates, bypassing authentication and enabling MitM attacks. Listing 1 shows a developer using an empty certificate-validation method, thereby accepting all certificates as valid. Fig. 6 shows the distribution of M2 across background levels, although no meaningful trends can be drawn given the small number of cases. M3: Missing hostname verification. Hostname verification is a critical security measure that ensures the hostname in the SSL certificate matches the server hostname the client is connecting to. Without it, an attacker can intercept communication by presenting a certificate for a malicious server. In our study, none of the implementations performed hostname verification. Listings 1 and 2 show examples from both conditions in which hostname verification was omitted.

Remark 1 • All participants completed the JSSE task with functional solutions. • Copilot significantly improved functionality for the more complex OAuth task.

4.2

RQ2: Security Analysis

This section addresses RQ2 by analyzing security outcomes. Our analysis revealed a concerning trend: none of the implementations were fully secure, regardless of Copilot use or participant background. This is particularly concerning because participants had been clearly informed that their work would be evaluated for both functionality and security, and the task descriptions explicitly highlighted the security-sensitive nature of the tasks. Fig. 5 presents the identified misuses and their corresponding rates. We define misuse rate as the percentage of analyzed implementations exhibiting a given misuse. We next discuss the misuses identified for each API, assigning each a unique identifier (M#).

1 private 2 3 4 5 6

4.2.1

JSSE.

7 8

Our study found 3 JSSE misuses as follows.

9 10

M1: Insecure SSL/TLS standard. SSL and older TLS versions (1.0/1.1) are vulnerable to attacks such as POODLE, BEAST, and CRIME, and are therefore considered insecure [43–45]. These protocols have been deprecated, with TLS 1.2 as the minimum recommended secure version. In our study, 19 participants in the control condition and 18 in the AI-assisted condition either used outdated versions or failed to specify the TLS version, potentially allowing insecure defaults. Listing 1 (line 9) shows a control-condition example using SSL, while Listing 2 shows an AI-assisted example in which the TLS version is not specified. Fig. 6 presents the rates of this misuse across background levels, with and without Copilot. In addition to Copilot use, prior API experience showed a trend toward lower misuse rates, although the effect was not statistically significant. M2: Trusting all certificates. The default JSSE trust manager validates certificates against the system trust store. In our study, we added our server’s certificate to the trust store on all machines, allowing participants to rely on the default trust manager for secure validation. Most participants used this mechanism. However, two participants in the control condition and one in the AI-

11 12 13 14 15 16 17

static SSLSocket createSSLSocket ( String host , int port ) throws UnknownHostException , IOException { TrustManager [] trustAllCerts = new TrustManager []{ new X509TrustManager () { @Override public java . security . cert . X509Certificate [] g et A cc e pt e dI s su er s () { return null ; } @Override public void ch e ck C li e nt Tr u st e d ( java . security . cert . X509Certificate [] certs , String authType ) {} @Override public void ch e ck S er v er Tr u st e d ( java . security . cert . X509Certificate [] certs , String authType ) {} }}; try { SSLContext sc = SSLContext.getInstance("SSL"); sc . init ( null , trustAllCerts , new java . security . SecureRandom () ) ; H t tp s UR L Co n ne c ti o n . s e t D e f a u l t S S L S o c k e t F a c t o r y ( sc . getSocketFactory () ) ; SSLSocketFactory factory = ( SSLSocketFactory ) SSLSocketFactory . getDefault () ; SSLSocket s =( SSLSocket ) factory . createSocket ( host , port ) ; s . startHandshake () ; return s ; } catch ( G e n e r a l S e c u r i t y E x c e p t i o n e ) { System . out . println ( e . getStackTrace () ) ; } return null ; }

Listing 1: JSSE code sample (control group) with M1 (L9), M2 (L2), and M3. 1 private 2 3 4 5 6

7 8 9 10 11 12 13 14 15

static SSLSocket createSSLSocket ( String host , int port ) { try { // Create an SSL context SSLContext sslContext = SSLContext . getInstance ( " TLS " ) ; // Get the default trust manager T r u st M a n a g e r F a c t o r y t r u s tM a n a g e r F a c t o r y = T r u s t M an a g e r F a c t o r y . getInstance ( T r u s t Ma n a g e r F a c t o r y . g e t D e f au l t A l g o r i t h m () ) ; t r u st M a n a g e r F a c t o r y . init (( java . security . KeyStore ) null ) ; TrustManager [] trustManagers = t r us t M a n a g e r F a c t o r y . getTrustManagers () ; // Initialize with the default trust manager sslContext . init ( null , trustManagers , new java . security . SecureRandom () ) ; // Create and return the SSL socket return ( SSLSocket ) sslContext . getSocketFactory () . createSocket ( host , port ) ; } catch ( N o S u c h A l g o r i t h m E x c e p t i o n | K e y M a n a g e m e n t E x c e p t i o n | Key Sto re Exc ept ion | IOException e ) { e . printStackTrace () ; return null ; }}

Listing 2: JSSE code sample (AI-assisted) with M1 (no TLS version) and M3. 7

Figure 5: Misuse types and their rates across code samples; for M5, the analysis is based only on implementations that include token storage.

Figure 6: Occurrence rates of M1, M2, and M9 across backgrounds. Other misuses were observed among participants of all backgrounds, regardless of Copilot use. 4.2.2

Google OAuth.

7 8

Our study identified 6 critical misuses of OAuth, with all occurring in the control condition and 5 in the AIassisted implementations.

9 10 11

M4: Local storage of application secrets. Application secrets used to authenticate with the Service Provider (SP), Google in this case, must be stored securely. In our study, participants either hardcoded secrets (Listing 3) or stored them in local resource files without encryption (Listing 4), enabling attackers to impersonate legitimate applications if the secrets are compromised [7, 46].

Listing 4: OAuth sample from the assisted condition with M4–8; M4 (L4), M5 (L8)

M5: Local storage of access tokens. Secure storage is critical for access tokens, as they grant access to protected user resources [35]. To keep the task manageable, participants were instructed to obtain, not store, tokens. Nonetheless, some implemented storage, and in all cases tokens were stored locally without encryption (Listings 3 and 4), exposing them to unauthorized access [7]. Although storage was not the focus of our evaluation, this misuse is noteworthy as it reflects common real-world practices, with no observable impact from Copilot. M6: Lack of the state parameter. The state parameter is essential for ensuring request authenticity and protecting user sessions against CSRF attacks. OAuth guidelines recommend generating and validating a unique state linked to the user’s session [35]. However, none of the analyzed programs—whether from the control or assisted condition—implemented this security measure. M7: Lack of SP authentication. OAuth transactions require mutual authentication between applications and SPs [47]. Yet, none of the analyzed programs from either group implemented SP authentication. M8: Lack of PKCE parameters for authorization code grant. OAuth security is highly impacted

1 private 2 private 3 private

static String CLIENT_ID = " 3430 ... " ; static String CLIENT_SECRET = "GOCS ..."; static JsonFactory JSON_FACTORY = JacksonFactory . g et D ef a ul t In st a nc e () ; 4 // Authorize using OAuth 2.0 with provided scopes 5 private static Credential authorize ( Collection < String > scopes ) throws Exception { 6 G o o g l e A u t h o r i z a t i o n C o d e F l o w flow = new G o o g l e A u t h o r i z a t i o n C o d e F l o w . Builder ( new NetHttpTransport(), JSON_FACTORY , CLIENT_ID , CLIENT_SECRET , scopes ) . s e t D a t a S to r e F a c t o r y ( new FileDataStoreFactory(new File("tokens"))) . setAccessType ( " offline " ) . se tAp pro val Pro mpt ( " force " ) . build () ; 7 L o ca l S e r v e r R e c e i v e r receiver = new L o c al S e r v e r R e c e i v e r . Builder () . setPort (8080) . build () ; 8 Credential credential = new A u t h o r i z a t i o n C o d e I n s t a l l e d A p p ( flow , receiver ) . authorize ( " user " ) ; 9 return credential ; }

Listing 3: OAuth sample (control group) with M4-9; M4 (L2), M5 (L6), M9 (L6) 1 private 2 3 4 5 6

// Build flow and trigger user authorization request . G o o g l e A u t h o r i z a t i o n C o d e F l o w flow = new G o o g l e A u t h o r i z a t i o n C o d e F l o w . Builder ( G o o g l e N e t H t t p T r a n s p o r t . n e w T r u st e d T r a n s p o r t () , JSON_FACTORY , clientSecrets , List . of ( scopes ) ) . s e t D a t aS t o r e F a c t o r y ( new FileDataStoreFactory(new File("tokens"))) . setAccessType ( " offline " ) . build () ; L o ca l S e r v e r R e c e i v e r receiver = new L o c al S e r v e r R e c e i v e r . Builder () . setPort (8888) . build () ; return new A u t h o r i z a t i o n C o d e I n s t a l l e d A p p ( flow , receiver ) . authorize ( " user " ) ; }

static Credential authorize ( String [] scopes ) throws Exception { JsonFactory JSON_FACTORY = JacksonFactory . g et D ef a ul t In st a nc e () ; // Load client secrets . InputStream in = Main . class . g e t R e s ou r c e A s S t r e a m ( "/client_secrets.json") ; if ( in == null ) throw new F i l e N o t F o u n d E x c e p t i o n ( " Resource not found : / client_secrets . json " ) ; G o og l e C l i e n t S e c r e t s clientSecrets = G o o g l e Cl i e n t S e c r e t s . load ( JSON_FACTORY , new I npu tSt rea mRe ade r ( in ) ) ;

8

by the chosen grant type. All implementations used the authorization code grant, a generally secure type, but it remains vulnerable to code interception attacks [48]. Current best practices recommend the authorization code flow with Proof Key for Code Exchange (PKCE) to ensure that the requesting application is the same one that initially requested it [49]. However, no implementations in our study included PKCE. M9: Inadequate transmission protection. Ensuring secure communication throughout the OAuth process is essential for its security. However, two participants in the control condition employed HTTP without SSL protection, thereby compromising transmission security (one example shown in Listing 3). In contrast, all other participants used SSL/TLS for encrypted communication during OAuth transactions. Fig. 6 shows the distribution of M9 across background levels, though the small number of cases limits reliable interpretation of trends.

Participant C9, after successfully generating a functionally correct implementation of the OAuth task, prompted Copilot to “review the authorize method from a security perspective”. Copilot responded with several security recommendations like “Use secure storage mechanisms for storing credentials, such as encrypted files or secure vault services”, which could help address the misuse related to M5, local storage of access tokens. However, despite showing security awareness, the participant left the task at this point without addressing the existing misuses, including M5. Following OpenAI’s prompting guidelines, Participant D6 assigned Copilot a predefined role: “You are a tech architect and security expert. You need to implement the createSSLSocket method . . . ”. Despite being assigned the role of a security expert, the initial response by Copilot contained all three JSSE misuses, including M2, trusting all certificates. However, the response also provided a relevant security recommendation: “Make sure to replace the trust manager with a proper implementation for production use to ensure secure communication”. Interestingly, the participant followed up with another prompt asking Copilot to “double check and make changes if required from a security pov ”. In response, Copilot corrected M2 with a more secure setup using the default trust manager for proper certificate validation. This interaction suggests that Copilot can help improve code security when explicitly prompted to perform security review and refinement. To further examine the effect of such prompts, we analyzed chat sessions from 38 participants whose Copilot interactions were still available. For each session, we appended a follow-up prompt to the end of the chat history, asking Copilot to review the code for security issues and improve it accordingly. We then examined the resulting modifications and recommendations. In designing this follow-up prompt, we considered two alternatives: (i) D6’s original phrasing, and (ii) a prompt we designed to more explicitly emphasize security best practices: “Please review the code and improve it based on security best practices”. We evaluated both prompts on a code sample containing M2, trusting all certificates. In both cases, Copilot successfully corrected the misuse; however, our designed prompt produced more comprehensive security guidance. We therefore adopted this prompt in the subsequent analysis. The follow-up prompt proved effective in addressing the identified JSSE misuses. For M1, insecure standard, Copilot corrected 13 of the 14 instances. In the remaining case, although no fix was applied, Copilot still provided a relevant recommendation: “Ensure the application uses a secure version of TLS ”. M2 appeared only once in the AI-assisted condition, and Copilot corrected it successfully. For M3, Copilot corrected 42% of instances and provided relevant recommendations in 45% of the remaining cases. For example, in one response it advised, “While not implemented here, you should ensure the hostname matches the server’s certificate”, and in another, “Ensure the hostname of the server matches the certificate to prevent man-in-the-middle attacks”. Fig. 7 compares the rates of JSSE misuses before and after

Remark 2 • No implementation was fully secure, and Copilot had no significant impact on misuse types. • 3 JSSE misuses were found; Copilot slightly reduced the frequency of two (M1–2). • 6 OAuth misuses were found: 5 (M4–8) in both groups and 1 (M9) only in the control group (2×).

4.3

RQ3: Security Awareness Analysis

In this section, we address RQ3 by analyzing participants’ security awareness through their prompts and self-assessments, as detailed below. Prompt Analysis. We examined the language of participants’ prompts to assess whether they explicitly raised security-related concerns when interacting with Copilot. Although participants had been informed that their code would be evaluated for both functionality and security, only seven participants made references to security in their prompts. In five of these cases, the security-related phrasing was directly copied from the JSSE task description. For example, participant A9 prompted Copilot with: “Give me the body of the createSSLSocket method in the code. The method needs to create and return a properly configured SSL/TLS socket to securely connect to the server ”. The second sentence, which highlights a secure connection, is a verbatim excerpt from the task description. Code outputs in response to such prompts included one or two of the three JSSE misuses targeted in our analysis. Beyond these cases, where security-related language has been included either intentionally or unintentionally, two participants, C9 with high and D6 with very high self-rated security familiarity, demonstrated clear and deliberate consideration of security in their prompts. This suggests that developers with stronger security backgrounds may be more inclined to raise security concerns, though the small number of such cases prevents drawing meaningful conclusions.

9

5

applying the follow-up prompt. In response to our follow-up prompt for the OAuth task, Copilot consistently recommended securing sensitive data such as secrets and tokens. For instance, it advised: “Avoid hardcoding sensitive data like client secrets in files”, addressing M4, and “If sensitive tokens are stored in the DATA_STORE_DIR, ensure they are encrypted to prevent unauthorized access”, relevant to M5. However, Copilot could not correct M4 (local storage of client secrets), as the secret had already been embedded by the developer, either hardcoded or locally stored. For M5 (local storage of access tokens), Copilot suggested hidden files or restrictive file permissions. While these practices reflect a degree of security awareness, they remain inadequate and potentially exploitable, especially if an attacker gains elevated privileges (e.g., root access). Copilot did not address or offer recommendations for other OAuth misuses, including lack of a state parameter, lack of SP authentication, and lack of PKCE parameters.

Discussion

This section discusses key findings and implications for developers and researchers.

5.1

Insights into AI-assisted Security API Use

Our study shows that Copilot can improve productivity and help developers produce functionally correct code, particularly for more complex security API tasks. This aligns with prior work reporting benefits of AI code assistants for productivity and functional quality [50–54]. However, Copilot’s impact on security was limited. It produced only a slight, statistically non-significant reduction in certain security API misuses. A key insight from our study is that insecure outcomes were not only a model-output problem, but also a developer-awareness problem. Even though participants were informed that their solutions would be evaluated for security, very few explicitly raised security concerns when engaging with Copilot, and many did not recognize that their final implementations remained insecure. This suggests that persistent security API misuses in AI-assisted development cannot be explained solely by model limitations; they also reflect limited developer security awareness during both code generation and evaluation. Our findings are broadly consistent with Sandoval et al. [19] and Asare et al. [24], who likewise found no significant security improvement from AI assistance. However, our results further suggest that security-oriented prompting can sometimes help Copilot mitigate misuses, especially in the simpler JSSE task. In contrast, Copilot showed limited awareness of several common OAuth misuses even when explicitly prompted, likely reflecting the greater complexity of OAuth flows and the prevalence of insecure patterns in public code. Compared with Mousavi et al. [25], who reported a 70% misuse rate in ChatGPT-generated security API code, our study observed a 100% misuse rate across the evaluated tasks. While direct comparison is limited by differences in task design, their OAuth task also showed a 100% misuse rate, consistent with our results. Importantly, the same overall misuse rate was observed in our control condition, suggesting that Copilot did not introduce additional security risk relative to unaided development in our setting. At the same time, secure use of security APIs remained difficult in both conditions, particularly when developers did not engage with the task in a sufficiently security-aware manner.

Self-Assessment Analysis. The self-assessments indicate that many participants did not recognize that their implementations remained insecure. Although none of the implementations were fully secure across either task or condition, most participants still expressed confidence in the security of their code. Fig. 8 shows the distribution of participants’ agreement with the statement “I believe that I solved this task securely” across tasks and conditions. For the JSSE task, only three participants in the control condition and one participant in the AI-assisted condition explicitly acknowledged that their solution was insecure. Similarly, for the OAuth task, only five participants in the control condition and two participants in the AI-assisted condition identified their code as insecure. These results reveal a clear mismatch between participants’ security self-assessments and the actual security outcomes of their implementations. This mismatch was observed across both tasks and conditions, suggesting limited awareness of important security requirements even when participants had been informed that their solutions would be evaluated for security. Although fewer participants in the Copilot condition explicitly recognized insecurity, we found no statistically significant effect of Copilot on these selfassessments. When examined by participant background, lower programming experience was associated with lower belief accuracy regarding security, although this trend was not statistically significant. Remark 3 • Participants showed limited security awareness: only two participants explicitly raised security concerns in their prompts, and many did not recognize that their final implementations remained insecure. • Prompting Copilot for security reviews and refinements helped generate fixes or recommendations for certain misuses; however, it lacked awareness of other misuses, particularly in the context of the more complex task.

5.2

Implications for Developers

Our study highlights three key takeaways to guide effective use of AI assistants: 1 Be explicit about security in prompts. Copilot is more likely to generate secure code or offer meaningful security recommendations when prompts contain clear and specific security-related instructions. For example, asking Copilot to “review the code and improve it based on security best practices” can, in some cases, lead to

10

Figure 7: JSSE misuse rates before and after security-oriented prompting.

Figure 8: Distribution of participants’ agreement with the statement “I believe that I solved this task securely”, across tasks and conditions (5-point Likert scale). security improvements or recommendations. Developers should therefore consider explicitly incorporating security considerations into their natural-language interactions with AI tools. 2 AI assistance is not a substitute for security verification. Blind reliance on AI-generated code can introduce serious vulnerabilities into production systems and undermine overall software security. Although Copilot may support certain security best practices when explicitly prompted, it remains far from generating fully secure solutions. Developers must therefore remain vigilant, critically assess AI-generated code, and apply appropriate security measures, especially when handling sensitive data and working with security APIs. 3 Security expertise and continuous learning remain essential. Effective use of AI coding assistants in secure development requires a strong foundation in security principles, reinforced through ongoing security education and hands-on training. Developers must keep pace with evolving standards and best practices, including awareness of deprecated APIs, insecure cryptographic algorithms, and insecure patterns that are no longer recommended, as AI tools may continue to suggest them based on outdated training data.

5.3

2 Advancing effective misuse detection tools. Despite the growing availability of static analysis and program repair tools, existing solutions remain insufficient for detecting and correcting security API misuses [1]. Most tools exhibit limited support for a broad range of security libraries, are prone to high false-positive and false-negative rates, and often provide generic or contextinsensitive recommendations. Furthermore, many are restricted to specific programming languages or fail to scale effectively to real-world projects [55]. These limitations underscore the need for more accurate and contextaware tools that can support developers in identifying and repairing security API misuses—particularly before integrating AI-generated code into software systems. 3 Improving API usability. The default trust manager in JSSE offers an important security advantage by automatically validating certificates against the system trust store. In our study, this secure-by-default feature effectively prevented the common misuse of trusting all certificates in both the experimental and control groups. In contrast, hostname verification was frequently misused because it requires explicit developer awareness and manual implementation, which many participants struggled with. This contrast highlights an important implication: security APIs should be designed with usability as a core principle. Developers, especially those without deep security expertise, should be able to use these APIs securely. Addressing this challenge requires research to identify usability barriers in current APIs and develop practical, developer-friendly solutions that support secure integration in real-world software systems. 4 Enhancing security of AI-generated code. When prompted to review code for security, Copilot could identify and suggest fixes for certain API misuses. However, it failed to detect and address other misuses, particularly for the more complex task in our study. This limitation aligns with the widespread presence of such misuses in real-world code repositories, likely inherited by LLMs trained on these datasets [25]. Recent research has investigated a range of techniques to improve LLMs’ understanding and handling of software security. These

Implications for Researchers

Our findings highlight several key avenues for advancing research at the intersection of AI-assisted software development and security: 1 Supporting security-aware engagement with AI tools. There is a pressing need to equip developers with the skills and resources required to engage with AI assistants in a security-aware manner when working with security APIs. Many participants in our study neglected security considerations in both their interactions with Copilot and their evaluation of the resulting code, despite being informed that security would be assessed. This highlights the need for practical interventions—such as security training modules, real-time feedback systems, and prompt-design support—that help developers both raise security concerns during interaction and critically assess the security of generated code. 11

include fine-tuning on security-specific datasets [56], prompt engineering [57], incorporating feedback from dynamic testing and static analysis [58], reinforcement learning with human or automated feedback [59], the integration of security constraints during decoding [60], adversarial training [61], and modular prefix-based techniques with prompting [62]. Future research should extend these efforts to the domain of security APIs. In particular, curating high-quality, security-focused datasets that accurately reflect the correct use of security APIs is essential to enable targeted fine-tuning. Moreover, advancing static analysis techniques to more reliably detect and repair security API misuse is a critical step; the feedback they provide can be integrated into the generation process to better guide LLMs toward secure API usage. Additionally, incorporating API specifications, using methods such as Retrieval-Augmented Generation (RAG), offers a promising direction for enriching LLM outputs with relevant security context. Finally, it is imperative to develop mechanisms that ensure LLMs remain aligned with evolving API specifications and emerging security best practices over time.

6

this bias, all participants were required to review a brief tutorial on Copilot and its use in VS Code prior to their sessions. They were also allowed to revisit the tutorial during the study if needed. Evaluation Validity. Our results might be influenced by the methodology used for misuse detection. Existing automated tools for identifying security API misuses suffer from high false positive and negative rates and lack comprehensive coverage across all security APIs. Given these limitations, the first author and a security researcher from our lab, each with four years of security experience, independently reviewed all submissions (κ = 0.97), and disagreements were resolved through discussion. Additionally, our results might be affected by participants’ backgrounds, such as prior familiarity with the specific security APIs used in the tasks. When evaluating functionality, we observed a difference in the OAuth task’s outcomes, where more participants in the experimental condition produced functional code compared to those in the control condition. Our statistical analysis confirmed that this difference remained significant even after controlling for background. Regarding security, only minor variations in a few misuse types were observed across conditions and backgrounds, but these were not large enough to yield meaningful conclusions.

Threats to Validity

This section discusses potential limitations that may affect the reliability and generalizability of our findings, and the steps taken to mitigate them.

7

Generalizability. Our findings may not fully generalize to AI code assistants beyond Copilot or its future evolutions. Nevertheless, Copilot’s widespread adoption among developers makes it a practical and relevant choice for evaluating real-world development workflows. Additionally, the study focuses on the Java programming language and two specific security APIs: JSSE and OAuth. These choices were driven by the need to maintain task feasibility within a two-hour session. Although this scope limits applicability to other languages or security APIs, the prevalence of Java (particularly in enterprise and Android development) and the critical role of the selected APIs support the relevance of our findings within common development contexts. Finally, we recruited developers through Freelancer, which may not fully represent the broader developer population. Nevertheless, Freelancer has been recommended by prior research for recruiting participants in security-oriented studies [36], and has also been used in several user studies involving security programming tasks [37–40], supporting its suitability for our study.

Conclusion

This study presents the first empirical investigation into how AI code assistants influence the use of security APIs in real-world development settings. In a withinsubject study with 44 professional developers, we compared outcomes on security API programming tasks completed with and without GitHub Copilot assistance. While Copilot improved participants’ ability to produce functionally correct code, particularly for more complex tasks, its support for secure API usage remained limited. No participant produced a fully secure implementation, regardless of AI assistance or background, highlighting the ongoing challenges of secure software development. Our analysis further shows that security challenges in AI-assisted development are not only a model-output problem, but also a developer-awareness problem. Despite being informed that security would be evaluated, only a few participants explicitly considered security when engaging with Copilot, and many did not recognize that their final implementations remained insecure. Although targeted security-focused prompts could sometimes help Copilot generate more secure revisions or recommendations, Copilot did not reliably identify or mitigate all types of misuses. These findings underscore the need for both improved tooling and greater developer awareness to enhance the security of AI-assisted development. Future research should explore ways to integrate explicit security guidance into code assistants and design mechanisms that foster a proactive, security-conscious mindset among developers.

Study Design Validity. All participants completed two tasks and experienced the control and experimental conditions, which could introduce learning or fatigue effects. To mitigate this, we counterbalanced both Copilot usage and task order across four groups. Participants in the control condition were instructed not to use any AIbased code assistants; however, we could not guarantee full compliance. To address this, we monitored sessions via screen recording and excluded non-compliant cases from the final analysis. Another potential limitation was varying levels of familiarity with Copilot, which could affect participants’ ability to use it effectively. To reduce 12

References

2016 IEEE Symposium on Security and Privacy (SP), pages 289–305. IEEE, 2016.

[1] Zahra Mousavi, Chadni Islam, Muhammad Ali Babar, Alsharif Abuadbba, and Kristen Moore. Detecting misuse of security APIs: A systematic review. ACM Computing Surveys, 57(12):1–39, 2025.

[11] Yasemin Acar, Christian Stransky, Dominik Wermke, Charles Weir, Michelle L Mazurek, and Sascha Fahl. Developers need support, too: A survey of security advice for software developers. In 2017 IEEE Cybersecurity Development (SecDev), pages 22–26. IEEE, 2017.

[2] Sascha Fahl, Marian Harbach, Thomas Muders, Lars Baumgärtner, Bernd Freisleben, and Matthew Smith. Why Eve and Mallory love Android: An analysis of Android SSL (in) security. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 50–61, 2012.

[12] Ethan Shernan, Henry Carter, Dave Tian, Patrick Traynor, and Kevin Butler. More guidelines than rules: CSRF vulnerabilities from noncompliant OAuth 2.0 implementations. In Detection of Intrusions and Malware, and Vulnerability Assessment: 12th International Conference, DIMVA 2015, Milan, Italy, July 9-10, 2015, Proceedings 12, pages 239–260. Springer, 2015.

[3] Martin Georgiev, Subodh Iyengar, Suman Jana, Rishita Anubhai, Dan Boneh, and Vitaly Shmatikov. The most dangerous code in the world: validating ssl certificates in non-browser software. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 38–49, 2012.

[13] Jan H Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, Heather Richter Lipford, et al. Using ai assistants in software development: A qualitative study on security practices and concerns. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 2726–2740, 2024.

[4] Manuel Egele, David Brumley, Yanick Fratantonio, and Christopher Kruegel. An empirical study of cryptographic misuse in android applications. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pages 73–84, 2013. [5] Sazzadur Rahaman, Ya Xiao, Sharmin Afrose, Fahad Shaon, Ke Tian, Miles Frantz, Murat Kantarcioglu, and Danfeng Yao. Cryptoguard: High precision detection of cryptographic vulnerabilities in massive-sized java projects. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2455–2472, 2019.

[14] Melvin Hipolito. Github copilot users surpass 20 million as ai tools surge in demand, Jul 2025. Accessed January 29, 2026. [15] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE, 2022.

[6] Antonio Bianchi, Yanick Fratantonio, Aravind Machiry, Christopher Kruegel, Giovanni Vigna, Simon Pak Ho Chung, and Wenke Lee. Broken Fingers: On the Usage of the Fingerprint API in Android. In NDSS, 2018. [7] Tamjid Al Rahat, Yu Feng, and Yuan Tian. Oauthlint: An empirical study on oauth bugs in android applications. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 293–304. IEEE, 2019.

[16] Mohammed Latif Siddiq, Shafayat H Majumder, Maisha R Mim, Sourov Jajodia, and Joanna CS Santos. An empirical study of code smells in transformer-based code generation techniques. In 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 71–82. IEEE, 2022.

[8] Stefan Krüger, Johannes Späth, Karim Ali, Eric Bodden, and Mira Mezini. CrySL: An extensible approach to validating the correct usage of cryptographic APIs. IEEE Transactions on Software Engineering, 47(11):2382–2400, 2019.

[17] Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. How secure is code generated by chatgpt? In 2023 IEEE international conference on systems, man, and cybernetics (SMC), pages 2445–2451. IEEE, 2023.

[9] Sarah Nadi, Stefan Krüger, Mira Mezini, and Eric Bodden. Jumping through hoops: Why do java developers struggle with cryptography apis? In Proceedings of the 38th International Conference on Software Engineering, pages 935–946, 2016.

[18] Owura Asare, Meiyappan Nagappan, and N Asokan. Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering, 28(6):1–24, 2023. [19] Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan DolanGavitt. Lost at c: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium (USENIX Security 23), pages 2205–2222, 2023.

[10] Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L Mazurek, and Christian Stransky. You get where you’re looking for: The impact of information sources on code security. In

13

[20] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with AI assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2785–2799, 2023.

Intelligence & AIOps (AIOps), pages 8–13. IEEE, 2023. [32] Ali Kazemi Arani, Triet Huynh Minh Le, Mansooreh Zahedi, and M Ali Babar. Systematic literature review on application of learning-based approaches in continuous integration. IEEE Access, 12:135419–135450, 2024.

[21] Madhav Nair, Rajat Sadhukhan, and Debdeep Mukhopadhyay. Generating secure hardware using ChatGPT resistant to CWEs. Cryptology ePrint Archive, 2023.

[33] Mike Melanson. Don’t Call It a Comeback: Why Java Is Still Champ, 2022. Accessed May 30, 2025.

[22] Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering, 2024.

[34] Pierre Carbonnelle. Pypl popularity of programming language, 2023. Accessed May 30, 2025. [35] Dick Hardt. The OAuth 2.0 Authorization Framework. RFC 6749, October 2012. Accessed May 26, 2025.

[23] M Mehdi Kholoosi, M Ali Babar, and Roland Croft. A qualitative study on using chatgpt for software security: Perception vs. practicality. In 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pages 107–117. IEEE, 2024.

[36] Harjot Kaur, Sabrina Amft, Daniel Votipka, Yasemin Acar, and Sascha Fahl. Where to recruit for security development studies: Comparing six software developer samples. In 31st USENIX Security Symposium (USENIX Security 22), pages 4041–4058, 2022.

[24] Owura Asare, Meiyappan Nagappan, and N Asokan. A user-centered security evaluation of copilot. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–11, 2024.

[37] Alena Naiakshina, Anastasia Danilova, Eva Gerlitz, and Matthew Smith. On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2020.

[25] Zahra Mousavi, Chadni Islam, Kristen Moore, Alsharif Abuadbba, and M Ali Babar. An investigation into misuse of Java security APIs by large language models. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, pages 1299–1315, 2024.

[38] Alena Naiakshina, Anastasia Danilova, Eva Gerlitz, Emanuel Von Zezschwitz, and Matthew Smith. "if you want, i can store the encrypted password" a password-storage field study with freelance developers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2019.

[26] Yujia Fu, Peng Liang, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and Jinfo Chen. Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study. ACM Transactions on Software Engineering and Methodology, 2025.

[39] Lisa Geierhaas, Anna-Marie Ortloff, Matthew Smith, and Alena Naiakshina. {Let’s} hash: Helping developers with password security. In Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022), pages 503–522, 2022.

[27] Amirali Sajadi, Binh Le, Anh Nguyen, Kostadin Damevski, and Preetha Chatterjee. Do LLMs consider security? an empirical study on responses to programming questions. Empirical Software Engineering, 30(3):101, 2025.

[40] Anastasia Danilova, Alena Naiakshina, Johanna Deuter, and Matthew Smith. Replication: On the [28] Softonic. Microsoft github copilot: Statistics and ecological validity of online security developer studtrends, August 2024. Accessed May 3, 2025. ies: Exploring deception in a {Password-Storage} [29] CIODIVE. Github copilot drives revstudy with freelancers. In Sixteenth Symposium on enue growth amid subscriber base expanUsable Privacy and Security (SOUPS 2020), pages sion. https://www.ciodive.com/news/ 165–183, 2020. github-copilot-subscriber-count-revenue-growth/ [41] Jonathan Lazar, Jinjuan Heidi Feng, and Harry 706201/, 2024. Accessed May 3, 2025. Hochheiser. Research methods in human-computer [30] anonym. User study material. https://figshare. interaction. Morgan Kaufmann, 2017. com/s/85124bf8ee443ff34dc3. [42] David W Hosmer Jr, Stanley Lemeshow, and Rod[31] Ali Kazemi Arani, Mansooreh Zahedi, Triet ney X Sturdivant. Applied logistic regression. John Huynh Minh Le, and Muhammad Ali Babar. Sok: Wiley & Sons, 2013. Machine learning for continuous integration. In 2023 IEEE/ACM International Workshop on Cloud 14

[43] Richard Barnes, Martin Thomson, Alfredo Pironti, and Adam Langley. Deprecating secure sockets layer version 3.0, 2015.

Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 21–29, 2022.

[44] K. Moriarty and S. Farrell. Deprecating TLSv1.0 and TLSv1.1, 2021. Accessed June 10, 2023.

[55] Ying Zhang, Md Mahir Asef Kabir, Ya Xiao, Danfeng Yao, and Na Meng. Automatic detection of Java cryptographic API misuses: Are we there yet? IEEE Transactions on Software Engineering, 49(1):288–303, 2022.

[45] Sean Turner and Tim Polk. Prohibiting secure sockets layer (SSL) version 2.0. Technical report, 2011.

[56] Adel ElZemity, Budi Arief, and Shujun Li. CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data, 2025.

[46] Eric Y Chen, Yutong Pei, Shuo Chen, Yuan Tian, Robert Kotcher, and Patrick Tague. Oauth demystified for mobile application developers. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, pages 892– 903, 2014.

[57] Marc Bruni, Fabio Gabrielli, Mohammad Ghafari, and Martin Kropp. Benchmarking prompt engineering techniques for secure code generation with gpt models. arXiv preprint arXiv:2502.06039, 2025.

[47] Hui Wang, Yuanyuan Zhang, Juanru Li, Hui Liu, Wenbo Yang, Bodong Li, and Dawu Gu. Vulnerability assessment of oauth implementations in android applications. In Proceedings of the 31st annual computer security applications conference, pages 61–70, 2015.

[58] Richard A Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C Cordeiro, and Norbert Tihanyi. Castle: Benchmarking dataset for static code analyzers and llms towards cwe detection. arXiv preprint arXiv:2503.09433, 2025.

[48] Amir Sharif, Roberto Carbone, Giada Sciarretta, and Silvio Ranise. Best current practices for oauth/oidc native apps: A study of their adoption in popular providers and top-ranked android clients. Journal of Information Security and Applications, 65:103097, 2022.

[59] Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, ChiMin Chan, Sirui Han, Yike Guo, and Yaodong Yang. Safe RLHF-V: Safe Reinforcement Learning from Human Feedback in Multimodal Large Language Models, 2025.

[49] Nat Sakimura, John Bradley, and Naveen Agarwal. Proof key for code exchange by OAuth public clients. Technical report, Internet Engineering Task Force (IETF), 2015.

[60] Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. Constrained decoding for secure code generation. arXiv preprint arXiv:2405.00218, 2024.

[50] Maxim Tabachnyk, Stoyan Nikolov, et al. Mlenhanced code completion improves developer productivity. Google Research Blog. July, 26, 2022.

[61] Liming Lu, Shuchao Pang, Siyuan Liang, Haotian Zhu, Xiyu Zeng, Aishan Liu, Yunhuai Liu, and Yongbin Zhou. Adversarial training for multimodal large language models against jailbreak attacks, 2025.

[51] Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code!= big vocabulary: Open-vocabulary models for source code. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 1073–1085, 2020.

[62] Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1865–1879, 2023.

[52] Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, pages 1–7, 2022. [53] Saki Imai. Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 319–321, 2022. [54] Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In 15