Open-Source Intelligence for Code Provenance and the Security Patterns that Separate Human and Large-Language-Model Implementations of Common Programming Tasks Mohammadreza Rashidi
arXiv:2607.12524v1 [cs.CR] 14 Jul 2026
Department of Computer Science AI and Media Analysis Lab Berlin, Germany [email protected]
Abstract—Developers now draw code from two very different sources, the accumulated human answers on sites such as Stack Overflow and the output of large language models. We ask two questions about that split. First, can the provenance of a code snippet be recovered from the code itself, and second, do the two sources differ in the security patterns they adopt for the same task. Using only open sources, a public gateway of open-weight language models and the public Stack Overflow API, we build a fully reproducible pipeline that collects real implementations of 31 security-sensitive programming tasks, among them OAuth with PKCE, JWT verification, password hashing, and SQL access, from 9 language models and from human answers, and scores every sample with deterministic security and style detectors. On 528 real samples we train a cross-validated classifier that recovers human versus model provenance with 93 percent accuracy against a 78 percent baseline, and a 7-way classifier that attributes a sample to the specific model that wrote it at 48 percent. We then report where the sources diverge on security, which patterns models adopt more often than the human corpus and which they inherit from it. Running the same tasks in Python, JavaScript, and Go, we find the security divergence holds in every language while the provenance boundary is partly language-specific and does not transfer symmetrically between them. A further case study on vulnerability repair, in which the models are handed insecure code and asked to fix it, finds a 77 percent repair rate but a recurring partial-fix failure in which the model removes the insecure pattern without adding the correct defense. The pipeline is data driven, so any new task or language is added as a single specification entry, and a fail-closed checker re-derives every number in this paper from the stored data. Index Terms—code provenance, large language models, software security, authorship attribution, open-source intelligence, secure coding
1. Introduction A developer who needs to implement a login flow, hash a password, or set up an OAuth exchange has, for over a
decade, most often consulted a human answer on a site such as Stack Overflow. That practice has a measurable security cost. Copying from Stack Overflow has been shown to move insecure patterns into shipped applications [1], and the information source a developer consults measurably changes the security of the code they write [2]. The practice is now being displaced by large language models, which developers query in the editor rather than the browser, and their output carries its own security cost. Language-model code assistants produce insecure code at a meaningful rate [3], and controlled user studies find that developers with an assistant write less secure code while believing the opposite [4], [5]. This paper studies the two sources side by side. We treat the question as one of provenance and pattern rather than of overall quality. Two questions drive the work. RQ1, provenance. Given only a code snippet for a common task, can we tell whether it was written by a human on Stack Overflow or generated by a language model, and can we tell which model. Code stylometry has long shown that human authors leave recoverable fingerprints [6], [7]. We ask whether the human-versus-machine boundary, and the boundary between models, is just as recoverable from lightweight, interpretable features. RQ2, security divergence. For the same task, do the two sources adopt the same security patterns. Does a model reach for PKCE, a bound query, or a strong password hash more or less often than the human corpus, and does it carry the human corpus’s insecure habits forward or leave them behind. We answer both with open sources only, which is what makes the study an exercise in open-source intelligence. The human side is the public Stack Overflow API. The model side is a local gateway that exposes open-weight models through an OpenAI-compatible interface, from which we use 9 models spanning several families and sizes. For 31 security-sensitive tasks we collect real implementations from both sides, 528 samples in total, and score each with two families of deterministic detectors, per-task security-pattern checks and language-agnostic style metrics. Nothing is simulated. A task with no sample from a given model simply has no sample, and every generation carries the model, latency,
and token counts that produced it. Our contributions are the following. •
•
•
•
A reproducible, open-source pipeline for crossprovenance code analysis. Every task is a single specification entry carrying its prompt, its Stack Overflow query, and its security-pattern detectors, so the study generalises to any subject without new code. A provenance-attribution result. A cross-validated classifier separates human from model code at 93 percent against a 78 percent baseline, and attributes a model-written sample to its specific model of origin at 48 percent in a 7-way task. A security-divergence analysis. We report, per task and per pattern, how often the human corpus and the models adopt each secure and insecure practice, and which insecure human patterns the models reproduce. A released dataset and a fail-closed numeric checker that re-derives every figure in this paper from the stored samples.
2. Background and Related Work Insecure code from human sources. A line of security research has measured the cost of the copy-and-paste development style. Stack Overflow contains both secure and insecure answers, and the insecure ones propagate into real software [1]. The information source a developer is steered to, official documentation versus a question-and-answer site, changes the security of the resulting code [2]. Our human corpus is drawn from the same source these studies examined, and we treat the age of an answer as a first-order confound, since a highly-voted answer written years ago predates the defaults that are now considered secure. Insecure code from language models. As models trained on code became capable of solving programming tasks [8], the same security questions were asked of them. An assessment of GitHub Copilot found a substantial fraction of security-relevant completions were vulnerable [3]. User studies then showed the effect on developers, who write less secure code with an assistant and misjudge its safety [4], [5]. These studies mostly evaluate one commercial assistant. We instead compare many open-weight models to each other and to the human baseline on identical tasks, and we organise the tasks around recurring web-application weakness classes in the spirit of the OWASP Top Ten [9]. Code authorship and provenance. Code stylometry recovers the human author of a program from stylistic features [6], and the approach scales to thousands of authors and across languages [7]. The naturalness of software, its predictability under a language model [10], is the property such attribution exploits. Recent work extends stylometry directly to the machine-authorship question, recognising AIwritten programs with multilingual code stylometry [11]. Our provenance result is close in spirit, and we add two elements that work does not address, a per-model attribution
rather than a binary human-or-machine label, and a securitydivergence analysis on the same samples. We apply the same idea to a coarser but timely boundary, human versus model and model versus model, using interpretable features rather than a deep model, so that the attribution result comes with an explanation. Naturalness and machine-generated code detection. The property that makes code authorship recoverable is that code is repetitive and predictable, its naturalness under a statistical language model [10]. Human authors and, as we find, individual models occupy distinguishable regions of that predictable space. A growing body of work asks the specific question of whether machine-generated code can be told apart from human code, motivated by plagiarism, academic integrity, and supply-chain provenance. We contribute a lightweight, interpretable point on that spectrum, using a handful of surface features rather than a learned detector, and we pair the detection question with a security question that detection work usually leaves aside. Position of this work. We connect the two literatures. The security studies ask whether a source is safe, the stylometry studies ask who wrote a piece of code. We ask both questions about the same samples, so that a security difference between sources can be read alongside the features that make the sources distinguishable in the first place. Two further choices set the work apart from its neighbours. We compare many open-weight models rather than a single commercial assistant, which lets us separate what is common to models from what is specific to one, and we run the same tasks in several languages, which lets us separate what is a property of the source from what is a property of a language.
3. Threat Model and Motivation The provenance question is not academic. Two settings make it concrete. First, in review and audit, a team that knows a change was machine-generated can route it to the checks that machine code most often fails, and the security-divergence profile of §8 tells the reviewer which checks those are. Second, in supply-chain and plagiarism analysis, the ability to attribute a snippet to a source, or to a specific model, is an open-source-intelligence capability that operates on the artifact alone, without access to the author’s environment. We assume the analyst has only the code, no metadata, no editor telemetry, and no account information. This is the open-source-intelligence setting. The material that feeds the study is likewise all public, the Stack Overflow API and a gateway of open-weight models, so the entire result is reproducible by a third party with no private access.
4. Data Collection Subjects. A subject is one security-sensitive programming task. Each subject is a single specification entry that carries a natural-language prompt, a Stack Overflow search
query, and a list of pattern detectors. Table 1 lists the 31 subjects, which span authentication and authorization (OAuth with PKCE, JWT verification, session cookies), data protection (password hashing, parameterized database access), input handling (file upload), transport policy (CORS), and a front-end design task (a responsive landing page). The design is deliberately data driven. Adding a subject is adding one entry, and the whole pipeline, generation, retrieval, extraction, and analysis, runs off the specification with no new code. Human corpus. For each subject we query the public Stack Overflow API with the subject’s search string, retrieve the answers to the most relevant answered questions, and keep the largest code block from each positively-scored answer. We record the answer identifier, its score, and its creation year, so the corpus is auditable and re-fetchable and so the paper can report the age of the human material. We keep only answers whose code block is substantive. Model corpus. The model side is a local gateway exposing open-weight models through an OpenAI-compatible interface. We first probe every model the gateway advertises with a trivial request and record whether it answers, is rate limited, or is unavailable, so that only models that genuinely respond enter the study. From the responding set we select 9 models chosen for diversity of family and size, listed in Table 2. For each model and subject we send the subject’s prompt verbatim, with a fixed system instruction asking for a single complete code solution, and store the returned code with the model, the resolved model name, the latency, and the token counts. Generation is repeated so that each model contributes more than one sample per subject, and the whole run is resumable and paced to respect the gateway’s rate limits. No output is edited or synthesised. The gateway throttles models independently and unpredictably, so coverage is filled by repeated resumable passes rather than forced, and a missing model-subject cell is reported as missing. Pipeline. Listing 1 states the collection and analysis pipeline in full. It is a deterministic sequence of small steps, each writing an artifact to disk, and the fail-closed checker of Appendix E re-derives every reported number from those artifacts. Adding a subject is a data change, a single specification entry rather than a code change, so the study extends to a new task without modifying the pipeline.
Listing 1. Collection and analysis pipeline. for subject in specification: # one entry per task human[subject] = stackoverflow_api(subject.query) for model in usable_models: # probed WORKS models for r in range(repeats): code = gateway.chat(model, subject.prompt) save(code, provenance=(model, latency, tokens)) for sample in human + model_samples: checks = [regex(c) for c in subject.checks] # secure ,→ /insecure style = style_metrics(sample.code) # size, ,→ structure sec = secure_frac(checks) - insecure_frac(checks) write_features(sample, checks, style, sec) attribution = cross_val(RandomForest, features, ,→ provenance) prevalence = adopt_rate(checks) by (subject, source) verify(every_macro == recomputed_from(features)) # ,→ fail closed
5. Feature Extraction Every sample, human or model, is reduced to two families of features by deterministic functions of the stored code, so that the reduction is reproducible and interpretable. Security-pattern checks. Each subject carries a list of checks, and each check is a regular expression tagged as secure or insecure. A check fires when its pattern is present in the code. For OAuth with PKCE, for example, the secure checks include the presence of a code challenge and verifier following PKCE [12], an anti-forgery state parameter, and validation of the redirect target, while an insecure check fires on a hardcoded client secret. For password hashing, a secure check fires on a strong key derivation function such as bcrypt, scrypt, argon2, or pbkdf2, and an insecure check fires on a bare md5 or sha1. From the checks we derive, for each sample, the fraction of secure patterns adopted, the fraction of insecure patterns present, and a single security score, sec =
#secure adopted #insecure present − , #secure defined #insecure defined
(1)
which lies in [−1, 1] and is higher for code that adopts the secure practices and avoids the insecure ones. Style metrics. Independently of the task, we compute language-agnostic style metrics that capture the shape of the code, its length in characters and lines, the comment density, the number of import statements, the blank-line ratio, the average line length, and indicators for the presence of functions and of error handling. These are the features that code stylometry uses, in a lightweight and interpretable form, and they carry most of the provenance signal of §7.
6. Dataset The collected corpus contains 528 real code samples, 117 from Stack Overflow and 411 generated by 9 openweight models, spread across 31 subjects. The subjects
TABLE 1: The 31 security-sensitive subjects. SO and LLM are the retained human and model sample counts.
comment ratio blank ratio n lines sec secure frac avg line len n imports has functions n chars n code lines sec insecure frac has error handling 0.00
0.02 0.04 0.06 0.08 0.10 permutation importance (human vs LLM attribution)
Figure 1: Permutation importance of each feature for humanversus-model attribution. Code shape, not security content, carries most of the provenance signal. reduce to 24 task families expressed across 5 languages, namely Python, JavaScript, Go, and a markup task in HTML, which is what makes the cross-language analysis of §12 possible. Table 1 lists the subjects with the number of security and style checks each carries and the count of human and model samples retained. Table 2 lists the models with their family, approximate size, and sample count. The model set spans several families and a wide size range, from small general models to very large mixtureof-experts systems, and includes both code-specialised and general models. Because the gateway throttles each backend independently, coverage is not uniform, and we report the true per-model counts rather than forcing an equal cell count.
7. Provenance Attribution Human versus model. We first ask whether a sample’s provenance can be recovered from its features alone. We train a random forest [13] on the style metrics of §5 together with the two aggregate security fractions, and evaluate it with stratified five-fold cross-validation against a majorityclass baseline. The classifier separates human from model code with 93 percent accuracy against a 78 percent baseline, at an F1 of 0.955 for the model class. Provenance is therefore highly recoverable from lightweight, interpretable features, without any learned representation of the code. Figure 1 reports the permutation importance of each feature for this task. The signal is dominated by the shape of the code rather than its security content. The strongest discriminators are the raw size of the sample, the presence of functions, and the number of imports, which reflects a consistent difference in how much scaffolding each source produces for the same request. The security fractions contribute, but they are secondary to style, so the human-versusmodel boundary is primarily stylistic rather than securitybased. Which model. We then restrict to the model samples and ask a harder question, which specific model wrote a given sample. A 7-way random forest over the same features reaches 48 percent accuracy against a 17 percent majority
Subject
Lang
Chk
SO
LLM
OAuth 2.0 Authorization Code flow with PKCE JWT verification for API authentication Password hashing for user storage Database query with user input Handling a user file upload CORS configuration for an API Responsive product landing page Session cookie setup OAuth 2.0 Authorization Code flow with PKCE (Node) JWT verification for API authentication (Node) Password hashing for user storage (Node) Database query with user input (Node) CORS configuration for an API (Node) Handling a user file upload (Node) Session cookie setup (Node) Password hashing for user storage (Go) JWT verification for API authentication (Go) Database query with user input (Go) CORS configuration for an API (Go) Handling a user file upload (Go) Session cookie setup (Go) OAuth 2.0 Authorization Code flow with PKCE (Go) Running a shell command with user input Rendering user input in an HTML page Fetching a URL provided by the user Loading an API key or secret in an app Serving a file by name from a directory OAuth 2.0 login with Spring Security (Java) OAuth 2.0 login with Django (Python) OAuth 2.0 login with Express and Passport (Node) Database query with user input in Spring (Java)
python
6
7
18
python
5
8
17
python
4
6
15
python
3
8
14
python python
4 3
8 8
15 15
html
6
0
13
python javascript
4 6
3 0
14 14
javascript
5
4
14
javascript
4
0
14
javascript
3
0
14
javascript
3
8
14
javascript
4
0
14
javascript
4
1
14
go
4
0
12
go
5
8
12
go
3
0
12
go
3
8
12
go
4
0
12
go go
4 6
0 0
12 12
python
3
7
12
python
3
4
12
python
3
2
12
python
2
5
12
python
2
1
12
java
6
8
12
python
6
2
12
javascript
5
3
12
java
3
8
12
TABLE 2: The 9 open-weight models used, with family, approximate size, and number of retained samples. Model
Family
Size
n
codestral mistral-large-3-675b mistral-small-4-119b nemotron-3-nano-30b nemotron-3-ultra-550b north-mini-code qwen3-coder-30b qwen3.5-397b tencent-hy3
Mistral Mistral Mistral Nemotron Nemotron North Qwen Qwen Tencent
22B 675B 119B 30B 550B mini 30B 397B MoE
68 30 62 62 60 62 1 4 62
Pattern
1.0 codestral
0.81
mistral-large-3-
true model
mistral-small-4-
0.11
nemotron-3-nano-
0.12
0.43
0.17
0.17
0.10
0.42
0.10
0.07
0.11
0.08
nemotron-3-ultra
north-mini-code
0.11
tencent-hy3
0.10
0.08
0.14
0.07
0.17
0.07
0.07
0.14
0.40
0.10
0.11
0.16
0.18
0.60
0.07
0.05
0.16
0.08
0.37
0.21
0.14
0.08
0.19
0.26
0.8
0.6
0.4
0.2
de
nthy 3 ce ten
rth -m ini no
-3u
-co
ltra
oan -3n
mo tro n ne
-4ne
mo tro n
-
mi str al sm al l
mi str al l ar ge -3
de str al
0.0 co
TABLE 3: Adoption rate of each security and style pattern in the human Stack Overflow corpus and in the model corpus, aggregated across subjects. Higher is more frequent.
predicted model
Figure 2: Row-normalised confusion matrix of the modelattribution classifier. Same-family models are confused with each other more than across families. baseline. Individual models therefore leave a recoverable fingerprint in their code, well above chance, though the boundary between models is far less sharp than the boundary between human and machine. Model attribution from code alone is a real but bounded capability, consistent with the view that models within a shared training regime converge on overlapping styles. Figure 2 shows the row-normalised confusion matrix of the model classifier. The structure is informative. Models from the same family are confused with each other more than with models from other families, and the models that produce the most elaborate scaffolding are the most cleanly separated. The confusions follow family and size rather than being uniform noise, which is what a genuine per-model fingerprint predicts.
8. Security Divergence The second question is whether the two sources adopt the same security patterns for the same task. Table 3 reports, for each detector, the adoption rate in the human corpus
alg none risk alg pinned alt text arg list auto escape constant time cmp credentials flag csp csrf disabled csrf protection debug on env or vault exp checked explicit origin extension check externalized secret flex or grid hardcoded key hardcoded secret hardcoded secret key host allowlist httponly https callback https enforced inline style input validation media query no validation risk param binding parameterized path traversal risk per user salt pkce raw html risk redirect validation safe join samesite scheme check secure filename secure flag semantic html session secret env shell true risk size limit state param string concat sql strong kdf traversal risk uses framework uses library uses orm verify signature viewport meta weak hash wildcard origin
SO adopt
LLM adopt
0.00 0.35 – 0.14 0.00 0.00 0.08 0.00 0.12 0.12 0.00 0.00 0.30 0.29 0.12 0.10 – 0.00 0.00 0.00 0.50 0.00 0.20 0.57 – 0.00 – 0.00 0.38 0.50 0.00 0.33 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 – 0.00 0.14 0.00 0.10 0.12 0.17 0.00 0.69 0.44 0.19 0.65 – 0.17 0.25
0.02 0.60 0.00 0.17 1.00 0.80 0.54 0.00 0.00 0.25 0.00 1.00 0.88 0.71 0.41 0.38 0.77 0.00 0.36 0.50 0.33 0.85 0.04 0.75 0.08 0.25 0.92 0.50 0.33 0.88 0.22 0.93 0.69 0.00 0.00 0.67 0.60 0.25 0.39 0.60 1.00 0.75 0.42 0.41 0.52 0.00 1.00 0.42 1.00 0.54 0.29 0.95 1.00 0.00 0.32
and in the model corpus, aggregated across subjects and weighted by sample count. Figure 3 plots the same contrast. Figure 4 gives a per-family, per-pattern view of the same contrast as the signed difference in adoption between the models and the human corpus, so the blue cells are patterns the models adopt more and the red cells are patterns the human corpus adopts more. The figure makes the structure of the divergence visible at a glance, a broad band of blue on the defensive patterns and a small number of red cells on the patterns where the human corpus leads. Models adopt defensive patterns more often. The clearest
raw html risk weak hash host allowlist https callback csrf disabled string concat sql param binding arg list wildcard origin inline style uses library uses orm csrf protection https enforced path traversal risk input validation scheme check alg pinned shell true risk externalized secret extension check verify signature uses framework hardcoded secret parameterized
0.0
secure filename size limit explicit origin traversal risk state param credentials flag hardcoded secret key no validation risk exp checked per user salt secure flag samesite safe join pkce session secret env flex or grid constant time cmp strong kdf httponly media query auto escape env or vault semantic html viewport meta
0.2
0.4
0.6
adoption rate
0.8
1.0
0.0
StackOverflow (human)
LLM
0.2
0.4
0.6
adoption rate
0.8
1.0
Figure 3: Per-pattern adoption, human Stack Overflow versus model, aggregated over all subjects and sorted by the modelminus-human gap. Secure defensive patterns are adopted far more often by the models, while a small number of insecure patterns, notably hardcoded secrets, are more common in model code. Patterns that neither source adopts are omitted. result is that the model corpus adopts the secure patterns at a substantially higher rate than the human corpus. The aggregate security score, the fraction of secure patterns adopted minus the fraction of insecure patterns present, is 0.40 for the models against 0.12 for the human corpus. The gap is largest on exactly the patterns that modern secure-coding guidance emphasises. The models reach for proof-key-forcode exchange in the OAuth task, a strong key-derivation function for password storage, a parameterized query for database access, signature verification and algorithm pinning and expiry checking for tokens, and the secure, http-only, and same-site flags for session cookies, all at markedly higher rates than the human answers, which frequently omit them. This is consistent with the models having absorbed
the more recent consensus on these tasks, while the highlyvoted human answers include material old enough to predate it. Where the human corpus carries insecure legacy patterns. The same table shows the human corpus retaining insecure habits that the models have largely shed. Human answers concatenate user input into SQL, use bare or fast hash functions for passwords, and configure a wildcard cross-origin policy more often than the models do. These are the classic copy-and-paste hazards, and their lower rate in model code is the mirror image of the previous finding. The exception, hardcoded secrets. The models are not uniformly safer. On one family of patterns they are worse. Model code introduces a hardcoded application secret key
1.00 0.50 0.25 0.00
LLM minus human adoption
0.75
−0.25 −0.50 −0.75 −1.00
a lg n algone ris pi k a nned con au alrt text st to g l creadnt timescapist ent e cme ia ls p csrf flag csrf dis csp pro able tec d envdebugtion exp or v on ex ch aul exteextenplicit eockedt rna sion rigin lize che d c h fle sec k har hardardcoxdor grret dco cod ed id ded ed key s hossecreet cret t all key ht h owli htttpps cattponslyt s e llba inpu inlinnforceck t va e sty d no v med lidat le a ia io parlidatioquerny a p pat aramm binn risk h tr et ding ave eriz per rsal ed u s e ris k red raw rpsalt irec htm kce t va l ri li s k sadatio sch samfe joinn s e c e me e s i ure ch te file eck ses semsaecurneame sion nti fla g shesecrecthtml ll t r e n u e v strinstatseize lirmisk g co par it a st ncat m usetraverrong ksql s fra sal df usemewroisk veri uss librarrk viefwy sigens ormy por atu wildweatk metrea card has orig h in
task family
command exec cors config file upload jwt verify landing page oauth2 django oauth2 express oauth2 spring oauth pkce password hash path serve secrets env session cookie spring jpa query sql query ssrf fetch xss escape
security pattern
Figure 4: Signed difference in pattern adoption, model minus human, per task family and pattern (language variants collapsed to families). Blue marks patterns the models adopt more, red marks patterns the human corpus adopts more, and a blank cell means the pattern does not apply to that family. and a hardcoded client secret more often than the human corpus, typically as an inline placeholder value in an otherwise complete program. This is a real and recurring hazard. A placeholder secret that ships unchanged is a live vulnerability, and it is a pattern the human answers, which more often leave configuration to the reader, produce less. The security picture is therefore not that models are safe and humans are not, but that the two sources fail in different places, the human corpus on legacy defensive omissions and the models on inlined secrets and completeness that invites unedited reuse. Security score by origin. Figure 5 breaks the security score down by origin, with the human corpus alongside each model. The human corpus sits at the bottom, and the models spread above it, so the aggregate gap is not driven by a single model but is a property of the model corpus as a whole. The spread among models is itself informative, and §8 discusses which models sit highest.
9. Style Divergence The provenance result of §7 is driven by style, and Table 4 makes the style contrast concrete. For the same task, model code is far larger than the human answer, 1801 characters on average against 552, roughly a threefold difference. The models wrap their solution in functions, add error handling, and import more supporting modules, where the human answers are terser fragments that assume a
nemotron-3-ultra-550b nemotron-3-nano-30b mistral-large-3-675b qwen3.5-397b tencent-hy3 north-mini-code mistral-small-4-119b codestral StackOverflow qwen3-coder-30b
−0.4
−0.2 0.0 0.2 0.4 mean security score (secure minus insecure patterns)
0.6
Figure 5: Mean security score by origin, the human corpus alongside each model. The human corpus is lowest and the models spread above it. surrounding context. Error handling is present in 31 percent of model samples against 8 percent of human samples. Human answers use longer individual lines, a density that comes from writing a minimal snippet rather than a complete program. These differences are the substance of the attribution result, and they also explain the completeness hazard of §8, since a longer, self-contained, runnable program is exactly the kind of artifact a developer is tempted to paste and ship with its placeholder secret intact. Figure 7 normalises each style feature across the two sources for a direct visual comparison, and shows the same pattern, models higher on size, structure, and error han-
TABLE 4: Mean style features by source. Model code is longer, more structured, and more defensively wrapped than the human answers for the same task. Feature
StackOverflow
LLM
14.368 0.054 0.444 0.085 51.364 0.117
50.148 0.17 1.555 0.314 35.321 0.401
n code lines comment ratio n imports has error handling avg line len sec score
code lines per sample
175 150
10.1. Password Hashing Password hashing is the subject with the sharpest divide. The human corpus scores at the floor, and the models near the ceiling. The reason is visible in the code. The representative human answer, Listing 2, reaches for a fast, general-purpose hash, the exact pattern that decades of guidance warn against for passwords, and it is a highlyvoted answer precisely because it is old. The representative model answer, Listing 3, uses a purpose-built password keyderivation function with a per-user salt and a constant-time verify. This is the cleanest case of the models carrying the current consensus while the human archive preserves the practice that consensus replaced.
125 100
Listing 2. Human SO answer 39660610 (score 9), password hash
75
>>> from passlib.hash import pbkdf2_sha256 >>> pbkdf2_sha256.hash("password", rounds=200000, salt= ,→ b'spamhameggs') '$pbkdf2,→ sha256$200000$c3BhbWhhbWVnZ3M$WL9OLVcb3f7HqHeNT./ ,→ kCJeunydL...
50 25 0 LLM
StackOverflow
Figure 6: Distribution of code length in lines, by source. The human and model distributions barely overlap. Listing 3. LLM codestral, password hash
mean (normalised per feature)
human
LLM
1.2
import bcrypt
1.0
def hash_password(password: str) -> str: """ Securely hash a user password for storage in a ,→ database.
0.8 0.6
Args: password: The plain text password to be hashed.
0.4 0.2 0.0
n code lines
comment ratio
n imports
has error handling
avg line len
sec score
Figure 7: Style features by source, each normalised across the two sources. Model code is larger and more structured, human code is denser per line. dling, humans higher on line density. Figure 6 shows the distribution of code length behind the mean, and the two distributions barely overlap, which is why length alone is a strong provenance signal.
10. Per-Subject Analysis The aggregate result of §8 hides real per-subject structure. The models are not uniformly safer than the human corpus. On some tasks they are far safer, on others they are worse, and the reasons differ by task. Figure 8 gives the security score of each source on each subject, and this section walks through the subjects that carry the clearest lesson, showing a real human answer and a real model generation for each.
Returns: The hashed password as a string. """ # Generate a salt and hash the password salt = bcrypt.gensalt() hashed_password = bcrypt.hashpw(password.encode(' ,→ utf-8'), salt) return hashed_password.decode('utf-8') def verify_password(stored_hash: str, provided_password ,→ : str) -> bool: """ Verify a user's password against the stored hash. # ...
10.2. Token Verification The JWT subject repeats the pattern. The human answers frequently decode a token without pinning the accepted algorithms or checking expiry, the ingredients of the classic algorithm-confusion and replay problems. The model answers, one of which is Listing 5, verify the signature, pin the algorithm list, and check expiry by default. The human example, Listing 4, shows the terser style that omits these steps because it answers a narrower question.
secrets env password hash go password hash js password hash jwt verify go jwt verify session cookie go jwt verify js spring jpa query landing page sql query go xss escape sql query sql query js cors config go oauth2 express cors config file upload oauth2 django path serve oauth pkce go oauth2 spring file upload js cors config js oauth pkce file upload go session cookie js session cookie ssrf fetch command exec oauth pkce js
StackOverflow LLM
−0.2
0.0 0.2 0.4 0.6 security score (secure minus insecure patterns)
0.8
1.0
Figure 8: Security score by subject, human Stack Overflow versus model, sorted by the model score. The models are far safer on password hashing and token verification, roughly even or worse on OAuth, SSRF, and session cookies, where their hardcoded-secret and missing-defense habits cost them.
Listing 4. Human SO answer 43145184 (score 29), jwt verify #!/usr/bin/env python import jwt from cryptography.x509 import load_pem_x509_certificate from cryptography.hazmat.backends import ,→ default_backend PEMSTART = "-----BEGIN CERTIFICATE-----\n" PEMEND = "\n-----END CERTIFICATE-----\n" mspubkey = "The value from the x5c property" IDTOKEN = "the id_token to be validated" tenant_id = "your tenant id" cert_str = PEMSTART + mspubkey + PEMEND cert_obj = load_pem_x509_certificate(cert_str, ,→ default_backend()) public_key = cert_obj.public_key() decoded = jwt.decode(IDTOKEN, public_key, algorithms=[' ,→ RS256'], aud... if decoded: print "Decoded!" # ...
Listing 5. LLM mistral-large-3-675b, jwt verify import jwt from jwt.exceptions import InvalidTokenError from functools import wraps from flask import request, jsonify # Assuming Flask; ,→ adjust for yo... # Configuration - replace with your actual values SECRET_KEY = "your-secret-key" # Use environment ,→ variables in prod... ALGORITHM = "HS256" # Algorithm used to sign the token def verify_jwt_token(f): """Decorator to verify JWT token in API requests ,→ .""" @wraps(f) def wrapper(*args, **kwargs): auth_header = request.headers.get(" ,→ Authorization") if not auth_header or not auth_header. ,→ startswith("Bearer "): return jsonify({"error": "Authorization ,→ token is missin... token = auth_header.split(" ")[1] try: payload = jwt.decode(token, SECRET_KEY, ,→ algorithms=[ALG... # ...
10.3. Cross-Origin Policy CORS is the subject where the human corpus scores below zero, its insecure patterns outnumbering its secure ones. The common human answer opens the API to any origin with a wildcard, the fastest way to make a crossorigin call succeed and a standing security hazard. The model answers more often restrict the allowed origins to an explicit list. Listings 6 and 7 show the contrast.
completeness here means filling the secret slot with a literal value, which the terser human answers leave to the reader. Listings 8 and 9 show the OAuth case, where the model code is longer and more correct on the protocol yet carries the inlined secret that the audit of §8 flags. Listing 8. Human SO answer 29629536 (score 46), oauth pkce from flask import url_for, current_app, redirect, ,→ request from rauth import OAuth2Service import json, urllib2
Listing 6. Human SO answer 27280939 (score 375), cors config // The following property can be used to configure ,→ cross-origin res... // in the HTTP nodes. // See https://github.com/troygoode/node-cors# ,→ configuration-options... // details on its contents. The following is a basic ,→ permissive set... httpNodeCors: { origin: "*", methods: "GET,PUT,POST,DELETE" },
class OAuthSignIn(object): providers = None def __init__(self, provider_name): self.provider_name = provider_name credentials = current_app.config[' ,→ OAUTH_CREDENTIALS'][provi... self.consumer_id = credentials['id'] self.consumer_secret = credentials['secret'] def authorize(self): pass def callback(self): pass # ...
Listing 7. LLM north-mini-code, cors config from flask import Flask, jsonify from flask_cors import CORS # Initialize the Flask application app = Flask(__name__) # Configure CORS for the application # This allows the frontend on a different origin to ,→ make requests. # For production, restrict 'origins' to specific ,→ domains (e.g., "ht... CORS(app, resources={r"/*": {"origins": "*"}}) @app.route('/api/data') def get_data(): """ Example endpoint returning JSON data. """ return jsonify({"message": "CORS is working!", " ,→ status": "succe... if __name__ == '__main__': # Run the development server # ...
Listing 9. LLM mistral-small-4-119b, oauth pkce import secrets import base64 import hashlib import urllib.parse import requests from typing import Dict, Tuple class OAuthPKCE: """ OAuth 2.0 Authorization Code Flow with PKCE ,→ implementation. Handles authorization request generation and token ,→ exchange. """ def __init__(self, client_id: str, client_secret: ,→ str, redirect... auth_url: str, token_url: str, scope: ,→ str = None): self.client_id = client_id self.client_secret = client_secret self.redirect_uri = redirect_uri self.auth_url = auth_url self.token_url = token_url # ...
10.4. OAuth and Session Cookies, Where the Models Slip 10.5. Parameterized Queries The two subjects where the models do not lead are the ones that expose their own failure mode. On OAuth the models reliably add the proof-key-for-code-exchange steps that the human answers omit, yet their overall score is dragged down because they also inline a client secret as a placeholder in the same file. On session cookies the same thing happens with the application secret key. The models produce a complete, runnable configuration, and
The database subject sits between the extremes. The human corpus is split. Some answers use a parameterized query, the safe form, while a substantial fraction build the query by concatenating or formatting the user input directly into the SQL string, the textbook injection hazard. The models almost always parameterize. Listings 10 and 11 show a human and a model answer. The human corpus also reaches
for an object-relational mapper more often than the models, which hand-write the parameterized query, a difference of idiom that does not change the security outcome but does change the style. Listing 10. Human SO answer 25029149 (score 21), sql query from django.shortcuts import render from django.shortcuts import HttpResponse from .models import Person from django.core.exceptions import * def index(request): return render(request, 'form.html') def search(request): if request.method == 'POST': search_id = request.POST.get('textfield', None) try: user = Person.objects.get(name = search_id) #do something with user html = ("<H1>%s</H1>", user) return HttpResponse(html) except Person.DoesNotExist: return HttpResponse("no such user") else: return render(request, 'form.html')
Listing 12. Human SO answer 62357616 (score 7), file upload #!/usr/bin/env python # -*- coding: utf-8 -*from flask import Flask from flask import request from flask import render_template import os app = Flask(__name__)
@app.route("/", methods=['POST', 'GET']) def index(): if request.method == "POST": f = request.files['audio_data'] with open('audio.wav', 'wb') as audio: f.save(audio) print('file uploaded successfully') return render_template('index.html', request=" ,→ POST") else: # ...
Listing 13. LLM mistral-large-3-675b, file upload
Listing 11. LLM mistral-small-4-119b, sql query import sqlite3 from typing import Optional
from fastapi import FastAPI, UploadFile, File, ,→ HTTPException from fastapi.responses import JSONResponse import os from pathlib import Path app = FastAPI()
def get_user_by_username(username: str) -> Optional[ ,→ dict]: """ Look up a user in a SQLite database by username. Args: username: The username to search for. Returns: A dictionary representing the user if found, ,→ None otherwise. """ conn = None try: conn = sqlite3.connect('users.db') # Replace ,→ with your act... cursor = conn.cursor() cursor.execute("SELECT * FROM users WHERE ,→ username = ?", (u... user = cursor.fetchone() # ...
10.6. File Upload File upload is a subject where neither source scores well, and the reason is the number of separate defenses the task needs. A safe upload validates the file type, sanitizes the filename against path traversal, and limits the size, and few samples from either source do all three. The models do somewhat more of the validation than the human answers, as Listings 12 and 13 show, but both sources leave gaps, which makes upload a subject where provenance is a weaker guide to safety than hashing or tokens.
# Ensure upload directory exists UPLOAD_DIR = Path("uploads") UPLOAD_DIR.mkdir(exist_ok=True) @app.post("/upload/") async def upload_file(file: UploadFile = File(...)): """ Endpoint to upload a file. The file will be saved ,→ to the 'uploa... Returns the saved file path on success. """ try: # Create a secure file path file_path = UPLOAD_DIR / file.filename # ...
10.7. Injection and the Limits of the Trend The injection-style subjects, added to probe tasks where the danger is subtler than a missing library call, complicate the clean picture of the earlier subjects and are the most interesting for it. On some of them the models keep their lead. For rendering user input into a page, the models reach for an auto-escaping template almost always, while the human answers more often build the HTML by hand and leave the escaping out, so the cross-site-scripting hazard is markedly lower in model code. For loading a secret, the models read it from an environment variable or a secret store, where the human answers more often inline a literal. On other injection subjects the models lose their lead or fall behind, and the reason is again their objective. Asked to run a shell command with a user-supplied argument,
the models invoke the shell directly, the convenient and dangerous form, more often than the human answers do, because it is the shortest complete solution. Asked to fetch a user-supplied URL, the models frequently fetch it with no validation of the scheme or the host, missing the serverside-request-forgery defense entirely, because nothing in the prompt named it and the happy-path solution does not need it. These are the same completeness hazard seen with secrets, in a more dangerous form. The model writes the whole feature and omits the defense that a security-aware developer would add against a threat the prompt did not mention. The lesson of the injection subjects is that the model advantage holds where the safe choice is a wellknown library and evaporates, or reverses, where the safe choice is to anticipate an unmentioned threat. Listings 14 and 15 put the two directions side by side, a model autoescaping the cross-site-scripting task and a model fetching a user URL with no server-side-request-forgery check in the same corpus. Listing 14. LLM codestral, xss escape from flask import Flask, request, ,→ render_template_string
Listing 15. LLM tencent-hy3, ssrf fetch import httpx from fastapi import FastAPI, HTTPException, Query from fastapi.responses import Response app = FastAPI(title="URL Fetcher") @app.get("/fetch") async def fetch_url(url: str = Query(..., description=" ,→ The URL to f... # Note: In production, validate/allowlist URLs to ,→ prevent SSRF ... try: async with httpx.AsyncClient(follow_redirects= ,→ True) as client: response = await client.get(url, timeout ,→ =10.0) response.raise_for_status() # Return the fetched content directly with ,→ its original... return Response(content=response.content, ,→ media_type=re... except httpx.HTTPError as e: raise HTTPException(status_code=502, detail=f" ,→ Error fetchin... if __name__ == "__main__": import uvicorn # ...
app = Flask(__name__) # HTML template with a form to input a message and ,→ display it HTML_TEMPLATE = """ <!DOCTYPE html> <html> <head> <title>Message Display</title> </head> <body> <h1>Enter a Message</h1> <form method="POST"> <input type="text" name="message" placeholder=" ,→ Enter your m... <button type="submit">Submit</button> </form> {% if message %} <h2>Your Message:</h2> <p>{{ message }}</p> # ...
Listing 16. Human SO answer 52544846 (score 39), command exec import subprocess
def start(executable_file): return subprocess.Popen( executable_file, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
def read(process): return process.stdout.readline().decode("utf-8"). ,→ strip()
def write(process, message): process.stdin.write(f"{message.strip()}\n".encode(" ,→ utf-8")) process.stdin.flush() # ...
Listing 17. LLM mistral-small-4-119b, command exec import subprocess def run_shell_command_with_file(filename: str) -> None: """Run a shell command with a user-provided ,→ filename.""" # Example command: cat the file and count lines command = ["wc", "-l", filename] try: result = subprocess.run( command, check=True, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) print(f"Command output:\n{result.stdout}") except subprocess.CalledProcessError as e: print(f"Command failed with error:\n{e.stderr ,→ }") except FileNotFoundError: print("Error: 'wc' command not found. Adjust ,→ command as nee... # ...
10.8. Summary of the Per-Subject Picture Across the subjects the lesson is consistent with the aggregate but more precise. The models lead by a wide margin wherever the secure practice is a well-known library call that the human archive predates, hashing and token verification most of all. They lose their lead, and occasionally fall behind, wherever the secure practice is to leave a value unset, because their objective pushes them to produce a complete program and a complete program fills every slot. Provenance therefore predicts not just whether code is likely safe but which specific hazard to look for, a library-omission hazard in human code and a completeness hazard in model code.
11. Framework-Level Comparison The subjects so far use a minimal library so that the security choice is isolated. A developer, however, asks a framework-level question, for example how to set up OAuth 2.0 login in a Java Spring Boot application, or how to add social login to Django, or how to wire up Passport in Express. We add four such subjects, phrased as the natural question a developer would type, and send the identical prompt to every model. Because every model answers the same question, we can place their implementations side by side and compare which security practices each one adopts. Table 5 does this for the Spring Security OAuth subject. Each row is one model’s answer to the same prompt, and each column is a security or configuration check, marked secure or insecure. The comparison makes the differences between models concrete. All of the models produce a working Spring Security configuration, but they diverge on the details that matter. Some enable proof-key-for-code exchange and read the client secret from externalised configuration, while
others inline the secret or disable the cross-site-requestforgery protection that Spring enables by default, the kind of convenience shortcut that is common in tutorials and dangerous in production. The same question therefore yields materially different security postures depending on which model answers it, and the table names exactly where. The framework subjects also confirm the completeness hazard of §8 in a realistic setting. The framework answers are longer and more complete than the minimal-library answers, and the extra completeness again carries risk, since a model that produces a full runnable Spring configuration is more likely to fill the client secret with a literal placeholder than one that leaves the wiring to the reader. The practical implication for a developer is direct. The convenience of a complete, copy-ready framework answer is exactly what makes it worth re-checking for an inlined secret and for a disabled default protection before it is used.
12. Cross-Language Analysis The core tasks are defined in three programming languages, Python, JavaScript, and Go, for the same underlying task, which lets us ask whether the two findings of the paper are properties of a language or of the source. We treat each task as a family with a variant per language, and compare within and across them. The security comparison uses all three languages. The provenance-transfer experiment, which needs a dense paired sample, uses the Python and JavaScript variants where coverage is deepest. The security gap holds in every language. The central security result does not depend on the language. In each of the three the model corpus sits well above the human corpus, a mean security score of 0.35 against the human corpus in Python and 0.34 in JavaScript, with Go higher for both sources but the same gap between them. The absolute level differs by ecosystem, Go scoring higher because its standard library steers even the human answers toward safer defaults, but the direction and the size of the human-tomodel gap are consistent across the three. Table 6 gives the per-family breakdown, and the pattern from §8 repeats. The models lead on the library-backed defensive tasks, hashing and token verification, and slip on the tasks where the safe choice is to leave a secret unset. That the same structure appears in three languages is strong evidence that it reflects how the two sources approach a task rather than a quirk of one ecosystem. Figure 9 plots the same per-family scores as grouped bars, one bar per language, and the visual impression is of two profiles that rise and fall together. Where the models are strong in one language, on hashing and token verification, they are strong in the other, and where they slip, on the tasks that reward leaving a secret unset, they slip in both. The correlation of the per-family scores across languages is the quantitative form of this observation, and it is high, which is the core evidence that the security divergence is a property of the source rather than of a single language ecosystem.
TABLE 5: Framework comparison for the Spring Security OAuth subject. Each row is one model’s answer to the same prompt. A check mark means the pattern is present. Columns marked (S) are secure practices and (I) are insecure ones. Model
fw (S)
codestral mistral-small-4-119b nemotron-3-nano-30b nemotron-3-ultra-550b north-mini-code tencent-hy3
✓ ✓ ✓ ✓ ✓ ✓
go
PKCE (S)
CSRF+ (S)
CSRF off (I)
✓
hard secret (I)
env secret (S)
✓
✓ ✓
✓
✓
html
✓
java
javascript
python
1.0 model security score
0.8 0.6 0.4 0.2 0.0 −0.2
c v o o o o o o o e e e e o h g d h g ie fy ry ry ss exe confi nfig g uploa oad g t veri rify g g pag djang xpre sprin h pkc kce g d has ash g serv ts en cook kie g que l que ery g f fetc escap d r o e l h t r e a u e s n p h 2 o r n t q s n o r p e w u o 2 i v s p q j s s h 2 u fil pa co ors c ma xs oa auth assw word sec essio ion c ring j jwt land oauth auth oaut s ql file c o s s s p com o sp ses pa s
task family
Figure 9: Model security score per task family, one bar per language. The languages rise and fall together, so the security divergence is a property of the source rather than of one ecosystem. Style varies by language. Table 7 reports the model style per language, and the reason the provenance boundary does not transfer becomes concrete. The models write a different amount of scaffolding in each language, driven by the conventions of the ecosystem, so the very features that carry the provenance signal, length and import count and structure, take different baseline values from one language to the next. A classifier that learned the model style of one language is therefore reading a different scale when it sees another, which is the mechanism behind the transfer failure below. Provenance signal is partly language-specific. The attribution result behaves differently across languages, and the asymmetry is informative. A human-versus- model classifier trained on one language and tested on the other does not transfer symmetrically. Trained on JavaScript and tested on Python it reaches 89 percent, close to its within-language accuracy, but trained on Python and tested on JavaScript it falls to 93 percent, near chance. The provenance signal therefore has a language-general component, the gross size and structure difference that survives the transfer in one direction, and a language-specific component that does not. The asymmetry suggests that the model style learned on
Python is narrower than the style learned on JavaScript, so a classifier calibrated on the richer JavaScript signal still recognises the Python one, while the reverse does not hold. This is a caution for anyone deploying a provenance classifier. It must be calibrated on the language it will see, because the human-versus-model boundary does not sit in the same place in every language. The same task in three languages. The concreteness of the cross-language result is easiest to see in the code. Listings 3, 18, and 19 show a model solving the passwordhashing task in Python, JavaScript, and Go. The three differ in every surface detail, the imports, the idioms, the shape, which is why a provenance classifier does not transfer freely between them. All three, however, use the same class of defense, a purpose-built password hash with a salt and a safe verify, which is why the security score is the same across the languages. The listings show both parts of the result, that the style is language-specific and the security posture is not.
TABLE 6: Model security score per task family in each language. The divergence structure of §8 repeats across languages. Task family
Py sec
Py n
JS sec
JS n
command exec cors config cors config go file upload file upload go jwt verify jwt verify go landing page oauth2 django oauth2 express oauth2 spring oauth pkce oauth pkce go password hash password hash go path serve secrets env session cookie session cookie go spring jpa query sql query sql query go ssrf fetch xss escape
-0.21 +0.30 – +0.27 – +0.87 – – +0.25 – – +0.17 – +0.89 – +0.25 +1.00 -0.14 – – +0.46 – -0.21 +0.50
12 15 – 15 – 17 – – 12 – – 18 – 15 – 12 12 14 – – 14 – 12 12
– +0.21 – +0.21 – +0.73 – – – +0.35 – -0.21 – +0.91 – – – +0.02 – – +0.46 – – –
– 14 – 14 – 14 – – – 12 – 14 – 14 – – – 14 – – 14 – – –
TABLE 7: Model code style per language. The amount of scaffolding differs by language, which is why the provenance boundary does not transfer freely. Language
Lines
Imports
Comment
Err. handling
go html java javascript python
62.88 136.77 35.62 49.64 40.2
1.0 0.0 3.88 0.35 2.35
0.17 0.06 0.13 0.22 0.15
0.01 0.0 0.0 0.43 0.45
Listing 18. LLM nemotron-3-nano-30b, password hash js // password-utils.js // Securely hash and verify passwords using the ,→ industry standard b... // Install with: npm install bcrypt const bcrypt = require('bcrypt'); /** * Generate a salt and hash a plaintext password. * @param {string} password - The user's plaintext ,→ password. * @returns {Promise<string>} A promise that resolves ,→ to the hashed... */ async function hashPassword(password) { // 12 rounds is a good default; adjust based on your ,→ performance/... const saltRounds = 12; const hashed = await bcrypt.hash(password, saltRounds ,→ ); return hashed; } /** * Verify a plaintext password against a stored hash. // ...
Listing 19. LLM nemotron-3-ultra-550b, password hash go package main import ( "fmt" "log" "golang.org/x/crypto/bcrypt" ) // HashPassword generates a bcrypt hash of the ,→ plaintext password. // The cost parameter (12) is a good default balancing ,→ security and... func HashPassword(password string) (string, error) { bytes, err := bcrypt.GenerateFromPassword([]byte( ,→ password), 12) return string(bytes), err } // CheckPasswordHash compares a bcrypt hash with a ,→ plaintext password. // Returns nil if they match, or an error (typically ,→ bcrypt.ErrMism... func CheckPasswordHash(password, hash string) error { return bcrypt.CompareHashAndPassword([]byte(hash), ,→ []byte(passw... // ...
Summary. The languages agree on the substantive result and disagree on the incidental one. The security divergence between human and model code is a property of the source and appears in both languages, while the exact decision boundary that separates human from model code is partly language-specific and does not transfer freely. A study in a single language would have reported the first and silently assumed the second.
13. Case Study, Vulnerability Repair The generation results of §8 measure what a model writes when asked for a task from scratch. A second, more practical question is what a model does when it is handed insecure code and asked to fix it. This is the common real workflow of a developer pasting a flagged snippet into an assistant, and it is a different capability from clean-slate generation. We test it directly. Setup. We assemble 21 short programs, each containing one well-known vulnerability drawn from 12 distinct Common Weakness Enumeration classes, among them a weak password hash (CWE-327), SQL injection (CWE89), OS command injection (CWE-78), a disabled signature check (CWE-347), a wildcard cross-origin policy (CWE942), a hardcoded credential (CWE-798), reflected crosssite scripting (CWE-79), path traversal (CWE-22), a noncryptographic random source (CWE-330), server-side request forgery (CWE-918), unsafe deserialization (CWE502), and an open redirect (CWE-601). The seeds span four languages, Python, JavaScript, Go, and Java. Each seed carries two regular expressions, one for the insecure pattern that a fix must remove and one for the secure pattern a correct fix must introduce. We send each seed to every model with a neutral instruction to fix the security
TABLE 8: Vulnerability-repair success rate by model. A repair succeeds only when the insecure pattern is removed and the correct secure pattern is added. Model nemotron-3-ultra-550b north-mini-code tencent-hy3 mistral-large-3-675b nemotron-3-nano-30b codestral mistral-small-4-119b
Repairs
Fixed
Rate
20 19 16 14 21 20 21
18 17 14 12 15 14 11
0.90 0.90 0.88 0.86 0.71 0.70 0.52
vulnerability and return the corrected code, and we score the result deterministically. A repair counts as successful only when the insecure pattern is gone and the expected secure pattern is present, so a model that deletes the dangerous call without adding the correct defense is not credited. This yields 131 scored repairs. Models repair most but not all vulnerabilities. The overall repair success rate is 77 percent. The models are competent at the task, and Table 8 shows the per-model breakdown, with the strongest model reaching 90 percent. The spread across models is real and follows capability, with the larger models repairing more reliably than the smaller ones, which matches the intuition that repair, like generation, rewards a model that knows the idiomatic secure replacement. The partial-fix hazard. The most useful result of the case study is the failure mode. In 16 percent of all repairs the model removed the insecure pattern but did not add the secure one. This is not a cosmetic miss. A repair that deletes an md5 call but stores the password in plain text, or that removes a wildcard origin but sets no origin at all, produces code that no longer matches the insecure signature yet is not secure. A regex or signature-based scanner would mark such a repair as clean. The partial fix is the repair-time analogue of the completeness hazard of §8, and it is a concrete caution against trusting an assistant’s fix without re-checking that the defense it was supposed to add is actually there. Some vulnerability classes are harder than others. Table 9 and Figure 10 give the success rate per vulnerability. The classes with a single canonical library fix, a parameterized query, a strong password hash, or a cryptographic random source, are repaired most reliably, often at or near a perfect rate, because the correct replacement is unambiguous and the models know it. The classes that require adding a check the original code did not have, validating a redirect target, a fetched URL, or a file path, are repaired least reliably, the open redirect and the path traversal being the hardest in our set. The reason is the same one that makes these classes written insecurely in the first place. The safe form is an addition the model must think to make, not a substitution it can pattern-match, and that is where both generation and repair are weakest. The pattern holds across all four languages, with the ranking of easy and hard classes preserved, so the effect is a property of the vulnerability
TABLE 9: Vulnerability-repair success rate by weakness class, across all models. CWE
Vulnerability
Lang
n
Rate
CWE-601 CWE-22 CWE-327 CWE-502 CWE-327 CWE-78 CWE-22 CWE-347 CWE-79 CWE-918 CWE-798 CWE-327 CWE-78 CWE-89 CWE-942 CWE-798 CWE-89 CWE-89 CWE-89 CWE-330 CWE-330
open redirect path traversal md5 password deserialize md5 password cmd exec path traversal jwt noverify xss raw ssrf hardcoded key md5 password shell true sql concat cors wildcard hardcoded key sql concat sql concat sql sprintf weak random weak random
py py ja py ja ja ja py py py py py py ja py ja ja py go ja py
3 7 7 4 7 6 6 7 7 5 7 7 7 7 7 4 7 7 7 5 7
0.00 0.29 0.43 0.50 0.57 0.67 0.67 0.71 0.71 0.80 0.86 0.86 0.86 0.86 1.00 1.00 1.00 1.00 1.00 1.00 1.00
class rather than of one language.
14. Discussion Provenance is a stylistic boundary. The attribution result of §7 is strong, and its explanation matters as much as its accuracy. The human-versus-model boundary is recovered mainly from the shape of the code, its size, its structure, and its scaffolding, and only secondarily from its security content. A reviewer or an audit tool can therefore flag likely machine-generated code cheaply, from surface features alone, and the same features that make the code distinguishable are the ones that make it feel finished. This connects the provenance question to the security question. Machine code is recognisable because it is complete, and its completeness is exactly what makes its embedded flaws dangerous to paste unedited. The security comparison is against a human baseline, not a gold standard. The finding that model code adopts secure patterns more often than the human corpus must be read carefully. It is a comparison between two real sources a developer might copy from, not a claim that model code is secure in absolute terms. The prior literature, which compares model output against a correct reference, finds that assistants still emit vulnerable code at a meaningful rate [3], [4]. Our result is compatible with that. Against the highlyvoted but often dated human answers on Stack Overflow, the models have shed many legacy omissions, yet they introduce their own failure mode. Both readings hold, and the accurate summary is that the two sources fail differently rather than that one is safe. Why the sources fail where they do. The pattern of divergence has a plausible cause. The human corpus is an archive.
By model
By vulnerability CWE-330 weak random CWE-330 weak random CWE-89 sql sprintf CWE-89 sql concat CWE-89 sql concat CWE-798 hardcoded key CWE-942 cors wildcard CWE-89 sql concat CWE-78 shell true CWE-327 md5 password CWE-798 hardcoded key CWE-918 ssrf CWE-79 xss raw CWE-347 jwt noverify CWE-22 path traversal CWE-78 cmd exec CWE-327 md5 password CWE-502 deserialize CWE-327 md5 password CWE-22 path traversal CWE-601 open redirect
nemotron-3-ultra-550b north-mini-code
model
tencent-hy3 mistral-large-3-675b nemotron-3-nano-30b codestral mistral-small-4-119b
0.0
0.2
0.4
0.6
repair success rate
0.8
1.0
0.0
0.2
0.4
0.6
repair success rate
0.8
1.0
Figure 10: Vulnerability-repair success rate by model (left) and by weakness class (right). Library-substitution fixes succeed most often, and fixes that require adding a missing check succeed least often. A highly-voted answer accumulates its score over years, and much of it predates the current defaults for hashing, token handling, and cross-origin policy, which is why the human corpus under-adopts exactly those defensive patterns. The models are trained on a snapshot that includes the more recent guidance, so they reproduce the current consensus on those tasks. The models’ own failure, the inlined placeholder secret, follows from their objective. They are asked for a single complete program, so they fill every slot, including the ones a human answer would leave to the reader, and a filled secret slot is a hazard the human fragment never creates. Implications for review. The two results combine into concrete guidance. Code that a lightweight classifier flags as machine-generated should be routed to the checks that model code most often fails, which the divergence profile of §8 names, foremost the scan for hardcoded secrets and placeholder configuration. Code identified as copied from a human source should instead be routed to the legacyomission checks, the hashing, token, and cross-origin defaults that the human corpus most often misses. Provenance is thus not an end in itself but a router that sends each artifact to the review it most needs. Deploying a provenance classifier. The cross-language result of §12 is a direct warning for anyone who would deploy the attribution classifier as a tool. The boundary between human and model code does not sit in the same place in every language, and a classifier calibrated on one language can perform below chance on another, producing confident but incorrect labels. A deployable detector must therefore be calibrated per language, and its confidence must be discounted on any language it was not trained on. The interpretable features we use make this tractable, because the small feature set can be re-fit on a modest per-language sample rather than requiring a full retraining
of a deep detector. It is therefore more accurate to treat provenance attribution as a language-scoped capability than as a universal one. The moving-target caveat. Both of our results are snapshots. The human corpus ages in one direction, its highlyvoted answers drifting further from current practice as time passes, which will tend to widen the security gap we measure. The models move in the other direction, retrained on newer data and tuned by their providers, which may narrow the gap or shift the failure mode. The value of the released pipeline is that it makes the measurement repeatable. The same specification re-run in a year will report where each source has moved, and the fail-closed checker guarantees that the new numbers are the ones the new data supports.
15. Limitations and Threats to Validity Pattern detectors are lexical. The security checks are regular expressions, not a semantic analysis. They detect the presence of a pattern, for example a call to a strong key-derivation function, but they do not prove the pattern is used correctly. A sample can adopt bcrypt and still store the result badly. The detectors are a transparent, reproducible first cut, not a verifier, and the paper’s claims are therefore about pattern adoption rather than about proven security. We treat the lexical nature of the detectors as the principal threat to construct validity and avoid any claim that a high security score means a program is safe. The human corpus is a snapshot of one site. The human material is drawn from Stack Overflow, retrieved through its public API with a fixed query per subject. A different query or a different site would surface different code, and the age distribution of highly-voted answers biases the corpus toward older practice, which is part of what the securitydivergence result measures rather than a nuisance to remove.
We record each answer’s score and year so this bias is visible and auditable. The model set is open-weight and gateway-limited. The models are those a public open-weight gateway makes available, and the widely-used closed commercial assistants are not among them. The results therefore describe open-weight models, which is a real and growing population, but not the specific commercial products the earlier user studies examined. The gateway also throttles models unevenly, so per-model sample counts differ, and a model with few samples contributes a noisier security profile. We report true counts and weight aggregates by them. Attribution is within a fixed pool. The model-attribution result is a closed-world classification among the models in the study. It does not address the open-world question of recognising an unseen model, and its accuracy would fall as the pool grows and model styles converge. We present it as evidence that a per-model fingerprint exists at this scale, not as a deployable detector. Prompt sensitivity. Each subject uses a single fixed prompt. Model output is sensitive to phrasing, and a prompt that explicitly asked for security would raise the adoption of defensive patterns. Our prompts are deliberately neutral, phrased as an ordinary developer request, so the measured adoption reflects the default behaviour of each source rather than its best behaviour under prompting.
16. Ethics and Responsible Use The two capabilities this paper demonstrates, attributing code to a source and profiling the security of that source, are dual use, and we state the intended and the misuse cases plainly. Intended use. The provenance classifier is meant to route code to the review it needs, as §8 describes, and to support consented uses such as a team understanding the composition of its own codebase or an educator discussing the difference between hand-written and generated solutions. The security-divergence profile is meant to make securecoding review more efficient by naming, per source, the failures most worth looking for. Misuse and why the risk is limited. A provenance classifier could in principle be used to penalise the use of an assistant, for example to detect and sanction machine help where it is disallowed. We note two limits that reduce this risk. The accuracy is bounded and, as §12 shows, does not transfer across languages, so a naive deployment would produce confident errors that make it unsuitable as a sole basis for any consequential decision. And the signal is dominated by surface style, which a user who wished to evade detection could alter trivially by reformatting, so the classifier is not a robust adversarial detector and should not be presented as one. Data handling. The human corpus is public Stack Overflow content retrieved through the official API within its terms, and we store only the code block, its public identifier, and its public score, no personal data. The model corpus is generated by us through a gateway we are entitled to use,
and no third-party private code is collected. The released dataset therefore contains only public human answers and our own generations. No claim of absolute security. The security scores are pattern-adoption measures, not verification, as §15 states. We are careful throughout not to certify any sample as secure, and we frame every result as a comparison between sources rather than as an audit of any individual program.
17. Conclusion We studied two sources of developer code side by side, the human answers of Stack Overflow and the output of 9 open-weight language models, on 31 security-sensitive tasks, using only open sources and a fully reproducible pipeline. Two results stand out. Provenance is recoverable. A lightweight classifier separates human from model code at 93 percent and attributes a sample to its specific model at 48 percent, and the boundary is stylistic before it is a matter of security. Security diverges. Against the human corpus, the models adopt modern defensive patterns far more often, with an aggregate security score of 0.40 against 0.12, yet they introduce their own hazard by inlining placeholder secrets into otherwise complete programs. The two sources do not rank as safe and unsafe. They fail in different places, and knowing which source a snippet came from tells a reviewer which failure to look for. The pipeline is data driven and released, so any new task is one specification entry away, and a fail-closed checker re-derives every number here from the stored samples.
Appendix A. Pattern Detector Catalogue Table 12 in Appendix F lists every security and style detector used, its subject, its polarity (secure or insecure), and a short description. Each detector is a regular expression evaluated against the sample text. The catalogue is the complete, reproducible definition of what the security-divergence analysis measures. A detector firing means its pattern is lexically present, which the paper treats as adoption of the pattern, not as proof of its correct use.
Appendix B. Per-Model Security Profile Table 10 gives the mean security score and sample count for each model and for the human corpus, the per-origin detail behind Figure 5. The human corpus is included as a row so the models can be read against it directly.
Appendix C. Per-Model Security by Language Table 11 gives each model’s mean security score in each language, the per-model detail behind the cross-language
TABLE 10: Mean security score and sample count by origin. Origin
Sec. score
n
+0.62 +0.47 +0.46 +0.44 +0.43 +0.32 +0.30 +0.26 +0.12 -0.40
60 62 30 4 62 62 62 68 117 1
nemotron-3-ultra-550b nemotron-3-nano-30b mistral-large-3-675b qwen3.5-397b tencent-hy3 north-mini-code mistral-small-4-119b codestral StackOverflow (human) qwen3-coder-30b
3)
4) 5)
result of §12. The scores are close across languages within a model, which is the model-level form of the finding that the security posture travels with the source rather than the language. TABLE 11: Mean security score per model in each language. Model codestral mistral-large-3-675b mistral-small-4-119b nemotron-3-nano-30b nemotron-3-ultra-550b north-mini-code qwen3-coder-30b qwen3.5-397b tencent-hy3
go
html
java
javascript
+0.59 – +0.22 +0.78 +0.62 +0.51 – – +0.51
+0.80 +0.80 +0.80 +0.10 – +0.60 – – +0.80
+0.25 – +0.38 +0.38 +0.69 +0.50 – – +0.50
+0.09 +0.40 +0.37 +0.36 +0.64 +0.14 – – +0.35
Appendix D. Full Adoption Table Table 13 in Appendix F gives the complete per-subject, per-pattern adoption rate for the human corpus and the model corpus, the detail behind the aggregate of §8 and the per-subject discussion of §10. H adopt and L adopt are the human and model adoption rates, and the paired counts give the sample size behind each cell so that a rate resting on few samples can be identified.
Appendix E. Reproducibility The study is reproducible end to end from public sources. The pipeline is a sequence of small scripts, each writing an intermediate artifact to disk, and every number in the paper is re-derived from those artifacts by a fail-closed checker. 1)
2)
probe_fast.py probes every model the gateway advertises and records, in model_availability.csv, whether each answers, is rate limited, or is unavailable. fetch_stackoverflow.py retrieves the human corpus for each subject through the public
6)
Stack Overflow API, storing each answer with its identifier, score, and year. generate.py sends each subject prompt to each usable model and stores the returned code with its provenance metadata. It is resumable and paced, and it is run as repeated passes so that models throttled during one pass are collected in a later one. extract_features.py reduces every sample to the security and style features of §5, writing features.csv. benchmark.py computes the attribution classifiers and the security-pattern prevalence, writing the analysis tables and numbers.json. gen_tables.py and gen_figures.py emit the paper’s tables and figures, and verify_numbers.py re-derives every macro from the data and fails if any disagrees.
The subject specification, the model list, and every collected sample are released with the code, so a third party can python re-run the pipeline or extend it to a new subject by adding a single specification entry. The classifiers use scikit-learn [13] +0.15 +0.47 with a fixed random seed, so the reported accuracies are +0.26 deterministic given the same samples. +0.41 +0.60 +0.28 -0.40 +0.44 +0.40
Appendix F. Reference Tables The two reference tables below are the full detector catalogue and the complete per-subject adoption data. They are set in a single column so they can run across pages without truncation. TABLE 12: Complete detector catalogue. Polarity S marks a secure pattern and I an insecure one. Subject
Pattern
oauth pkce
pkce
oauth pkce
state param
oauth pkce
redirect validation
oauth pkce
https enforced
oauth pkce
hardcoded secret
oauth pkce
uses library
jwt verify
verify signature
jwt verify
alg pinned
jwt verify
alg none risk
jwt verify
exp checked
jwt verify
uses library
password hash
strong kdf
password hash
weak hash
password hash
per user salt
password hash
constant time cmp
sql query
parameterized
sql query
string concat sql
sql query
uses orm
file upload
extension check
file upload
secure filename
file upload
path traversal risk
file upload
size limit
cors config
wildcard origin
cors config
explicit origin
cors config
credentials flag
landing page
viewport meta
landing page
semantic html
landing page
alt text
landing page
media query
landing page
flex or grid
landing page
inline style
session cookie
httponly
session cookie
secure flag
session cookie
samesite
session cookie
hardcoded secret key
oauth pkce js
pkce
oauth pkce js
state param
oauth pkce js
redirect validation
oauth pkce js
https enforced
oauth pkce js
hardcoded secret
oauth pkce js
uses library
jwt verify js
verify signature
jwt verify js
alg pinned
jwt verify js
alg none risk
jwt verify js
exp checked
jwt verify js
uses library
password hash js
strong kdf
Pol. S S S S I S S S I S S S I S S S I S S S I S I S S S S S S S I S S S I S S S S I S S S I S S S
Description uses PKCE (code challenge/verifier) uses anti-CSRF state parameter validates redirect uri uses https endpoints hardcodes client secret uses an OAuth library vs hand-rolled verifies the signature pins allowed algorithms permits alg=none or verify=False checks expiry uses a JWT library uses bcrypt/scrypt/argon2/pbkdf2 uses md5/sha1/plain sha256 uses a salt constant-time comparison uses parameterized query concatenates/format user input into SQL uses an ORM validates file type/extension sanitizes the filename joins raw filename into a path limits upload size allows any origin (*) restricts to explicit origins handles credentials explicitly responsive viewport meta uses semantic elements images have alt text uses responsive media queries uses flexbox or grid relies on inline styles sets HttpOnly sets Secure sets SameSite hardcodes the app secret key uses PKCE (code challenge/verifier) uses anti-CSRF state parameter validates redirect uri uses https endpoints hardcodes client secret uses an OAuth library verifies the signature pins allowed algorithms permits alg none or decode without verify checks expiry uses a JWT library uses bcrypt/scrypt/argon2
continued
Table 12 continued from previous page
Subject
Pattern
Pol.
password hash js
weak hash
password hash js
per user salt
password hash js
constant time cmp
sql query js
parameterized
sql query js
string concat sql
I S S S I S I S S S S I S S S S I S I S S S S I S S S I S I S S S S I S S S S I S S S S I S I S S S I S S S
sql query js
uses orm
cors config js
wildcard origin
cors config js
explicit origin
cors config js
credentials flag
file upload js
extension check
file upload js
secure filename
file upload js
path traversal risk
file upload js
size limit
session cookie js
httponly
session cookie js
secure flag
session cookie js
samesite
session cookie js
hardcoded secret key
password hash go
strong kdf
password hash go
weak hash
password hash go
per user salt
password hash go
constant time cmp
jwt verify go
verify signature
jwt verify go
alg pinned
jwt verify go
alg none risk
jwt verify go
exp checked
jwt verify go
uses library
sql query go
parameterized
sql query go
string concat sql
sql query go
uses orm
cors config go
wildcard origin
cors config go
explicit origin
cors config go
credentials flag
file upload go
extension check
file upload go
secure filename
file upload go
path traversal risk
file upload go
size limit
session cookie go
httponly
session cookie go
secure flag
session cookie go
samesite
session cookie go
hardcoded secret key
oauth pkce go
pkce
oauth pkce go
state param
oauth pkce go
redirect validation
oauth pkce go
https enforced
oauth pkce go
hardcoded secret
oauth pkce go
uses library
command exec
shell true risk
command exec
arg list
command exec
input validation
xss escape
auto escape
xss escape
raw html risk
xss escape
csp
ssrf fetch
scheme check
ssrf fetch
host allowlist
Description uses md5/sha1 uses a salt safe compare / library verify uses parameterized query concatenates user input into SQL uses an ORM allows any origin (*) restricts to explicit origins handles credentials explicitly validates file type/extension sanitizes the filename joins raw filename into a path limits upload size sets HttpOnly sets Secure sets SameSite hardcodes the session secret uses bcrypt/scrypt/argon2 uses md5/sha1 uses a salt safe compare / library verify verifies the signature pins allowed algorithms does not check signing method checks expiry uses a JWT library uses parameterized query builds SQL with Sprintf/concat uses an ORM allows any origin (*) restricts to explicit origins handles credentials explicitly validates file type/extension sanitizes the filename joins raw filename into a path limits upload size sets HttpOnly sets Secure sets SameSite hardcodes the session key uses PKCE uses anti-CSRF state parameter validates redirect uri uses https endpoints hardcodes client secret uses an OAuth library uses shell=True with input passes arguments as a list validates/sanitizes input uses a template/auto-escaping inserts input into HTML string directly sets a content security policy validates the URL scheme restricts the host
continued
Table 12 continued from previous page
Subject
Pattern
Pol.
ssrf fetch
no validation risk
secrets env
env or vault
secrets env
hardcoded key
path serve
safe join
path serve
traversal risk
oauth2 spring
uses framework
oauth2 spring
pkce
oauth2 spring
csrf protection
oauth2 spring
csrf disabled
oauth2 spring
hardcoded secret
oauth2 spring
externalized secret
oauth2 django
uses framework
oauth2 django
pkce
oauth2 django
https callback
oauth2 django
hardcoded secret
oauth2 django
externalized secret
oauth2 django
debug on
oauth2 express
uses framework
oauth2 express
state param
oauth2 express
session secret env
oauth2 express
hardcoded secret
oauth2 express
https callback
spring jpa query
uses orm
spring jpa query
param binding
spring jpa query
string concat sql
I S I S I S S S I I S S S S I S I S S S I S S S I
Description fetches the raw url with no checks reads the secret from env/secret store hardcodes the key literal uses a safe path join / basename joins the raw name into a path uses Spring Security OAuth2 client enables PKCE keeps CSRF protection disables CSRF protection hardcodes the client secret reads secret from config/env uses a Django OAuth library enables PKCE uses https callback hardcodes the client secret reads secret from env/config ships with DEBUG True uses passport oauth2 strategy uses state parameter reads session secret from env hardcodes the client secret uses https callback url uses Spring Data repository uses parameter binding concatenates input into a query
TABLE 13: Complete per-subject, per-pattern adoption for the human (H) and model (L) corpora, with sample counts. Subject
Pattern
command exec
arg list input validation shell true risk
cors config
credentials flag explicit origin wildcard origin
cors config go
credentials flag explicit origin wildcard origin
cors config js
credentials flag explicit origin wildcard origin
file upload
extension check path traversal risk secure filename size limit
file upload go
extension check path traversal risk secure filename size limit
file upload js
extension check path traversal risk secure filename
H adopt
Hn
L adopt
Ln
0.143 0.0 0.143 0.0 0.375 0.5 0.25 0.25 0.125 0.0 0.25 0.125 0.125 0.0 0.0 0.0 – – – – – – –
7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 8 – – – – – – –
0.167 0.25 0.417 0.267 0.6 0.133 0.833 0.833 0.417 0.571 0.714 0.429 0.267 0.0 0.4 0.133 0.5 0.667 0.667 1.0 0.5 0.071 0.143
12 12 12 15 15 15 12 12 12 14 14 14 15 15 15 15 12 12 12 12 14 14 14
continued
Table 13 continued
Subject
Pattern
jwt verify
alg none risk
size limit alg pinned exp checked uses library verify signature jwt verify go
alg none risk alg pinned exp checked uses library verify signature
jwt verify js
alg none risk alg pinned exp checked uses library verify signature
landing page
alt text flex or grid inline style media query semantic html viewport meta
oauth2 django
debug on externalized secret hardcoded secret https callback pkce uses framework
oauth2 express
hardcoded secret https callback session secret env state param uses framework
oauth2 spring
csrf disabled csrf protection externalized secret hardcoded secret pkce uses framework
oauth pkce
hardcoded secret https enforced pkce redirect validation state param uses library
oauth pkce go
hardcoded secret https enforced pkce redirect validation state param uses library
oauth pkce js
hardcoded secret https enforced pkce
H adopt
Hn
L adopt
Ln
– 0.0 0.375 0.25 0.5 0.625 0.0 0.5 0.375 0.375 0.75 0.0 0.0 0.25 0.25 0.5 – – – – – – 0.0 0.0 0.0 0.5 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.125 0.125 0.125 0.0 0.0 0.875 0.0 0.571 0.0 0.0 0.143 0.571 – – – – – – – – –
– 8 8 8 8 8 8 8 8 8 8 4 4 4 4 4 – – – – – – 2 2 2 2 2 2 3 3 3 3 3 8 8 8 8 8 8 7 7 7 7 7 7 – – – – – – – – –
0.214 0.059 0.882 0.882 1.0 0.941 0.0 0.833 0.833 0.917 0.917 0.0 0.071 0.929 0.929 1.0 0.0 0.769 0.077 0.923 1.0 1.0 0.0 0.25 0.25 0.0 0.25 1.0 0.167 0.083 0.75 0.25 1.0 0.0 0.25 0.5 0.417 0.0 1.0 0.333 0.778 1.0 0.0 0.667 0.056 0.333 0.667 1.0 0.0 0.75 0.417 0.643 0.786 1.0
14 17 17 17 17 17 12 12 12 12 12 14 14 14 14 14 13 13 13 13 13 13 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 18 18 18 18 18 18 12 12 12 12 12 12 14 14 14
continued
Table 13 continued
Subject
Pattern redirect validation state param uses library
password hash
constant time cmp per user salt strong kdf weak hash
password hash go
constant time cmp per user salt strong kdf weak hash
password hash js
constant time cmp per user salt strong kdf weak hash
path serve
safe join
secrets env
env or vault
session cookie
hardcoded secret key
traversal risk hardcoded key httponly samesite secure flag session cookie go
hardcoded secret key httponly samesite secure flag
session cookie js
hardcoded secret key httponly samesite secure flag
spring jpa query
param binding string concat sql uses orm
sql query
parameterized string concat sql uses orm
sql query go
parameterized string concat sql uses orm
sql query js
parameterized string concat sql uses orm
ssrf fetch
host allowlist no validation risk scheme check
xss escape
auto escape csp raw html risk
H adopt
Hn
L adopt
Ln
– – – 0.0 0.333 0.167 0.167 – – – – – – – – 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 – – – – 0.0 0.0 0.0 0.0 0.375 0.0 0.0 0.5 0.25 0.375 – – – – – – 0.5 0.0 0.0 0.0 0.0 0.25
– – – 6 6 6 6 – – – – – – – – 1 1 5 5 3 3 3 3 – – – – 1 1 1 1 8 8 8 8 8 8 – – – – – – 2 2 2 4 4 4
0.0 0.357 0.0 0.733 0.933 1.0 0.0 0.917 0.917 1.0 0.0 0.786 0.929 1.0 0.0 0.667 0.417 1.0 0.0 0.786 0.571 0.5 0.857 0.083 1.0 0.667 0.833 0.571 1.0 0.643 0.143 0.333 0.0 1.0 0.929 0.0 0.0 0.75 0.0 0.25 0.929 0.0 0.0 0.333 0.5 0.25 1.0 0.0 0.0
14 14 14 15 15 15 15 12 12 12 12 14 14 14 14 12 12 12 12 14 14 14 14 12 12 12 12 14 14 14 14 12 12 12 14 14 14 12 12 12 14 14 14 12 12 12 12 12 12
References [1]
[2]
[3]
F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes, and S. Fahl, “Stack overflow considered harmful? the impact of copy&paste on android application security,” in IEEE Symposium on Security and Privacy (S&P), 2017. Y. Acar, M. Backes, S. Fahl, D. Kim, M. L. Mazurek, and C. Stransky, “You get where you’re looking for: The impact of information sources on code security,” in IEEE Symposium on Security and Privacy (S&P), 2016. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in IEEE Symposium on Security and Privacy (S&P), 2022.
[4]
N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” in ACM SIGSAC Conference on Computer and Communications Security (CCS), 2023.
[5]
G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. DolanGavitt, “Lost at c: A user study on the security implications of large language model code assistants,” in 32nd USENIX Security Symposium, 2023.
[6]
A. Caliskan-Islam, R. Harang, A. Liu, A. Narayanan, C. Voss, F. Yamaguchi, and R. Greenstadt, “De-anonymizing programmers via code stylometry,” in 24th USENIX Security Symposium, 2015.
[7]
M. Abuhamad, T. AbuHmed, A. Mohaisen, and D. Nyang, “Largescale and language-oblivious code authorship identification,” in ACM SIGSAC Conference on Computer and Communications Security (CCS), 2018.
[8]
M. Chen, J. Tworek, H. Jun et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
[9]
OWASP Foundation, “Owasp top 10,” 2021, open Worldwide Application Security Project.
[10] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the naturalness of software,” in 34th International Conference on Software Engineering (ICSE), 2012. [11] A. Gurioli, M. Gabbrielli, and S. Zacchiroli, “Is this you, llm? recognizing ai-written programs with multilingual code stylometry,” arXiv preprint arXiv:2412.14611, 2024. [12] N. Sakimura, J. Bradley, and N. Agarwal, “Proof key for code exchange by oauth public clients,” 2015, rFC 7636, Internet Engineering Task Force. [13] F. Pedregosa et al., “Scikit-learn: Machine learning in python,” 2011.