Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection

Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection Alberick Euraste Djire✉ University of Luxembourg Luxembourg, Luxembourg euraste.djire@uni.lu

arXiv:2610.01664v1 [cs.SE] 1 Oct 2026

Abstract Code detectors can become obsolete as code-generating models evolve: a detector validated on one generation of models may not transfer to the next. We call this limited useful life a detector halflife. We evaluate eight general-purpose LLM judges and three dedicated detectors on human-written code and code produced by seven generators across C++, Java, and Python. Our results reveal two problems. First, performance varies considerably across generators and prompting strategies, suggesting that some detectors rely on generator-specific patterns rather than general evidence of code provenance. Second, accuracy can conceal severe prediction bias. DetectCodeGPT and GPT-Sniffer achieved an accuracy of 0.50 but an 𝐹 1 score of 0.00 across all generators because they classified almost every sample as AI-generated. However, general-purpose LLM judges achieved stronger accuracy and 𝐹 1 scores. Our results show that general-purpose LLMs are promising training-free judges of code provenance and can outperform dedicated detectors. However, their reliability depends on the judge model, the code generator, and the prompting strategy. We therefore recommend evaluating LLM judges across multiple generators and reporting macro-𝐹 1 alongside class-specific precision and recall.

CCS Concepts • Software and its engineering → Software usability; • Security and privacy → Software security engineering.

Keywords LLM, Generated code detection ACM Reference Format: Alberick Euraste Djire. 2026. Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3832783.3844566

1

Introduction

LLMs now generate, complete, and refactor source code [2, 16]. Undisclosed use can undermine educational assessment, complicate license and intellectual-property review, and make securitysensitive code harder to vet, as LLMs can reproduce insecure patterns from their training data. Reliable code-provenance detection

This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3844566

is therefore increasingly important. However, code is harder to classify than natural language because syntactic constraints compress the token-distribution differences used by text detectors [10, 15]. Existing detectors commonly require white-box access, labeled data, or dedicated infrastructure [4, 8, 13]. More fundamentally, they are validated against a bounded set of generators. As generator families and output styles evolve, reported performance may expire a limitation we call detector half-life. Furthermore, the common reliance of detectors on accuracy creates a second risk: on a balanced dataset, a detector that predicts only one class still scores 0.50. General-purpose LLMs offer a simple alternative. Instead of training a separate classifier, an LLM can be asked whether a program was written by a human or generated by another model. This makes LLM judges attractive as training-free detectors. However, LLM judgments can be affected by model preferences, prompting choices, and superficial code features [7, 9]. This paper studies both the promise and the limits of LLM judges. We compare eight general-purpose LLMs with three dedicated detectors across seven code generators and three programming languages. Specifically, we ask how accurately general-purpose LLMs distinguish AI-generated from human-written code without task-specific training (RQ1), and how lexical, structural, and lexico-semantic guidance affect their accuracy and decision biases (RQ2). We make the following contribution. ❶ evidence that current code detectors have a half-life; ❷ an empirical demonstration that accuracy can mask detector collapse; and ❸ a cross-generator evaluation of LLM judges as a more adaptable, though still imperfect, alternative.

2

Related Works

LLM-output detection uses trained classifiers, zero-shot statistics, or watermarking [5, 6, 8, 13]. Code-specific variants add stylistic, task-conditioned, finetuned, or signal-based features [10, 11, 14, 15]. These methods are typically tested on a fixed set of existing generators and reported mainly using accuracy, leaving their robustness to newer generators and susceptibility to class collapse unclear. General-purpose LLM evaluators avoid detector training [9] and may adapt as foundation models improve, though prompt and judge biases remain [7]. Unlike Pan et al. [9], who evaluate a single prompting strategy, we systematically compare direct and featureguided prompting across eight judges, seven generators, and three languages, revealing that prompting strategy interacts with judge identity in ways a single-condition evaluation cannot expose.

3

Methodology

We use the AI-Human-Generated-Program-Code Dataset [3]: 18 programming problems (C++, Python, Java), each paired with one

ASE ’26, October 12–16, 2026, Munich, Germany

Alberick Euraste Djire and Alberick Euraste Djire

Table 1: Accuracy/𝐹 1𝑠𝑐𝑜𝑟𝑒 by prompting condition and generator (human = positive class); bold = best per column. Model / Approach

StarCoder

ChatGPT

BingAI Chat

Claude Sonnet 5 Gemini-2.5 Gemini-3 Llama Ministral Mistral Large GPT-3.5 GPT-5-mini

0.37/0.45 0.53/0.68 0.14/0.06 0.50/0.00 0.50/0.55 0.56/0.20 0.50/0.67 0.82/0.82

0.71/0.64 0.56/0.69 0.50/0.11 0.50/0.00 0.50/0.55 0.53/0.19 0.50/0.67 0.96/0.95

Claude Sonnet 5 Gemini-2.5 Gemini-3 Llama Ministral Mistral Large GPT-3.5 GPT-5-mini

0.56/0.68 0.44/0.62 0.17/0.12 0.47/0.49 0.48/0.64 0.58/0.55 0.50/0.67 0.56/0.50

0.94/0.94 0.58/0.68 0.56/0.20 0.56/0.53 0.45/0.62 0.61/0.56 0.50/0.67 0.77/0.73

Claude Sonnet 5 Gemini-2.5 Gemini-3 Llama Ministral Mistral Large GPT-3.5 GPT-5-mini

0.43/0.50 0.50/0.65 0.17/0.12 0.69/0.70 0.56/0.68 0.64/0.55 0.50/0.67 0.61/0.46

0.77/0.71 0.47/0.64 0.56/0.20 0.44/0.57 0.47/0.64 0.64/0.55 0.50/0.67 0.86/0.67

Claude Sonnet 5 Gemini-2.5 Gemini-3 Llama Ministral Mistral Large GPT-3.5 GPT-5-mini

0.55/0.65 0.44/0.62 0.26/0.32 0.64/0.68 0.50/0.67 0.64/0.55 0.50/0.67 0.57/0.25

0.87/0.88 0.44/0.62 0.67/0.50 0.53/0.62 0.50/0.67 0.72/0.62 0.50/0.67 0.83/0.40

DetectCodeGPT [12] GPT-Sniffer [8] GPT-Zero

0.50/0.00 0.50/0.00 0.55/0.27

0.50/0.00 0.50/0.00 0.58/0.28

Code Llama

Code Whispered

Github Copilot

Instruct Code T5+

0.39/0.45 0.50/0.67 0.22/0.07 0.47/0.00 0.50/0.55 0.50/0.18 0.50/0.67 0.82/0.82

0.33/0.43 0.50/0.67 0.08/0.06 0.44/0.00 0.50/0.55 0.53/0.19 0.53/0.68 0.92/0.90

0.41/0.47 0.53/0.68 0.26/0.07 0.42/0.00 0.47/0.54 0.39/0.15 0.50/0.67 0.76/0.75

0.37/0.45 0.53/0.68 0.11/0.06 0.44/0.00 0.47/0.54 0.50/0.18 0.56/0.69 0.85/0.82

0.49/0.64 0.44/0.62 0.08/0.11 0.47/0.49 0.47/0.64 0.44/0.47 0.50/0.67 0.73/0.67

0.52/0.67 0.47/0.63 0.31/0.14 0.50/0.50 0.47/0.64 0.42/0.46 0.50/0.67 0.43/0.50

0.53/0.67 0.47/0.63 0.28/0.13 0.56/0.53 0.52/0.67 0.56/0.53 0.50/0.67 0.72/0.62

0.37/0.48 0.47/0.64 0.14/0.11 0.67/0.68 0.56/0.68 0.44/0.44 0.53/0.68 0.79/0.60

0.47/0.53 0.47/0.64 0.28/0.13 0.61/0.65 0.56/0.68 0.44/0.44 0.50/0.67 0.59/0.46

0.41/0.50 0.47/0.64 0.25/0.13 0.81/0.79 0.56/0.68 0.58/0.52 0.50/0.67 0.70/0.46

0.53/0.67 0.44/0.62 0.17/0.29 0.67/0.70 0.50/0.67 0.47/0.46 0.55/0.71 0.73/0.33

0.55/0.67 0.44/0.62 0.37/0.35 0.61/0.67 0.50/0.67 0.50/0.47 0.53/0.69 0.62/0.29

0.61/0.68 0.44/0.62 0.25/0.31 0.72/0.74 0.53/0.68 0.61/0.53 0.50/0.67 0.75/0.33

0.50/0.00 0.50/0.00 0.58/0.28

0.50/0.00 0.50/0.00 0.55/0.27

0.50/0.00 0.50/0.00 0.55/0.27

(a) Accuracy

Zero-shot 0.59/0.56 0.50/0.67 0.47/0.10 0.47/0.00 0.36/0.49 0.44/0.17 0.50/0.67 0.77/0.78

Lexical Context Aware 0.69/0.74 0.50/0.64 0.50/0.18 0.47/0.49 0.50/0.67 0.56/0.53 0.50/0.67 0.60/0.57

0.51/0.65 0.44/0.62 0.31/0.14 0.53/0.51 0.50/0.65 0.56/0.53 0.50/0.67 0.65/0.57

Structural-layout Context Aware 0.71/0.67 0.47/0.64 0.47/0.17 0.56/0.62 0.47/0.64 0.53/0.48 0.50/0.67 0.67/0.50

0.51/0.54 0.47/0.64 0.28/0.13 0.72/0.72 0.58/0.69 0.56/0.50 0.50/0.67 0.67/0.46

Lexico-semantic Context Aware 0.79/0.82 0.44/0.62 0.47/0.39 0.72/0.74 0.50/0.67 0.53/0.48 0.51/0.68 0.58/0.29

0.53/0.65 0.44/0.62 0.39/0.38 0.64/0.68 0.50/0.67 0.61/0.53 0.57/0.71 0.80/0.40

Existing Detectors 0.50/0.00 0.50/0.00 0.58/0.28

0.50/0.00 0.50/0.00 0.52/0.26

human-written solution and one AI-generated solution per generator (StarCoder, ChatGPT, Bing AI Chat, Code Llama, CodeWhisperer, GitHub Copilot, InstructCodeT5+), yielding 18 human and 18×7=126 AI snippets, with the same 18 human solutions used as the baseline for every generator. We evaluate eight LLM judges (Llama 3.1-70B, Gemini-2.5-Flash, Gemini-3-Flash, Ministral-14B, Mistral Large, GPT-3.5-Turbo, GPT-5-mini, Claude Sonnet 5) at temperature 0.1, returning a JSON verdict ("yes" = LLM-generated, "no" = human-written) with justification; human-written code is the positive class. Each query is repeated three times and averaged. We assess four prompting strategies: ❶ Direct zero-shot classification; ❷ Lexical-guided, extracting comments, identifiers, keywords, and libraries first; ❸ Structural-guided, examining control-flow and formatting first; and ❹ Lexico-semantic-guided, combining lexical cues with inferred purpose and semantics first.

4

Experiments and Results

Table 1 and Figure 1 reports Accuracy/𝐹 1𝑠𝑐𝑜𝑟𝑒 (human-written as positive class) for all eight judges across the four prompting conditions and the three baselines, by generator. Finding 1: Existing detectors exhibit a short half-life. DetectCodeGPT [12] and GPT-Sniffer [8], despite strong results in their original settings, collapse to Accuracy= 0.50/𝐹 1𝑠𝑐𝑜𝑟𝑒 =0.00 on every

(b) 𝐹 1𝑠𝑐𝑜𝑟𝑒

Figure 1: Per-language accuracy and 𝐹 1𝑠𝑐𝑜𝑟𝑒 for each judge under the four prompting conditions. generator; GPT-Zero remains weak (𝐹 1𝑠𝑐𝑜𝑟𝑒 0.26–0.28). Their failure to transfer to this generator set demonstrates that a detector’s useful life is bounded by changes in the generation distribution. Finding 2: LLM judges are a promising but conditional alternative. GPT-5-mini reaches zero-shot 𝐹 1𝑠𝑐𝑜𝑟𝑒 of 0.75–0.95, far above the dedicated detectors. Feature guidance recovers Llama (up to 0.79) and lifts Claude to 0.94, but degrades GPT-5-mini to 0.25–0.40 under lexico-semantic prompting. Adaptability therefore does not remove the need for judge- and prompt-specific validation. Finding 3: Accuracy can mask biased detectors. The dedicated detectors’ 0.50 accuracy looks like chance performance, but 𝐹 1𝑠𝑐𝑜𝑟𝑒 =0.00 reveals near-constant “AI” predictions. The same illusion appears in LLM judges: Llama predicts almost only “AI,” whereas Gemini-2.5 and GPT-3.5 lean toward “human.” Claude also ranges from above 0.82 on ChatGPT and BingAI Chat to below 0.50 on StarCoder. Accuracy alone therefore neither diagnoses class bias nor establishes transfer. Finding 4: Per-language sensitivity is judge-specific. C++ is Claude’s weakest language under every prompting condition, while GPT-5-mini stays largely language-invariant throughout; Llama and Mistral-Large both flip C++ from their worst language under zero-shot/lexical prompting to their best under structural/stylosemantic guidance, showing that richer context can reshape which language a judge finds easiest rather than uniformly improving all of them.

5

Conclusion

We examined whether general-purpose LLMs can serve as trainingfree judges of code provenance (RQ1) and how feature-guided prompting affects their reliability (RQ2). Zero-shot judging is highly model-dependent: GPT-5-mini achieves strong discrimination, while several judges collapse toward a single class despite near-chance accuracy. Feature-guided prompting is not uniformly beneficial: it recovers collapsed judges such as Llama but degrades the strongest zero-shot judge, GPT-5-mini. Judgments also vary markedly by generator, suggesting sensitivity to generator-specific style rather than a universal AI/human signal. LLM-based provenance judges should therefore be validated per model and generator, not selected from aggregate scores alone. Future work should examine hybrid human-AI code and provenance-preserving transformations.

Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection

Data-Availability Statement The data used in this research are derived from publicly available sources. To ensure transparency and reproducibility, the source code, datasets, and usage instructions are publicly accessible on Zenodo [1].

References [1] Alberick Euraste Djire. 2026. LLM4GCD : LLMs for Generated Code Detection. doi:10.5281/zenodo.22248135 [2] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology (Sept. 2024), 3695988. doi:10.1145/3695988 [3] Baskhad Idrisov and Tim Schlippe. 2024. Program Code Generation with Generative AIs. Algorithms 17, 2 (Feb. 2024), 62. doi:10.3390/a17020062 Number: 2. [4] JianWang, Shangqing Liu, Xiaofei Xie, and Yi Li. 2024. An Empirical Study to Evaluate AIGC Detectors on Code Content. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. ACM, Sacramento CA USA, 844–856. doi:10.1145/3691620.3695468 [5] Boquan Li, Mengdi Zhang, Peixin Zhang, Jun Sun, and Xingmei Wang. 2024. Resilient Watermarking for LLM-Generated Codes. http://arxiv.org/abs/2402. 07518 arXiv:2402.07518 [cs]. [6] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International conference on machine learning. PMLR, 24950–24962. [7] Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim, and Kyomin Jung. 2026. Don’t Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation. In Findings of the Association for Computational Linguistics: EACL 2026. Association for Computational Linguistics, 1364–1389. doi:10.18653/v1/2026.findings-eacl.70 [8] Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. GPTSniffer: A CodeBERT-based

ASE ’26, October 12–16, 2026, Munich, Germany

classifier to detect source code written by ChatGPT. Journal of Systems and Software 214 (2024), 112059. https://www.sciencedirect.com/science/article/pii/ S0164121224001043 [9] Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. 2024. Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education. In 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 1–11. doi:10.1145/3639474.3640068 [10] Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2024. Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers. doi:10.48550/arXiv.2401.06461 arXiv:2401.06461. [11] Adam Skurla, Dominik Macko, and Jakub Simko. 2026. mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code. In Proceedings of the 20th International Workshop on Semantic Evaluation (2026). Association for Computational Linguistics, 322–327. doi:10.18653/v1/2026.semeval-1.47 [12] Zhenyu Xu and Victor S. Sheng. 2024. Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 23155–23162. https://ojs.aaai.org/index.php/ AAAI/article/view/30361 [13] Xianjun Yang, Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. 2023. Zero-Shot Detection of Machine-Generated Codes. doi:10. 48550/arXiv.2310.05103 arXiv:2310.05103 [cs]. [14] Xingyu Yao, Zhendong Mao, and Quan Wang. 2026. CodeRipple: Wavelet-Based Detection of LLM-Generated Code. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 38351–38364. doi:10.18653/v1/2026.acl-long.1777 [15] Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, and Wenhai Wang. 2024. Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting. doi:10.48550/arXiv.2405.16133 arXiv:2405.16133. [16] Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2023. Large Language Models Meet NL2Code: A Survey. http://arxiv.org/abs/2212.09420 arXiv:2212.09420 [cs].

Received 2026-07-31; accepted 2026-08-25

Related documents

• Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection#1156935
• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation#429448
• Towards the Readability of LLM-Generated Codes through Multitask Representation Engineering#259536
• SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References#429435
• Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models#639744