arXiv:2604.16058v1 [cs.SE] 17 Apr 2026
LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning Mahir Labib Dihan Abir Muhtasim Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET) Dhaka, Bangladesh {mahirlabibdihan, auntor505}@gmail.com
Abstract
missions, code review pipelines need provenance information, and security audits may require distinguishing model-generated patterns from human idioms. Educators are increasingly concerned about LLM usage in programming education [Hung et al., 2024]. Existing solutions such as GPTZero or DetectGPT are primarily tailored for general text and often struggle with code content due to the unique structural and syntactic properties of programming languages. Detecting LLM-generated code is fundamentally a binary classification task: given a code snippet x ∈ C, learn a function f : C → {0, 1} indicating whether the code is human-written (0) or LLM-generated (1). Recent efforts have focused on zero-shot detectors [Shi et al., 2024, Ye et al., 2024] and ML-based detectors [Idialu et al., 2024]. We emphasize the latter, building on the GPTSniffer framework [Nguyen et al., 2024b]. We propose LLMSniffer, combining (1) GraphCodeBERT [Guo et al., 2021], which encodes code tokens and data-flow structure, with (2) supervised contrastive learning [Khosla et al., 2020], which clusters same-class representations while separating different-class ones, and (3) a comment-stripping preprocessing step to remove potentially confounding stylistic metadata. Our main contributions are:
The rapid proliferation of Large Language Models (LLMs) in software development has made distinguishing AI-generated code from human-written code a critical challenge with implications for academic integrity, code quality assurance, and software security. We present LLMSniffer, a detection framework that fine-tunes GraphCodeBERT using a two-stage supervised contrastive learning pipeline augmented with comment removal preprocessing and an MLP classifier. Evaluated on two benchmark datasets—GPTSniffer and Whodunit— LLMSniffer achieves substantial improvements over prior baselines: accuracy increases from 70% to 78% on GPTSniffer (F1: 68%→78%) and from 91% to 94.65% on Whodunit (F1: 91%→94.64%). t-SNE visualizations confirm that contrastive fine-tuning yields well-separated, compact embeddings. We release our model checkpoints, datasets, codes and a live interactive demo to facilitate further research.
1 Introduction The emergence of capable code-generating LLMs— including GitHub Copilot [GitHub, 2021], ChatGPT [OpenAI, 2023], and Code Llama [Rozière et al., 2023]—has transformed how software is written. Developers increasingly rely on these models for routine coding tasks, bug fixes, and algorithm implementation. However, this shift introduces new challenges: plagiarism detection systems in educational settings must account for AI-generated sub-
• A two-stage contrastive fine-tuning strategy for GraphCodeBERT with comment removal preprocessing and an MLP classification head, achieving state-of-the-art results on two benchmarks. 1
• Detailed ablation analysis comparing a linear classifier baseline against our improved MLPbased architecture.
work.
• Public model checkpoints, datasets on Hugging Face, an interactive Streamlit demo, and a video demonstration for full reproducibility.
3.1 GraphCodeBERT
3 Background GraphCodeBERT [Guo et al., 2021] is a transformer-based model pre-trained on code– documentation pairs from CodeSearchNet [Husain et al., 2019]. Unlike CodeBERT, it augments the self-attention mechanism with a data-flow graph that captures variable-level semantic dependencies (e.g., where a variable is computed and where it is used). Two additional pre-training objectives— edge prediction and node alignment—encourage the model to internalize these structural relationships. We use the [CLS] token representation as a fixed-dimensional embedding h ∈ R768 for downstream classification.
2 Related Work AI-Generated Text Detection. Detecting machinegenerated text has received considerable attention since GPT-2 [Radford et al., 2019]. Early approaches used statistical signals such as perplexity [Gehrmann et al., 2019] and likelihood ratios [Mitchell et al., 2023], while more recent work focuses on fine-tuned classifiers [Solaiman et al., 2019] and watermarking [Kirchenbauer et al., 2023]. These methods target natural language and do not transfer directly to source code due to syntactic and structural differences.
3.2 Supervised Contrastive Learning
LLM-Generated Code Detection. Nguyen et al. [2024b] introduced GPTSniffer, one of the first dedicated benchmarks for detecting ChatGPTgenerated code, using a CodeBERT encoder trained with binary cross-entropy loss. Nguyen et al. [2024a] presented Whodunit, focusing on GPT-4generated Python solutions to CodeChef problems and demonstrating that code-specific features substantially improve detection. Idialu et al. [2024] further explored zero-shot and ML-based pipelines on competitive programming benchmarks. Our work builds directly on these benchmarks and surpasses both baselines.
Given a batch of N labeled pairs {(xi , yi )}N i=1 , the supervised contrastive loss [Khosla et al., 2020] is: N X −1 X ezi ·zp /τ LSupCon = log X |P (i)| ezi ·za /τ i=1 p∈P (i) a∈A(i)
(1) where zi = ℓ2 -normalize(MLP(hi )) is the projected representation, P (i) = {p ̸= i : yp = yi } is the set of same-class positives, A(i) = {a ̸= i} is all other examples, and τ is a temperature hyperparameter. This objective directly optimizes for inter-class separation and intra-class compactness in embedding space.
Code Representation Learning. Pre-trained models such as CodeBERT [Feng et al., 2020], GraphCodeBERT [Guo et al., 2021], and CodeT5 [Wang et al., 2021] advance code understanding by encoding token sequences alongside structural information (ASTs, data-flow graphs). We leverage GraphCodeBERT for its demonstrated superiority on code search and clone detection.
4 Methodology 4.1 Preprocessing: Comment Removal A key design decision in LLMSniffer is removing code comments before encoding. LLM-generated code often includes verbose, explanatory comments that reflect LLM writing style rather than algorithmic structure. Stripping comments ensures the model learns to distinguish based on syntactic and structural code properties, reducing the risk of exploiting superficial stylistic cues and improving generalization.
Contrastive Learning. Supervised contrastive learning [Khosla et al., 2020] extends selfsupervised contrastive objectives [Chen et al., 2020] to the labeled setting. It has been applied to NLP fine-tuning [Gunel et al., 2021] and code clone detection [Ding et al., 2022], but not, to our knowledge, to AI-generated code detection prior to this 2
Figure 1: Overview of LLMSniffer. Stage 1 trains the encoder and projection head with supervised contrastive loss. Stage 2 freezes the encoder and trains an MLP classifier with binary cross-entropy loss.
4.2 Model Architecture LLMSniffer consists of three components (Figure 1): (1) GraphCodeBERT Encoder. After comment removal, the code snippet is tokenized and fed along with its data-flow graph edges into GraphCodeBERT. The [CLS] embedding h ∈ R768 serves as the code representation. (2) Projection Head. A two-layer MLP with ReLU activation maps h to z ∈ R128 for contrastive learning. The head is discarded at inference time, following Chen et al. [2020]. (3) MLP Classification Head. A multi-layer perceptron with sigmoid activation maps h to a binary prediction (human-written or AI-generated). We find that this MLP head outperforms the simpler linear classifier used in prior work.
pre-training step warms up the encoder to produce discriminative representations before the classifier is trained, following the approach of Khosla et al. [2020]. 4.4 Implementation Details We initialize from the microsoft/graphcodebert-base checkpoint (HuggingFace). Both training stages use AdamW [Loshchilov and Hutter, 2019] with learning rate 2×10−5 and linear warmup over 10% of training steps. Maximum sequence length is 512 tokens. Batch sizes are 32 for contrastive training and 64 for classification fine-tuning. The projection head uses hidden dimension 512 and output dimension 128. All experiments are conducted on a single GPU.
5 Experimental Setup
4.3 Training Procedure Training proceeds in two stages. Stage 1: Contrastive Pre-training. We fine-tune the encoder and projection head using LSupCon (Equation 1) with temperature τ =0.07. Batches are constructed with balanced classes to maximize the number of positive pairs per anchor. Stage 2: Classification Fine-tuning. The projection head is discarded, the MLP classification head is attached, and the full model is fine-tuned end-toend with binary cross-entropy loss. The contrastive
5.1 Datasets GPTSniffer [Nguyen et al., 2024b] is a dataset of code snippets with binary labels indicating human- or ChatGPT-authorship, spanning multiple programming languages and competitive programming problem types. We use the official split: 600 human + 600 ChatGPT samples for training, and 131 human + 142 ChatGPT samples for testing. We use the dataset hosted at https://huggingface.co/datasets/ mahirlabibdihan/GPTSniffer. Crucially, 3
the test set comes from a completely different domain than the training set, making this a challenging cross-domain evaluation. Whodunit [Nguyen et al., 2024a] is a dataset of Python solutions to CodeChef problems, with code generated by GPT-4 and human-written counterparts. We use: 639 human + 639 ChatGPT samples for training, and 159 human + 159 ChatGPT samples for testing. The dataset is hosted at https://huggingface.co/datasets/ mahirlabibdihan/Whodunit.
Model
Acc.
Prec.
Rec.
F1
GPTSniffer [2024b] LLMSniffer (Linear) LLMSniffer (Ours)
0.70 0.70 0.78
0.77 0.81 0.78
0.69 0.69 0.77
0.68 0.67 0.78
Table 1: Results on the GPTSniffer test set (macroaveraged). Metrics are reported as proportions. The linear classifier variant shows no improvement over the baseline; the MLP with comment removal achieves +8 pp accuracy and +10 pp F1.
Pre-training Data. Both datasets are augmented with CodeSearchNet [Husain et al., 2019] (humanwritten code) and CodeNet [Puri et al., 2021] (human competitive programming solutions) for encoder warming.
Model
Acc.
Prec.
Rec.
F1
Whodunit [2024a] LLMSniffer
0.91 0.947
0.91 0.949
0.91 0.943
0.91 0.946
∆
+3.65
+3.94
+3.34
+3.64
Table 2: Results on the Whodunit test set. Consistent gains across all metrics.
5.2 Baselines and Ablations We compare against two baselines and an intermediate ablation:
both important components. Our full model (MLP + comment removal) achieves 0.78 accuracy and 0.78 F1, improving both precision and recall simultaneously, unlike the prior baseline which suffered from a severe precision–recall imbalance (high precision, low recall for the human class).
• GPTSniffer [Nguyen et al., 2024b]: CodeBERT fine-tuned with binary cross-entropy loss. • Whodunit [Nguyen et al., 2024a]: Featurebased XGBoost classifier with 140 handcrafted code features.
6.2 Whodunit Benchmark Table 2 presents results on the Whodunit benchmark. On Whodunit, LLMSniffer achieves consistent improvements across all metrics (+3.6 pp), reflecting the dataset’s balanced structure and the robustness of our approach to GPT-4 generated code in Python.
• LLMSniffer (Linear): Our Stage 1+2 pipeline with a linear classification head instead of an MLP (no comment removal). • LLMSniffer (Ours): Full pipeline with comment removal and MLP classification head. 5.3 Evaluation Metrics We report Accuracy, Precision, Recall, and F1 Score on each benchmark’s test set. All metrics are macro-averaged across the two classes.
6.3 Embedding Visualization Figure 2 shows t-SNE [van der Maaten and Hinton, 2008] projections of [CLS] embeddings on the GPTSniffer test set for the baseline and LLMSniffer. The baseline model produces heavily overlapping clusters for the two classes. LLMSniffer, by contrast, forms a compact, well-isolated cluster of AI-generated snippets that is cleanly separable from human-written code. This qualitative result directly explains the quantitative improvements and confirms that supervised contrastive learning is the core driver of discriminability.
6 Results 6.1 GPTSniffer Benchmark Table 1 compares all systems on the GPTSniffer test set. The linear classifier variant fails to improve over the GPTSniffer baseline, achieving the same overall accuracy (0.70). This highlights that contrastive pre-training alone is insufficient—the classification head architecture and comment removal are 4
Figure 2: t-SNE of test set [CLS] embeddings (GPTSniffer benchmark). Left (GPTSniffer baseline): The two classes (human=blue, AI=orange) are highly interleaved with no clear decision boundary. Right (LLMSniffer): Contrastive fine-tuning produces a compact, isolated AI-generated cluster (right) that is linearly separable from the human-written cluster, directly explaining the improved classification performance.
Configuration
Acc.
7 Demo and Deployment
F1
We deployed LLMSniffer as a Streamlit web application hosted on Hugging Face Spaces:1 Users paste a code snippet in any supported language and receive a real-time binary prediction with a confidence probability. Model checkpoints for both GPTSniffer and Whodunit are publicly available.2 A video demonstration of the tool is available at https://youtu. be/xkb_vrEfJIY. The datasets are hosted at https://huggingface.co/datasets/ mahirlabibdihan/GPTSniffer and https://huggingface.co/datasets/ mahirlabibdihan/Whodunit. The implementation code is available at https: //github.com/mahirlabibdihan/ llmsniffer.
CodeBERT + BCE (baseline) 0.70 0.68 GraphCodeBERT + SupCon + Linear 0.70 0.67 GraphCodeBERT + SupCon + MLP 0.78 0.78 Table 3: Ablation on GPTSniffer. Upgrading from CodeBERT to GraphCodeBERT with contrastive learning but keeping a linear head yields no gain; switching to the MLP classifier (with comment removal) provides the decisive improvement.
6.4 Ablation Study Table 3 summarizes the contribution of each component on GPTSniffer. The ablation reveals that the gain is not attributable to GraphCodeBERT or contrastive learning alone. The decisive improvement comes from combining contrastive pre-training with the MLP classification head and comment removal preprocessing. We hypothesize that the MLP head is better able to exploit the non-linear structure of the contrastively-learned embedding space, while comment removal prevents the model from learning superficial authorship cues in docstrings and inline comments.
8 Conclusion We presented LLMSniffer, a detection framework for LLM-generated code that combines GraphCodeBERT with supervised contrastive learning, comment removal preprocessing, and an MLP classification head. Our ablation study shows that the full combination is necessary—contrastive learning 1
https://huggingface.co/spaces/ mahirlabibdihan/LLMSniffer 2 https://huggingface.co/ mahirlabibdihan/LLMSniffer
5
References
alone with a linear head does not improve over the baseline. The complete system achieves state-ofthe-art results on both the GPTSniffer and Whodunit benchmarks, with the t-SNE visualizations providing a clear geometric explanation for the gains. Future work will explore: (1) multi-class attribution distinguishing among specific LLMs (GPT4, Claude, Code Llama, etc.); (2) adversarial robustness against obfuscation attacks; (3) integration of AST features alongside data-flow graphs; and (4) extension to low-resource programming languages not covered by current benchmarks.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597– 1607, 2020. URL https://arxiv.org/ abs/2002.05709. Z. Ding et al. Towards understanding the capability of large language models on code clone detection: A survey. In Proceedings of the International Conference on Software Engineering (ICSE), 2022. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547, 2020. doi: 10.18653/v1/2020.findings-emnlp.139.
Limitations LLMSniffer processes snippets up to 512 tokens; longer files require truncation which may degrade accuracy. The GPTSniffer training and test sets come from different domains, and while our model handles this well, further domain shifts (e.g., different LLM families or programming paradigms) may require retraining. As LLMs produce increasingly human-like code, detection performance may degrade over time without periodic retraining on updated data distributions. Adversarial robustness against deliberate paraphrasing or obfuscation remains an open question.
S. Gehrmann, H. Strobelt, and A. M. Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111– 116, 2019. doi: 10.18653/v1/P19-3019. GitHub. GitHub Copilot: Your AI pair programmer, 2021. URL https://github.com/ features/copilot.
Ethics Statement LLM-generated code detection tools carry potential for misuse. False positives could unfairly penalize students or developers whose code style resembles LLM output. We caution against using LLMSniffer as the sole basis for consequential academic or professional decisions without human review. We encourage policies that treat model-assisted coding as a tool to be used responsibly, rather than a practice to be categorically prohibited.
B. Gunel, J. Du, A. Conneau, and V. Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/ 2011.01403. D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou. GraphCodeBERT: Pre-training code representations with data flow. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/ forum?id=jLoC4ez43PZ.
Acknowledgments The authors thank the Bangladesh University of Engineering and Technology for computational support. We also thank the creators of the GPTSniffer, Whodunit, CodeSearchNet, and CodeNet datasets for making their resources publicly available. 6
C.-Y. Hung et al. Exploring the feasibility of automated detection of LLM-generated code through explainable AI. arXiv preprint, 2024.
of Systems and Software, 2024b. doi: 10.1016/j.jss.2024.112031. URL https: //www.sciencedirect.com/science/ article/pii/S0164121224001043.
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019. URL https://arxiv.org/abs/ 1909.09436.
OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https:// arxiv.org/abs/2303.08774. R. Puri, D. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, et al. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks, 2021. URL https: //arxiv.org/abs/2105.12655.
J. Idialu et al. Our students are using ChatGPT: Detecting AI-generated code in programming assignments. arXiv preprint, 2024. P. Khosla, Y. Tian, M. Tschannen, J. Lienen, Y. Zhang, L. Jiang, D. Krishnan, and Y. Tian. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 18661–18673, 2020. URL https://arxiv.org/abs/ 2004.11362.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. URL https://openai.com/blog/ better-language-models. B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. URL https:// arxiv.org/abs/2308.12950.
J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A watermark for large language models. In International Conference on Machine Learning (ICML), 2023. URL https: //arxiv.org/abs/2301.10226. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://arxiv.org/abs/ 1711.05101.
H. Shi et al. Zero-shot detection of LLM-generated code. arXiv preprint, 2024. I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, and J. Wang. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019. URL https://arxiv.org/abs/ 1908.09203.
E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. DetectGPT: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. URL https://arxiv.org/abs/ 2301.11305.
L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579–2605, 2008. URL https://www.jmlr.org/papers/ v9/vandermaaten08a.html.
P. T. Nguyen et al. Whodunit? classifying code as human authored or GPT-4 generated: A case study on CodeChef problems. arXiv preprint arXiv:2403.04013, 2024a. URL https:// arxiv.org/abs/2403.04013.
Y. Wang, W. Wang, S. Joty, and S. C. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan-
P. T. Nguyen et al. GPTSniffer: Detecting ChatGPT-generated code via fine-tuned language models. Journal 7
guage Processing (EMNLP), pages 8696–8708, 2021. doi: 10.18653/v1/2021.emnlp-main.685. J. Ye et al. Detecting LLM-generated code: A zeroshot approach. arXiv preprint, 2024.
A Dataset Statistics Dataset
Split
Human
AI-Gen.
Total
GPTSniffer
Train Test
600 131
600 142
1200 273
Whodunit
Train Test
639 159
639 159
1278 318
Table 4: Dataset statistics. Note that the GPTSniffer test set comes from a completely different domain (source) than the training set.
B Hyperparameter Details Hyperparameter Base model Max sequence length Contrastive batch size Classification batch size Learning rate Optimizer Warmup ratio Contrastive temp. τ Projection dim Projection hidden dim
Value graphcodebert-base 512 32 64 2 × 10−5 AdamW 0.10 0.07 128 512
Table 5: Hyperparameter settings for LLMSniffer.
8