When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions Ravishka Rathnasuriya
The University of Texas at Dallas USA [email protected]
arXiv:2605.19369v1 [cs.SE] 19 May 2026
Abstract Code language models are increasingly adopted for both understanding and generative tasks. Despite their success, these models frequently produce overconfident incorrect predictions and underconfident correct predictions, undermining their reliability in deployment. Practical deployment demands three capabilities: accurately estimating the likelihood of correctness, abstaining on uncertain predictions, and invoking external mechanisms to validate or repair abstained outputs. Existing calibration and uncertainty estimation methods, primarily developed for natural language tasks, do not readily transfer to code. Notably, post-hoc calibration techniques often reduce probability misalignment but fail to improve the ranking of predictions by correctness likelihood—a requirement for selective prediction under partial coverage. Furthermore, most approaches treat uncertainty as a passive indicator rather than an actionable signal. This work introduces a unified framework that integrates uncertainty estimation, model calibration, and tool-based abstention handling for code models. The proposed design enables models to assign reliable correctness probabilities, abstain under uncertainty, and invoke lightweight program analysis procedures to process abstained cases. By combining these components within a single deployment-oriented workflow, this framework supports risk-aware, coverage-controlled use of code models across both classification and generation settings. ACM Reference Format: Ravishka Rathnasuriya and Wei Yang. 2026. When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-NIER ’26), April 12–18, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3786582.3786845
1
Introduction
Code language models are increasingly applied across two primary classes of tasks: classification tasks, including vulnerability detection and defect prediction [7, 12, 15, 17, 27, 28, 31–33], and generative tasks, such as code completion, synthesis, and repair [4, 9, 13, 14, 22]. These models are now integral to developer workflows via integration into IDEs, CI pipelines, and automated programming assistants. However, despite their success, these models frequently exhibit poor alignment between predictive confidence
This work is licensed under a Creative Commons Attribution 4.0 International License. ICSE-NIER ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2425-1/2026/04 https://doi.org/10.1145/3786582.3786845
Wei Yang
The University of Texas at Dallas USA [email protected] and actual correctness [25, 34]. In practice, they often assign high confidence to incorrect predictions and low confidence to correct ones which undermine their reliability, especially when used in high-stakes or partially automated environments. Reliable deployment of code models requires more than producing high-confidence outputs. It requires mechanisms that can estimate, for each prediction, whether it is likely to be correct. This entails three key capabilities. First, models must produce a wellcalibrated estimate of per-sample correctness, that is, the predicted confidence should reflect the true likelihood of correctness. Second, systems must support selective abstention, allowing models to withhold predictions when uncertainty is high. Third, there must be principled mechanisms for acting on abstained inputs, for example, by routing them to lightweight analysis tools, heuristic checks, or human review. These capabilities enable a deployment setting where model outputs can be accepted only when trustworthy, and otherwise deferred for further handling. While uncertainty estimation and calibration have been studied extensively in other domains [3, 5, 30, 34], existing methods face fundamental limitations when applied to code. Confidence signals often fail to capture the structured and semantic nature of code predictions. Similarly, post-hoc calibration techniques [3, 25, 34] may reduce average miscalibration but do not improve per-sample reliability in a way that supports effective abstention decisions. Moreover, these methods typically operate in isolation, with no integrated support for resolving the uncertain cases they identify. Empirical studies from our prior work further underscore this gap [25, 34]. We find that uncertainty signals [2, 6, 10, 16, 19– 21, 24, 26, 29], across both classification and generation tasks, often correlate poorly with correctness, particularly in code scenarios involving rare patterns, ambiguous syntax, or long-range dependencies. Calibration improves score alignment but fails to improve abstention efficacy: the calibrated probabilities do not reliably distinguish between trustworthy and untrustworthy predictions. These findings motivate the need for a unified formulation that not only estimates uncertainty and calibrates output confidence, but also makes abstention operationally useful. We propose a deployment-oriented framework for code models that addresses this gap. Our formulation supports both classification and generation tasks, and integrates three essential components: (i) the construction of more informative uncertainty signals tailored to code, (ii) both pre-deployment and post-deployment calibration strategies that improve the interpretability of model confidence, and (iii) the incorporation of model–context–protocol (MCP) [23] tools to handle abstained predictions through lightweight rule-based, static, or semantic analyses. This approach supports coverage-controlled decision-making and enables actionable
ICSE-NIER ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Ravishka Rathnasuriya and Wei Yang
abstention pipelines without requiring model retraining as a prerequisite. Together, these components provide a foundation for robust and risk-aware deployment of code models in real-world software engineering workflows.
2
Our Envision
We propose a structured framework for deploying code intelligence systems that unifies model uncertainty, selective abstention, and tool-informed recovery into a single decision process (Figure 1). This design allows models to generate confidence-aware outputs, withhold unreliable predictions, and route uncertain cases through task-specific post-processing. Our goal is to move beyond passive confidence reporting and toward operational systems that act cautiously under uncertainty and adapt their behavior based on calibrated, interpretable signals. Calibration Techniques Pre-Deployment Calibration
Task Specific Models Classification Models
Classification Model
Generative Models
a. Abstention Head b. Extra Reject Class
a. Reward-Aware Training
Uncertainty Estimation
Inputs
Generative Models Calibrated Scores
No
Post-Deployment Calibration
Abstain
?
Calibrated Scores
Results
Classification Model
Generative Models
a. Weighted-Based b. Variance Based
a. Token-level b. Sequence-level
Yes
Task Specific MCP Tools Classification Tasks
Generative Tasks Feedback
a. Tool-Based Analysis (e.g. Static Analysis)
a. Adjust Decoding Parameters b. Inject New Information c. Tool-Based Analysis Information d. Use Model Reasoning
Figure 1: Overview of the Proposed Framework Inputs and model scope. The framework begins with taskspecific inputs, each routed to either a classification model or a generative model [4, 8, 11, 22]. Classification tasks include vulnerability detection, defect prediction, or API misuse identification, where the model maps a code representation to a discrete label. Generative tasks include code synthesis, completion, or repair, where the model produces a token sequence given a partial specification or prompt [7, 12, 15, 17, 27, 28, 31–33]. The inputs may consist of code snippets, function signatures, or contextual metadata, and are tokenized and encoded according to model requirements. Each model then produces an output, which is a prediction or a completion, along with internal signals used to compute uncertainty. Uncertainty estimation. The framework begins by extracting internal uncertainty signals from the model, serving as indicators of prediction confidence. Unlike evaluation-centric confidence measures, these signals are used to inform downstream decisions, specifically, whether to trust a model output or defer it. To support this, we adopt uncertainty estimation strategies that reflect model reliability in both classification and generative tasks. These include metrics derived from predictive distributions as well as methods that quantify model variability under controlled perturbations [2, 6, 10, 16, 19–21, 24, 26, 29]. The motivation is to enable the system to make task-specific deferral decisions without
relying on costly external validation. These signals are not used directly to filter outputs but instead provide the basis for computing correctness likelihood via calibration. Pre-deployment calibration. Standard code models are not trained to produce abstention behavior or calibrated uncertainty. Their objectives focus on maximizing prediction accuracy without accounting for decision risk. To enable confidence-aware abstention, we introduce pre-deployment calibration mechanisms that adjust model behavior during training. For classification models, we introduce mechanisms that allow the model to explicitly abstain. One approach adds a dedicated abstention output pathway trained to identify low-confidence examples. This abstention head is supervised with selective risk-aware losses, encouraging the model to defer when it cannot produce reliable predictions. Another approach extends the label space to include a reject class, allowing the model to route ambiguous inputs away from fixed categories. These strategies convert the model from a closed-world classifier into a system that can explicitly indicate when it should not decide. For generative models, pre-deployment calibration is implemented via reward-aware training. The model is optimized not only to produce syntactically fluent outputs, but to align generation with correctness proxies such as validation pass rates or task-specific behavior. This changes the behavioral geometry of the model where the model learns to be conservative when uncertain and to structure its generations around downstream acceptability. These techniques require retraining and modify the predictive behavior itself, enabling the model to internalize abstention as a learned action. Post-deployment calibration. While pre-deployment techniques change model behavior directly, post-deployment calibration enables abstention over existing models without retraining. The objective here is to convert raw uncertainty signals into calibrated correctness scores that can support abstention decisions. For classification models, we introduce two forms of post deployment calibration. The first is a weighted calibration scheme, where standard calibration mappings [3, 25, 34] are trained using loss functions that emphasize certain examples—such as high-error or underrepresented classes—more predominantly. This reflects the real-world need to prioritize reliability over uniform calibration error. The second approach is a variance-aware calibration method, where confidence scores are derived from repeated perturbations of model predictions, and their dispersion is mapped to correctness probabilities. This captures distributional uncertainty more effectively and allows calibrated scores to reflect stability, not just average alignment. For generative models, post-deployment calibration can operate at multiple granularities. At the token level, uncertainty scores associated with each step of the generation are recalibrated to reflect empirical correctness likelihoods. This is useful in settings like code completion or repair, where partial outputs must be trusted incrementally. At the sequence level, the entire output is assigned a correctness score based on features such as output consistency, structure, and coverage, calibrated against reference data. This holistic calibration allows the system to determine whether the generation, taken as a whole, should be served or deferred. These
When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions
calibrated scores are then compared to abstention thresholds to decide whether to accept or withhold predictions. Abstention decisions and recovery. Once a calibrated correctness score is available, the system enforces a policy such that, if the score exceeds a tunable threshold, the prediction is accepted; otherwise, the input is abstained. However, abstention is not terminal. The system activates recovery modules based on the nature of the input and task. For generative tasks, we introduce a MCP layer that resolves abstained predictions via targeted interventions. This layer addresses three distinct failure modes. First, when the model fails due to missing task information, such as under-specified prompts, the MCP layer augments the input. It may invoke model reasoning strategies to generate intermediate plans or sketches that clarify intent. If the uncertainty indicates multiple plausible completions, decoding is diversified, and candidates are re-ranked based on follow-up uncertainty estimation. This turns an ambiguous prompt into a controlled exploration. Second, when failure arises from missing external knowledge, such as unknown libraries or undocumented behaviors, the system injects contextual information, such as documentation snippets or relevant specifications. In addition, lightweight analyzers or compilers may be invoked to validate outputs. These tools convert knowledge uncertainty into evidence the model can act on, reducing epistemic uncertainty and improving downstream calibration. Third, if the model faces a capability gap where the input falls outside the model’s learned competence, the MCP system enforces simplifications. This may include constraining generation length, enforcing intermediate validation steps, or breaking the task into smaller subtasks. If uncertainty remains high after such interventions, the system abstains definitively, ensuring correctness is never sacrificed for coverage. For classification tasks, abstention triggers domain-specific recovery. Static analyzers, program slicing tools, or rule-based validators are applied to the input. These tools assess properties such as control flow, data dependencies, or security patterns to validate or reject predictions. This external analysis layer ensures that classification decisions are always supported by either model confidence or symbolic evidence. This work reframes abstention as a principled decision-making mechanism rather than a fixed thresholding heuristic in future code intelligence systems.
3
Contributions
In summary, this research makes the following contributions: • Unified framework for abstention across code modeling tasks: We envision a framework that generalizes across classification and generative tasks, enabling selective abstention and structured recovery under uncertainty. • Task- and model-specific uncertainty calibration: We introduce calibration strategies tailored to both classification and generative models. • Pre- and post-deployment abstention mechanisms: We demonstrate how models can be trained to abstain explicitly with abstention heads, reject classes or retrofitted via calibrated confidence thresholds without retraining.
ICSE-NIER ’26, April 12–18, 2026, Rio de Janeiro, Brazil
• MCP-guided abstention recovery: We define a MCP policy layer for resolving abstained predictions via tool-guided interventions. • Abstention as an actionable decision primitive: We reconceptualize abstention not as failure, but as a route to recovery. The framework incorporates fallback execution and optional feedback loops that adapt to uncertainty in real time. • Open science and reproducibility commitment: All code, configuration files, and calibration datasets will be released under a permissive license upon acceptance.
4
Preliminary Results
We investigate whether uncertainty metrics and calibration techniques developed for language and vision tasks can transfer effectively to pretrained code models under both classification and generation settings. Our evaluation is guided by three core questions: (1) Do uncertainty metrics align with correctness in code tasks? (2) Can calibration techniques produce trustworthy abstention decisions? (3) Does abstention improve accuracy under coverage constraints? The findings below support the necessity of behavioral calibration and abstention-aware deployment.
4.1
Impact of Weighted Scaling for Generative Models
We implement a lightweight calibration method for generative models based on a weighted logistic scaling approach. The technique maps raw model confidence scores to calibrated correctness probabilities using a monotonic transformation, where the loss function upweights incorrect predictions to penalize overconfident errors. Due to space constraints, we report results on two representative models: DeepSeek-Coder-7b and CodeLlama-7b on the MBPP+ [1] benchmark. However, we have applied the technique across multiple models and tasks with similar trends. For comparison, we evaluate this against platt scaling and isotonic regression. Results are summarized in Table 1. Our findings show that weighted scaling consistently improves both calibration error (ECE) [3] and brier score (Brier) [3] across models. Selective prediction at 80% coverage achieves over 70% accuracy, indicating that calibrated abstention decisions help isolate reliable outputs. These results validate our framework’s emphasis on conservative, task-specific calibration mechanisms for generative code modeling.
4.2
Effectiveness of Logit-Based Calibration in Classification
We propose a behavioral calibration method that operates directly on model logits, bypassing softmax compression and preserving richer information about class separability. Instead of mapping post-softmax scores, our approach trains a correctness estimator directly on raw logit features using a lightweight regressor. This design allows the model to retain decision boundary information and improve per-instance reliability. We apply this method to defect prediction [18] using DeepSeek-Coder-7b and Qwen-Coder-7b, with results shown in Table 2. Due to space constraints, only these two models are reported here, but we have evaluated the method across multiple model-task pairs and observe consistent improvements in calibration and selective prediction. The method yields
ICSE-NIER ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Ravishka Rathnasuriya and Wei Yang
Table 1: Calibration Results on MBPP+ (↓) DeepSeek–Coder–7B
CodeLlama–7B
Method
Brier
ECE
Brier
ECE
Base Model Platt Scaling Isotonic Regression Weighted Platt Calibration
0.273 0.224 0.216 0.162
0.223 0.103 0.143 0.072
0.220 0.248 0.215 0.172
0.108 0.062 0.054 0.045
Table 2: Calibration Results on defect prediction (↓). DeepSeek–Coder–7B
Qwen–Coder-7B
Method
Brier
ECE
Brier
ECE
Base Temperature Scaling Platt Scaling Isotonic Regression Confidence
0.130 0.131 0.134 0.130 0.098
0.029 0.035 0.056 0.015 0.012
0.137 0.137 0.140 0.137 0.089
0.023 0.023 0.050 0.016 0.011
the lowest brier score and ECE scores among tested strategies, and at 80% coverage, achieves over 90% selective accuracy. These results highlight the benefit of behavioral calibration, thus aligning confidence to correctness at the instance level, rather than relying solely on distributional metrics.
4.3
Evaluation of Uncertainty Metrics Across Tasks
We evaluate 16 uncertainty metrics [2, 6, 10, 16, 19–21, 24, 26] across multiple tasks: defect prediction, vulnerability detection, and code generation. These include entropy-based scores, confidence margins, variance estimators, and sampling-based disagreement measures. Our findings show that while many metrics correlate with prediction correctness, few can be directly used to implement abstention in a reliable way. Without calibration, uncertainty metrics often overestimate reliability on erroneous predictions. Even after applying standard calibration mappings, improvements in selective prediction vary widely by task and model. In particular, no single uncertainty metric emerges as a task-agnostic solution. This supports one of our key insights: effective abstention in code tasks requires behavioral calibration, i.e., alignment of internal confidence with correctness at the per-instance level. Unlike settings where probabilistic confidence reflects quality (e.g., classification on natural images), code models often emit confident but incorrect predictions due to superficial pattern matching, semantic ambiguity, and structural repetition. These characteristics limit the utility of uncalibrated uncertainty metrics and motivate our call for abstention-aware calibration pipelines.
5
Future Plan
Our current efforts have focused on a systematic investigation of existing calibration methods, uncertainty metrics, and their impact on confidence estimation in pretrained code models. Based on these insights, we have begun building a structured foundation for abstention-aware deployment in both classification and generation
tasks. Looking forward, our research roadmap includes both shortterm objectives and longer-term ambitions to operationalize, scale, and generalize the proposed framework. Short-Term Plan. Our short-term goals focus on solidifying the empirical foundation and translating the proposed design into implementable modules: (1) Publishing empirical findings: We aim to disseminate our results on the limitations of existing calibration methods and the effectiveness of abstention-aware strategies. These findings will help motivate the community to re-express confidence as a decision-making primitive; (2) Designing and validating envisioned calibration techniques: We will systematically implement the techniques described in our envisioned framework including both pre-deployment and post-deployment calibration methods for classification and generative models. This includes reward-aware training, weighted scaling, and logit-based behavioral calibration; and (3) Integrating MCP tools: We will implement concrete MCP modules for both model types. For classification, this includes rulebased validators and static analysis backends and for generation, we will integrate constraint-based decoding, prompt augmentation, and lightweight synthesis validators. These milestones are planned for completion within a 12 month timeframe and will result in a reproducible, modular codebase covering the full pipeline. Long-Term Plan. Beyond initial implementation, we aim to generalize, scale, and evaluate the framework in real-world deployment contexts. Our long-term vision includes: (1) Expanding across tasks, models, and domains: We plan to extend the evaluation across broader model families and additional code intelligence tasks, such as code summarization, clone detection, and test generation. This will help assess task- and model-specific calibration needs; Blackbox model integration and robustness testing: We aim to validate abstention strategies in scenarios where model internals are not accessible. This will test the applicability of our techniques under limited observability and motivate proxy calibration methods; (3) Developer-facing deployment integration: We will integrate our abstention framework into real-world development tools, including IDEs and CI systems. This will allow us to evaluate not only model behavior, but also the usability and trustworthiness of abstention as perceived by developers; (4) End-to-end system engineering and nextgeneration MCP tooling: We plan to build a full abstention-aware inference stack, supporting calibration-aware serving, feedbackdriven refinement, and dynamic tool invocation. Over time, we will co-develop next-generation MCP tools that combine program analysis with model introspection to guide abstention and recovery adaptively; and (5) Open ecosystem and community contribution: All tools, datasets, and evaluation protocols will be released with support for easy extension, benchmarking, and reproducibility. This will facilitate downstream research on abstention-aware learning, calibration, and human-in-the-loop code AI. These milestones are planned for completion within a 12 to 18 month timeframe. Together, these directions aim to transform abstention from an evaluation artifact into a principled mechanism for safe, adaptive, and trustworthy deployment of code intelligence models.
6
ACKNOWLEDGMENTS
This work was partially supported by NSF grants NSF CCF2146443 and Amazon Trust AI Research Award.
When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions
References [1] Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. Codescore: Evaluating code generation by learning code execution. ACM Transactions on Software Engineering and Methodology 34, 3 (2025), 1–22. [2] Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. PMLR, 1050–1059. [3] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. arXiv:1706.04599 [cs.LG] [4] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [5] Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. 2020. Calibration of neural networks using splines. arXiv preprint arXiv:2006.12800 (2020). [6] Dan Hendrycks and Kevin Gimpel. 2018. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv:1610.02136 [cs.NE] [7] Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Lei Ma, and Yves Le Traon. 2023. CodeS: towards code model generalization under distribution shift. In International Conference on Software Engineering (ICSE): New Ideas and Emerging Results (NIER). [8] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [9] Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 1–30. [10] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017). [11] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). [12] Yufei Li, Simin Chen, and Wei Yang. 2021. Estimating predictive uncertainty under program data distribution shift. arXiv preprint arXiv:2107.10989 (2021). [13] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1qvx610Cu7 [14] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating Language Models for Efficient Code Generation. In First Conference on Language Modeling. https://openreview.net/forum?id= IBCBMeAhmC [15] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021). [16] Robert Munro Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster. [17] Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: fuzzing by program transformation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 697–710. [18] Anh Viet Phan and Minh Le Nguyen. 2017. Convolutional neural networks on assembly code for predicting software defects. In 2017 21st Asia Pacific Symposium on Intelligent and Evolutionary Systems (IES). 37–42. doi:10.1109/IESYS.2017. 8233558 [19] Ravishka Rathnasuriya. 2025. A Framework for On the Fly Input Refinement for Deep Learning Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 140–144. [20] Ravishka Rathnasuriya. 2025. On the Fly Input Refinement for Code Language Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 230–231. [21] Ravishka Rathnasuriya, Zijie Zhao, and Wei Yang. 2025. CodeImprove: Program Adaptation for Deep Code Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 676–676. [22] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [23] Anjana Sarkar and Soumyendu Sarkar. 2025. Survey of LLM Agent Communication with MCP: A Software Design Pattern Centric Review. arXiv preprint arXiv:2506.05364 (2025). [24] Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 (1948), 379–423.
ICSE-NIER ’26, April 12–18, 2026, Rio de Janeiro, Brazil
[25] Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2024. Calibration and correctness of language models for code. arXiv preprint arXiv:2402.02047 (2024). [26] Jacob Steinhardt and Percy S Liang. 2016. Unsupervised risk estimation using only conditional independence structure. Advances in Neural Information Processing Systems 29 (2016). [27] Zhao Tian, Junjie Chen, and Xiangyu Zhang. 2023. On-the-fly Improving Performance of Deep Code Models via Input Denoising. arXiv preprint arXiv:2308.09969 (2023). [28] Rijnard van Tonder and Claire Le Goues. 2020. Tailoring programs for static analysis via program transformation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 824–834. [29] Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Kirill Grishchenkov, Sergey Petrakov, et al. 2024. Benchmarking uncertainty quantification methods for large language models with lm-polygraph. arXiv preprint arXiv:2406.15627 (2024). [30] Ruslan Vasilev and Alexander D’yakonov. 2023. Calibration of neural networks. arXiv preprint arXiv:2303.10761 (2023). [31] Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural Attack for PreTrained Models of Code. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1482–1493. doi:10.1145/3510003.3510146 [32] Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. Proceedings of the ACM on Programming Languages 4, OOPSLA (2020), 1–30. [33] Weiwei Zhang, Shengjian Guo, Hongyu Zhang, Yulei Sui, Yinxing Xue, and Yun Xu. 2023. Challenging Machine Learning-based Clone Detectors via Semanticpreserving Code Transformations. IEEE Transactions on Software Engineering 49, 5 (May 2023), 3052–3070. doi:10.1109/TSE.2023.3240118 arXiv:2111.10793 [cs] [34] Zhenhao Zhou, Chaofeng Sha, and Xin Peng. 2024. On calibration of pre-trained code models. In Proceedings of the IEEE/ACM 46th international conference on software engineering. 1–13.